Finish leg-05 reliability harness and record the M2 baseline-to-after result

coilysiren commented

2026-07-02 04:46:38 +00:00

Owner

The phase-1 proxy core is built locally, but M2 is still missing a durable measured artifact. scripts/reliability_loop.py exists and docs/proxy.md still describes it as a scaffold. This issue is to close that gap.

Acceptance criteria:

run the reliability harness against both targets described in the repo docs: direct (tower /v1 with no num_ctx) and proxy (local proxy fast-think)
make the harness output durable and reproducible enough for future before/after checks, not a one-off terminal run
add or refine tests around any harness logic that is changed so the offline suite still covers the scoring path
document the measured result in the repo, including run shape, turn count, reliability percentage, and failure histogram for both targets
update the phase/status docs that claim M2 state if the measured result changes what they should say
keep the work inside phase 1. no capability-phase work, no deploy manifests, no harness repointing

Notes from this director pass:

this surfaced-session clone is read-only, so the implementation should happen in a sibling engineer run
the engineer will need tailnet reachability to the tower and SSM-backed config resolution
while touching docs, reconcile the actual ward invocation shape with what the repo tells operators to run if that blocks reproducibility (ward exec <verb> vs bare ward <verb>)

The phase-1 proxy core is built locally, but M2 is still missing a durable measured artifact. `scripts/reliability_loop.py` exists and `docs/proxy.md` still describes it as a scaffold. This issue is to close that gap. Acceptance criteria: - run the reliability harness against both targets described in the repo docs: `direct` (tower `/v1` with no `num_ctx`) and `proxy` (local proxy `fast-think`) - make the harness output durable and reproducible enough for future before/after checks, not a one-off terminal run - add or refine tests around any harness logic that is changed so the offline suite still covers the scoring path - document the measured result in the repo, including run shape, turn count, reliability percentage, and failure histogram for both targets - update the phase/status docs that claim M2 state if the measured result changes what they should say - keep the work inside phase 1. no capability-phase work, no deploy manifests, no harness repointing Notes from this director pass: - this surfaced-session clone is read-only, so the implementation should happen in a sibling engineer run - the engineer will need tailnet reachability to the tower and SSM-backed config resolution - while touching docs, reconcile the actual ward invocation shape with what the repo tells operators to run if that blocks reproducibility (`ward exec <verb>` vs bare `ward <verb>`)

coilyco-ops commented

2026-07-02 04:46:51 +00:00

Member

🔒 Reserved by ward agent --driver goose — container engineer-goose-agent-proxy-19 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T04:46:49Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Goose, via ward agent

🔒 Reserved by `ward agent --driver goose` — container `engineer-goose-agent-proxy-19` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T04:46:49Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Goose, via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 04:48:36 +00:00

Finish leg-05 reliability harness and record M2 baseline-to-after result

coilyco-ops commented

2026-07-02 09:19:04 +00:00

Member

🔒 Reserved by ward agent --driver codex — container engineer-codex-agent-proxy-19 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T09:19:02Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Codex, via ward agent

🔒 Reserved by `ward agent --driver codex` — container `engineer-codex-agent-proxy-19` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T09:19:02Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Codex, via `ward agent`

coilysiren referenced this issue

2026-07-02 09:31:15 +00:00

agent-proxy: instrument the request path with OTel spans + structlog events (tracer/logger are declared but never called) #28

coilyco-ops added the

consult

P4

labels

2026-07-02 16:42:46 +00:00

coilysiren commented

2026-07-03 18:25:47 +00:00

Author

Owner

Director pass from the Windows tower: local reachability for the engineer run is
confirmed, and one doc reconciliation this issue already flagged is now pinned
down.

Live backend for the run

Ollama is reachable from a ward container at http://host.docker.internal:11434
(the Docker Desktop host gateway on the Windows tower). No tailnet route to the
3026 tower is needed for this run.
Models present: qwen3:4b (context_length 262144) and qwen3:8b
(context_length 40960). Neither matches the current default alias tags
(qwen3:30b-a3b, qwen3-coder:30b), so the harness needs one of:
- a PROXY_MODELS_JSON override pointing fast-think at
  {"num_ctx":49152,"backends":[{"name":"local","url":"http://host.docker.internal:11434","ollama_tag":"qwen3:4b"}]}, and
- for the direct target, TOWER=host.docker.internal plus a present tag - the
  proof/harness currently hardcode qwen3-coder:30b, which is absent locally, so
  either pull that tag or parametrize the direct model tag.
The alias retirement in #32 (pass-through tags + auto num_ctx) removes this
friction, but this issue does not depend on it - the override above is enough.

ward invocation shape (the reconciliation this issue asked for)

The verbs run as ward exec <verb> (e.g. ward exec test, ward exec serve,
ward exec reliability, ward exec proof), not bare ward <verb>. README.md
and the .ward/ward.yaml header comments say ward serve / ward test etc,
which do not resolve. Fix those to ward exec ... while recording the baseline so
the run is reproducible.

Dispatching a sibling engineer run for this now.

Director pass from the Windows tower: **local reachability for the engineer run is confirmed**, and one doc reconciliation this issue already flagged is now pinned down. **Live backend for the run** - Ollama is reachable from a ward container at `http://host.docker.internal:11434` (the Docker Desktop host gateway on the Windows tower). No tailnet route to the 3026 tower is needed for this run. - Models present: `qwen3:4b` (`context_length` 262144) and `qwen3:8b` (`context_length` 40960). **Neither matches the current default alias tags** (`qwen3:30b-a3b`, `qwen3-coder:30b`), so the harness needs one of: - a `PROXY_MODELS_JSON` override pointing `fast-think` at `{"num_ctx":49152,"backends":[{"name":"local","url":"http://host.docker.internal:11434","ollama_tag":"qwen3:4b"}]}`, and - for the `direct` target, `TOWER=host.docker.internal` plus a present tag - the proof/harness currently hardcode `qwen3-coder:30b`, which is absent locally, so either pull that tag or parametrize the direct model tag. - The alias retirement in #32 (pass-through tags + auto `num_ctx`) removes this friction, but this issue does not depend on it - the override above is enough. **ward invocation shape (the reconciliation this issue asked for)** The verbs run as **`ward exec <verb>`** (e.g. `ward exec test`, `ward exec serve`, `ward exec reliability`, `ward exec proof`), not bare `ward <verb>`. `README.md` and the `.ward/ward.yaml` header comments say `ward serve` / `ward test` etc, which do not resolve. Fix those to `ward exec ...` while recording the baseline so the run is reproducible. Dispatching a sibling engineer run for this now.

coilyco-ops commented

2026-07-03 18:26:02 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-19 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T18:25:56Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-19` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T18:25:56Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-03 18:36:37 +00:00

feat(leg-05): make the reliability harness durable, tested, and ward-runnable

coilysiren commented

2026-07-03 18:48:41 +00:00

Author

Owner

⚠️ Reopened: this run's work did not land on `main`

An ephemeral ward container (claude mode) dispatched for this issue finished but its work was not merged to main, so the reaper preserved it on a branch before teardown and reopened the issue (its closes #19 never reached main). Recover from the salvage branch below.

Repo: coilyco-flight-deck/agent-proxy
Salvage branch: ward-salvage/agent-proxy-6d869fe2
Reason: merge conflict integrating onto main
Container uptime at reap: 22m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token)

Reap diagnostics

--- reap diagnostics ---
ward version:      v0.359.0
version source:    pinned via WARD_VERSION/--ward-version (v0.359.0)
HEAD:              9e0f6300c98a
origin/main:       8655304b6279
ancestry:          HEAD is NOT yet on origin/main - residual work remains to land
decision gate:     provenance missing or unreadable
reason:            merge conflict integrating onto main
provenance:        missing or unreadable
run-owned landed:  no
working tree:      2 dirty path(s)
container uptime:  22m (baked Forgejo PAT age proxy)
--- end reap diagnostics ---

Recover

git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy.git ward-salvage/agent-proxy-6d869fe2
git checkout -b ward-salvage/agent-proxy-6d869fe2 FETCH_HEAD

Working tree at reap time

 M scripts/reliability_loop.py
 M tests/test_reliability.py

— Claude (she/her), via ward agent

## ⚠️ Reopened: this run's work did not land on `main` An ephemeral `ward container` (claude mode) dispatched for this issue finished but its work was **not merged to `main`**, so the reaper preserved it on a branch before teardown and reopened the issue (its `closes #19` never reached `main`). Recover from the salvage branch below. - **Repo:** `coilyco-flight-deck/agent-proxy` - **Salvage branch:** `ward-salvage/agent-proxy-6d869fe2` - **Reason:** merge conflict integrating onto main - **Container uptime at reap:** 22m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token) ## Reap diagnostics ``` --- reap diagnostics --- ward version: v0.359.0 version source: pinned via WARD_VERSION/--ward-version (v0.359.0) HEAD: 9e0f6300c98a origin/main: 8655304b6279 ancestry: HEAD is NOT yet on origin/main - residual work remains to land decision gate: provenance missing or unreadable reason: merge conflict integrating onto main provenance: missing or unreadable run-owned landed: no working tree: 2 dirty path(s) container uptime: 22m (baked Forgejo PAT age proxy) --- end reap diagnostics --- ``` ## Recover ```bash git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy.git ward-salvage/agent-proxy-6d869fe2 git checkout -b ward-salvage/agent-proxy-6d869fe2 FETCH_HEAD ``` ## Working tree at reap time ``` M scripts/reliability_loop.py M tests/test_reliability.py ```  — Claude (she/her), via `ward agent`

coilysiren referenced this issue

2026-07-03 19:22:08 +00:00

num_ctx injection silently halved by OLLAMA_NUM_PARALLEL: verify effective context, fail loud #33

coilysiren commented

2026-07-03 19:41:50 +00:00

Author

Owner

Salvage-branch audit from the read-only director surface, relevant to finishing
this issue.

State of the #19 work right now

Landed on main: 34d08f5 feat(leg-05): make the reliability harness durable, tested, and ward-runnable, plus the .ward/ward.yaml schema migration to the
commands: map (so ward exec reliability resolves).
Stranded, NOT on main: commit 9e0f630 ("residual claude work"), preserved
by the reaper on branch ward-salvage/agent-proxy-6d869fe2. Its parent is
34d08f5, so it cherry-picks cleanly onto current main. It adds exactly the
reproducibility piece this issue needs:
- scripts/reliability_loop.py: RELIABILITY_DIRECT_MODEL env override for the
  direct target (default qwen3-coder:30b), so the harness runs where the
  present tags differ from the default table.
- tests/test_reliability.py: _endpoint_and_model resolution tests
  (test_proxy_target_uses_fast_think, test_direct_target_model_is_overridable).

Recover the stranded commit

git fetch origin ward-salvage/agent-proxy-6d869fe2
git cherry-pick 9e0f630   # clean: its parent is already on main

What still remains to close #19

The durable numbers are still not recorded. On the Windows tower the live backend
is reachable at http://host.docker.internal:11434 (qwen3:4b ctx 262144, qwen3:8b
ctx 40960). Run both targets and record real reliability % + failure histograms in
docs/reliability_baseline.md (still placeholder). For an apples-to-apples run set
RELIABILITY_DIRECT_MODEL to the same base model fast-think/pass-through
resolves to, so the only difference is the injected num_ctx.

Caveat worth noting in the baseline: on a backend with OLLAMA_NUM_PARALLEL>1
the injected num_ctx is silently halved (#33), so the "after" numbers depend on
the backend's parallelism - note the backend's NUM_PARALLEL alongside the result.

Re-dispatching an engineer to land the stranded commit, record the baseline, and
close this out.

(Also: earlier salvage branch ward-salvage/agent-proxy-65e1e88a held a superseded
earlier durable-harness attempt - 34d08f5 replaced it. Redundant, safe to prune.)

Salvage-branch audit from the read-only director surface, relevant to finishing this issue. **State of the #19 work right now** - **Landed on main:** `34d08f5 feat(leg-05): make the reliability harness durable, tested, and ward-runnable`, plus the `.ward/ward.yaml` schema migration to the `commands:` map (so `ward exec reliability` resolves). - **Stranded, NOT on main:** commit `9e0f630` ("residual claude work"), preserved by the reaper on branch `ward-salvage/agent-proxy-6d869fe2`. Its parent is `34d08f5`, so it **cherry-picks cleanly onto current main**. It adds exactly the reproducibility piece this issue needs: - `scripts/reliability_loop.py`: `RELIABILITY_DIRECT_MODEL` env override for the `direct` target (default `qwen3-coder:30b`), so the harness runs where the present tags differ from the default table. - `tests/test_reliability.py`: `_endpoint_and_model` resolution tests (`test_proxy_target_uses_fast_think`, `test_direct_target_model_is_overridable`). **Recover the stranded commit** ```bash git fetch origin ward-salvage/agent-proxy-6d869fe2 git cherry-pick 9e0f630 # clean: its parent is already on main ``` **What still remains to close #19** The durable numbers are still not recorded. On the Windows tower the live backend is reachable at `http://host.docker.internal:11434` (qwen3:4b ctx 262144, qwen3:8b ctx 40960). Run both targets and record real reliability % + failure histograms in `docs/reliability_baseline.md` (still placeholder). For an apples-to-apples run set `RELIABILITY_DIRECT_MODEL` to the same base model `fast-think`/pass-through resolves to, so the only difference is the injected `num_ctx`. **Caveat worth noting in the baseline:** on a backend with `OLLAMA_NUM_PARALLEL>1` the injected `num_ctx` is silently halved (#33), so the "after" numbers depend on the backend's parallelism - note the backend's `NUM_PARALLEL` alongside the result. Re-dispatching an engineer to land the stranded commit, record the baseline, and close this out. (Also: earlier salvage branch `ward-salvage/agent-proxy-65e1e88a` held a superseded earlier durable-harness attempt - `34d08f5` replaced it. Redundant, safe to prune.)

coilysiren closed this issue

2026-07-04 10:36:31 +00:00

Rows
Columns

Finish leg-05 reliability harness and record the M2 baseline-to-after result #19

⚠️ Reopened: this run's work did not land on main

Reap diagnostics

Recover

Working tree at reap time

⚠️ Reopened: this run's work did not land on `main`