coilyco-flight-deck/ward

Fork 0

Dispatch health and agent list disagree on active engineers, leaving stale starting records at capacity #1174

New issue

Closed

opened 2026-07-13 04:24:26 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-13 04:24:26 +00:00

Member

Problem

The director surface is getting contradictory broker state from ward. dispatch-health says no ward engineers are running, but ward agent list reports many active/reserved launches and says the pool is at capacity. The stale records include long-lived container starting entries and failed-before-start entries that cannot be stopped as real containers.

Observed on 2026-07-13:

ward agent dispatch-health --repo coilyco-flight-deck/ward --json reported running: 0, queued: 12, backpressure: false, runaway: false.
ward agent list at the same time reported active engineer launches (running + reserved) (25/12, at capacity).
The list included ward entries stuck in container starting for 35-60+ minutes, including issues that already produced PRs or merged/closed outcomes.
Earlier ward agent stop attempts against stale entries reported no matching running engineer container, so at least some records are ledger/broker residue rather than live containers.

Do

Make the broker state model converge:

Define one source of truth for active engineer count, capacity checks, dispatch-health, and ward agent list.
Reconcile or expire container starting records that never become visible containers within a bounded interval.
Ensure failed-before-start entries do not count against capacity or open-run pressure after their terminal state is recorded.
Add a cleanup path visible from the director surface for stale broker records that have no backing container.
Make dispatch-health surface stale-record counts separately instead of reporting healthy while agent list says at capacity.

Acceptance

ward agent list, capacity checks, and dispatch-health agree on live running/reserved counts or explicitly explain excluded categories.
A stale container starting record older than the startup timeout is automatically terminalized or marked cleanup-needed without counting as active capacity.
A director can diagnose and clear stale records without raw Docker/API access.
Tests cover a launch accepted by the broker but never visible as a container.

Related: ward#1016, ward#1041, ward#1077, ward#1166.

## Problem The director surface is getting contradictory broker state from ward. `dispatch-health` says no ward engineers are running, but `ward agent list` reports many active/reserved launches and says the pool is at capacity. The stale records include long-lived `container starting` entries and failed-before-start entries that cannot be stopped as real containers. Observed on 2026-07-13: * `ward agent dispatch-health --repo coilyco-flight-deck/ward --json` reported `running: 0`, `queued: 12`, `backpressure: false`, `runaway: false`. * `ward agent list` at the same time reported `active engineer launches (running + reserved) (25/12, at capacity)`. * The list included ward entries stuck in `container starting` for 35-60+ minutes, including issues that already produced PRs or merged/closed outcomes. * Earlier `ward agent stop` attempts against stale entries reported no matching running engineer container, so at least some records are ledger/broker residue rather than live containers. ## Do Make the broker state model converge: * Define one source of truth for active engineer count, capacity checks, dispatch-health, and `ward agent list`. * Reconcile or expire `container starting` records that never become visible containers within a bounded interval. * Ensure failed-before-start entries do not count against capacity or open-run pressure after their terminal state is recorded. * Add a cleanup path visible from the director surface for stale broker records that have no backing container. * Make `dispatch-health` surface stale-record counts separately instead of reporting healthy while `agent list` says at capacity. ## Acceptance * `ward agent list`, capacity checks, and `dispatch-health` agree on live running/reserved counts or explicitly explain excluded categories. * A stale `container starting` record older than the startup timeout is automatically terminalized or marked cleanup-needed without counting as active capacity. * A director can diagnose and clear stale records without raw Docker/API access. * Tests cover a launch accepted by the broker but never visible as a container. Related: ward#1016, ward#1041, ward#1077, ward#1166.

coilyco-ops commented

2026-07-13 09:24:53 +00:00

Author

Member

Findings from the burn-down session, reproduced live on Kai's desktop tower right after the PR jam cleared.

ward agent list reports 3 active engineer launches (3/12), and none of them is a real container:

reserved-engineer-03 - ref ward#883, host test-host, status reserved, phase "broker accepted / queued". A host named test-host in the production tracker is test-suite pollution, the same isolation failure class the fix/test-env-isolation branch targets.
engineer-codex-ward-1033 - host director-box, phase "failed before container start", status failed, yet it still holds an active slot 17 minutes later.
a bare (reserved) record - ref ward#184, no harness, no host, budget expired by 3292s against the 90m limit, still counted as reserved after 2h24m.

ward agent dispatch-health simultaneously reports running=3 (counting those three phantoms) and flags 12 other reservations as "safe-to-redispatch (reservation stale)". So the agent list and the health view agree on the count but disagree with reality, and at capacity these phantom records are exactly what refuses new dispatch.

Suggested shape for the fix: a failed-before-start record should release its slot immediately, an expired-budget reservation should never count toward capacity, and the tracker should refuse writes from hosts outside the fleet roster (which also fences off test pollution). The session did not delete the records by hand - the documented ward#570 comment-cleanup path was unavailable in the moment and the records are better evidence live.

Findings from the burn-down session, reproduced live on Kai's desktop tower right after the PR jam cleared. `ward agent list` reports 3 active engineer launches (3/12), and none of them is a real container: * `reserved-engineer-03` - ref ward#883, host `test-host`, status reserved, phase "broker accepted / queued". A host named test-host in the production tracker is test-suite pollution, the same isolation failure class the fix/test-env-isolation branch targets. * `engineer-codex-ward-1033` - host director-box, phase "failed before container start", status failed, yet it still holds an active slot 17 minutes later. * a bare `(reserved)` record - ref ward#184, no harness, no host, budget expired by 3292s against the 90m limit, still counted as reserved after 2h24m. `ward agent dispatch-health` simultaneously reports `running=3` (counting those three phantoms) and flags 12 other reservations as "safe-to-redispatch (reservation stale)". So the agent list and the health view agree on the count but disagree with reality, and at capacity these phantom records are exactly what refuses new dispatch. Suggested shape for the fix: a failed-before-start record should release its slot immediately, an expired-budget reservation should never count toward capacity, and the tracker should refuse writes from hosts outside the fleet roster (which also fences off test pollution). The session did not delete the records by hand - the documented ward#570 comment-cleanup path was unavailable in the moment and the records are better evidence live.

coilyco-ops added the

headless

labels

2026-07-14 06:50:05 +00:00

coilyco-ops commented

2026-07-14 07:19:18 +00:00

Author

Member

Additional director-surface evidence from the 2026-07-14 headless burn-down heartbeat: ward agent dispatch-health --repo coilyco-flight-deck/ward --json returned quickly with running=3, held=3, backpressure=false, but timeout 20s ward agent list --json produced no output and exited 124. This is the same operator-facing convergence gap: the fast health path is usable, while the inventory path can hang long enough that the director cannot use it as a reliable capacity source.

Additional director-surface evidence from the 2026-07-14 headless burn-down heartbeat: `ward agent dispatch-health --repo coilyco-flight-deck/ward --json` returned quickly with `running=3`, `held=3`, `backpressure=false`, but `timeout 20s ward agent list --json` produced no output and exited 124. This is the same operator-facing convergence gap: the fast health path is usable, while the inventory path can hang long enough that the director cannot use it as a reliable capacity source.

coilyco-ops commented

2026-07-14 10:29:07 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1174 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-14T10:29:06Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1174 · branch issue-1174 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1174 · ward v0.679.0 · dispatched 2026-07-14T10:29:02Z
Comment thread: 2 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-13T09:24:53Z), @coilyco-ops (2026-07-14T07:19:18Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.679.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1174` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-14T10:29:06Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1174` · branch `issue-1174` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1174` · ward `v0.679.0` · dispatched `2026-07-14T10:29:02Z` - **Comment thread:** 2 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-13T09:24:53Z), @coilyco-ops (2026-07-14T07:19:18Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.679.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-14 10:47:56 +00:00

Dispatch health and agent list disagree on active engineers, leaving stale starting records at capacity #1274

coilyco-ops closed this issue

2026-07-14 10:50:58 +00:00

coilyco-ops commented

2026-07-14 10:52:53 +00:00

Author

Member

WARDED_WORKFLOW: #1274

details

workflow: pull-request-and-merge; review summary: skipped by ~/.ward/config.yaml default

Shared active-inventory model is landed. Confidence high. Surprise: stale rows had to stay visible for diagnosis while capacity stopped counting them.

Follow-ups: none.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1274 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped by ~/.ward/config.yaml default Shared active-inventory model is landed. Confidence high. Surprise: stale rows had to stay visible for diagnosis while capacity stopped counting them. Follow-ups: none. </details>