Make dispatch overrides recover stale prelaunch blockers #1196

Closed

opened 2026-07-13 21:10:15 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-13 21:10:15 +00:00

Owner

Make dispatch override flags bypass stale prelaunch capacity blockers.

Problem:

The current override flags do not override enough to recover from the failure mode they are most needed for.
--override-reservation bypasses the per-issue reservation path, but dispatch still fails when repo capacity is filled by stale container starting records.
--override-capacity is documented as global/OOM pool bypass, not as a repo prelaunch-stale reclaim path.
The result is an operator hard stop even when ward itself reports no meaningful running backpressure.

Concrete live state from 2026-07-13:

ward agent list showed global capacity open: 9/12 active, 3 slots free.
coilyco-flight-deck/ward was blocked by repo capacity because #1191 and #1188 were container starting, while #1195 was the only actual running ward engineer.
coilyco-flight-deck/agentic-os was similarly blocked by expired container starting reservations.
ward agent dispatch-health --repo coilyco-flight-deck/ward,coilyco-flight-deck/agentic-os --json reported backpressure: false, runaway: false, and running: 1.
Despite that, dispatching #1194 with --override-reservation failed on repo engineer limit is reached.

Required behavior:

An override intended for stale dispatch blockers must bypass or reclaim stale prelaunch capacity records, including repo-level capacity records.
If --override-reservation is not meant to do that, add a clearly named override such as --override-stale-prelaunch or make --override-capacity cover stale prelaunch repo slots when the record is not a visible/running engineer.
The override must not blindly exceed real running capacity. It should distinguish stale/non-visible prelaunch holds from actual running engineers.
When dispatch-health says there is no backpressure/runaway and only stale starting records block a repo, an override dispatch should proceed or clear the stale holds first.
Error messages should tell the operator which override can recover, not suggest waiting for a run that is not actually running.

Relationship to #1191:

#1191 is the larger architectural reflow so prelaunch launch intents stop consuming running engineer capacity.
This ticket is the operator recovery path: while stale prelaunch records exist, the override flags must be able to get work moving without manual issue-comment deletion.

Acceptance:

Tests cover repo capacity filled by stale container starting records and show the selected override succeeds.
Tests prove the override still refuses to exceed real visible/running engineer capacity unless the explicit OOM-risk capacity override is used.
Dispatch errors for repo limit include stale-prelaunch counts separately from running counts.
Docs/help text explain which override recovers stale prelaunch blockers.

Make dispatch override flags bypass stale prelaunch capacity blockers. Problem: - The current override flags do not override enough to recover from the failure mode they are most needed for. - `--override-reservation` bypasses the per-issue reservation path, but dispatch still fails when repo capacity is filled by stale `container starting` records. - `--override-capacity` is documented as global/OOM pool bypass, not as a repo prelaunch-stale reclaim path. - The result is an operator hard stop even when ward itself reports no meaningful running backpressure. Concrete live state from 2026-07-13: - `ward agent list` showed global capacity open: 9/12 active, 3 slots free. - `coilyco-flight-deck/ward` was blocked by repo capacity because #1191 and #1188 were `container starting`, while #1195 was the only actual running ward engineer. - `coilyco-flight-deck/agentic-os` was similarly blocked by expired `container starting` reservations. - `ward agent dispatch-health --repo coilyco-flight-deck/ward,coilyco-flight-deck/agentic-os --json` reported `backpressure: false`, `runaway: false`, and `running: 1`. - Despite that, dispatching #1194 with `--override-reservation` failed on `repo engineer limit is reached`. Required behavior: - An override intended for stale dispatch blockers must bypass or reclaim stale prelaunch capacity records, including repo-level capacity records. - If `--override-reservation` is not meant to do that, add a clearly named override such as `--override-stale-prelaunch` or make `--override-capacity` cover stale prelaunch repo slots when the record is not a visible/running engineer. - The override must not blindly exceed real running capacity. It should distinguish stale/non-visible prelaunch holds from actual running engineers. - When dispatch-health says there is no backpressure/runaway and only stale starting records block a repo, an override dispatch should proceed or clear the stale holds first. - Error messages should tell the operator which override can recover, not suggest waiting for a run that is not actually running. Relationship to #1191: - #1191 is the larger architectural reflow so prelaunch launch intents stop consuming running engineer capacity. - This ticket is the operator recovery path: while stale prelaunch records exist, the override flags must be able to get work moving without manual issue-comment deletion. Acceptance: - Tests cover repo capacity filled by stale `container starting` records and show the selected override succeeds. - Tests prove the override still refuses to exceed real visible/running engineer capacity unless the explicit OOM-risk capacity override is used. - Dispatch errors for repo limit include stale-prelaunch counts separately from running counts. - Docs/help text explain which override recovers stale prelaunch blockers.

coilyco-ops commented

2026-07-13 21:10:48 +00:00

Author

Owner

Dispatch proof for this bug: after filing #1196, normal dispatch and --override-reservation dispatch both failed with repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3). Current list has only #1195 as a running ward engineer; #1191 and #1188 are non-visible container starting reservations. This issue is ready to dispatch once a ward slot clears or after manual stale reservation cleanup.

Dispatch proof for this bug: after filing #1196, normal dispatch and `--override-reservation` dispatch both failed with `repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3)`. Current list has only #1195 as a running ward engineer; #1191 and #1188 are non-visible `container starting` reservations. This issue is ready to dispatch once a ward slot clears or after manual stale reservation cleanup.

coilyco-ops referenced this issue

2026-07-13 21:21:18 +00:00

Document manual stale reservation cleanup #1198

coilyco-ops referenced this issue

2026-07-13 21:47:02 +00:00

Make stop --print honest for ghost launch records #1200

coilyco-ops commented

2026-07-13 22:03:18 +00:00

Author

Owner

Additional live evidence from the 2026-07-13 director train: after #506 merged, agentic-os PRs #507 and #508 needed conflict repair. The original #503/#504/#505 runs had all posted WARD-OUTCOME: submitted; #503 was closed/merged. The director deleted the targeted terminal reservation comments 32421, 32422, and 32423. After that, ward agent list still counted all three as phase: failed before container start, and fresh repair dispatches for agentic-os#504 and #505 with both --override-reservation --override-capacity failed on repo engineer limit is reached for coilyco-flight-deck/agentic-os: 3 active engineer(s) (limit 3). This confirms the override gap is not only active reservation comments; broker-side terminal launch records can keep consuming repo capacity after the durable comments are gone.

Additional live evidence from the 2026-07-13 director train: after #506 merged, agentic-os PRs #507 and #508 needed conflict repair. The original #503/#504/#505 runs had all posted `WARD-OUTCOME: submitted`; #503 was closed/merged. The director deleted the targeted terminal reservation comments 32421, 32422, and 32423. After that, `ward agent list` still counted all three as `phase: failed before container start`, and fresh repair dispatches for agentic-os#504 and #505 with both `--override-reservation --override-capacity` failed on `repo engineer limit is reached for coilyco-flight-deck/agentic-os: 3 active engineer(s) (limit 3)`. This confirms the override gap is not only active reservation comments; broker-side terminal launch records can keep consuming repo capacity after the durable comments are gone.

coilyco-ops referenced this issue from a commit

2026-07-14 00:28:48 +00:00

fix: recover stale prelaunch repo blockers

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-14 00:29:19 +00:00

Make dispatch overrides recover stale prelaunch blockers #1222