coilyco-flight-deck/ward

Watch

Recover created-but-never-started engineers without no-work salvage #1609

New issue

Closed

opened 2026-07-28 10:57:23 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-28 10:57:23 +00:00

Member

During the headless burndown on 2026-07-28, two fresh Codex engineer dispatches were accepted by the broker but did not reach a running engineer:

coilyco-flight-deck/ward#1608: first attempt left engineer-codex-ward-1608 in Docker Created state with no readable docker logs and no live transcript. A retry archived immediately as outcome=preserved-salvage-branch after the launch prompt, with no issue comment and no assistant work.
coilyco-flight-deck/ward#1570: accepted by broker, then Ward reported cleanup-needed / container starting while Docker showed engineer-codex-ward-1570 as Created and not running.

The director-visible evidence for #1570:

count=2 cleanup=12 partial=0
coilyco-flight-deck/ward#1570 cleanup-needed container starting ...
NAMES                      STATUS    SIZE
engineer-codex-ward-1570   Created   338MB (virtual 5.33GB)

For #1608, ward agent logs selected the archive path and ended with:

WARD-RUN-SUMMARY: outcome=preserved-salvage-branch meta=meta.json transcript=/root/.ward/agent-logs/engineer-codex-ward-1608/transcript.jsonl

This is operator-hostile: the issue remains open, the lane appears to have cleanup-needed work, and there is no clear reason in the issue thread. Ward should classify and explain pre-agent container start failures instead of turning them into no-diff/no-work salvage.

Acceptance:

A container that never starts is reported as a launch/pre-agent failure with the Docker state/error and the broker artifact path.
Ward releases the reservation and removes or clearly marks the created container so capacity accounting is not ambiguous.
The issue gets a concise WARD-WORKFLOW: failed or blocked comment when a dispatch cannot enter the harness.
No salvage branch is created or advertised when there is no agent worktree delta and the harness never performed work.

Labels: headless, P1

During the headless burndown on 2026-07-28, two fresh Codex engineer dispatches were accepted by the broker but did not reach a running engineer: - `coilyco-flight-deck/ward#1608`: first attempt left `engineer-codex-ward-1608` in Docker `Created` state with no readable docker logs and no live transcript. A retry archived immediately as `outcome=preserved-salvage-branch` after the launch prompt, with no issue comment and no assistant work. - `coilyco-flight-deck/ward#1570`: accepted by broker, then Ward reported `cleanup-needed` / `container starting` while Docker showed `engineer-codex-ward-1570` as `Created` and not running. The director-visible evidence for #1570: ``` count=2 cleanup=12 partial=0 coilyco-flight-deck/ward#1570 cleanup-needed container starting ... NAMES STATUS SIZE engineer-codex-ward-1570 Created 338MB (virtual 5.33GB) ``` For #1608, `ward agent logs` selected the archive path and ended with: ``` WARD-RUN-SUMMARY: outcome=preserved-salvage-branch meta=meta.json transcript=/root/.ward/agent-logs/engineer-codex-ward-1608/transcript.jsonl ``` This is operator-hostile: the issue remains open, the lane appears to have cleanup-needed work, and there is no clear reason in the issue thread. Ward should classify and explain pre-agent container start failures instead of turning them into no-diff/no-work salvage. Acceptance: - A container that never starts is reported as a launch/pre-agent failure with the Docker state/error and the broker artifact path. - Ward releases the reservation and removes or clearly marks the created container so capacity accounting is not ambiguous. - The issue gets a concise `WARD-WORKFLOW: failed` or `blocked` comment when a dispatch cannot enter the harness. - No salvage branch is created or advertised when there is no agent worktree delta and the harness never performed work. Labels: headless, P1

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-28 10:57:29 +00:00

coilyco-ops commented

2026-07-28 11:00:13 +00:00

Author

Member

Self-dispatch reproduced this immediately.

After warded engineer coilyco-flight-deck/ward#1609 --harness codex was accepted:

count=2 cleanup=12 partial=0
coilyco-flight-deck/ward#1609 cleanup-needed container starting ...
NAMES                      STATUS    SIZE
engineer-codex-ward-1609   Created   2.81GB (virtual 7.81GB)

ward agent logs coilyco-flight-deck/ward#1609 --tail 40 reported no readable docker bytes and an empty live transcript tree. This is not capacity-related: only #1501 and #1551 were running.

Self-dispatch reproduced this immediately. After `warded engineer coilyco-flight-deck/ward#1609 --harness codex` was accepted: ``` count=2 cleanup=12 partial=0 coilyco-flight-deck/ward#1609 cleanup-needed container starting ... NAMES STATUS SIZE engineer-codex-ward-1609 Created 2.81GB (virtual 7.81GB) ``` `ward agent logs coilyco-flight-deck/ward#1609 --tail 40` reported no readable docker bytes and an empty live transcript tree. This is not capacity-related: only `#1501` and `#1551` were running.

coilyco-ops commented

2026-07-28 11:00:36 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run never started. ward container reap released container engineer-codex-ward-1609 (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run never started. `ward container reap` released container `engineer-codex-ward-1609` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-28 11:26:15 +00:00

Report pre-agent container start failures

coilyco-ops closed this issue

2026-07-28 11:26:18 +00:00

coilyco-ops commented

2026-07-28 11:36:31 +00:00

Author

Member

WARD-WORKFLOW: done ✅

details

Implemented and pushed 8ec2843e to main for coilyco-flight-deck/ward#1609.

workflow: merge-remote-main; review summary: skipped: the in-container review gate was intentionally skipped because the temporary ward default is pending brokered QA.

Validation: ward exec test, ward exec build, ward exec vet, ward exec lint, and commit hooks passed. Post-push verification: aosguard ops forgejo tasks list could not resolve its SSM/IMDS token in this container; the public Forgejo API showed promote.yml run 2525 for 8ec2843e reached failure, with no same-commit release run created.

Implementation felt direct once the silent broker failure path was found: the old tests explicitly pinned deleting the reservation without posting a failure, which was the bad operator experience. Confidence: medium-high on the launch classification and created-container cleanup; lower on release health because the post-push promote run failed outside the code path changed here.

Surprises: commentFailedDispatch already had the name of the desired behavior but intentionally left no issue comment. Follow-up: inspect Forgejo run 2525 failure and decide whether deferred dispatch comments should also post a visible replacement instead of only deleting stale markers.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> Implemented and pushed `8ec2843e` to `main` for `coilyco-flight-deck/ward#1609`. workflow: merge-remote-main; review summary: skipped: the in-container review gate was intentionally skipped because the temporary ward default is pending brokered QA. Validation: `ward exec test`, `ward exec build`, `ward exec vet`, `ward exec lint`, and commit hooks passed. Post-push verification: `aosguard ops forgejo tasks list` could not resolve its SSM/IMDS token in this container; the public Forgejo API showed `promote.yml` run 2525 for `8ec2843e` reached `failure`, with no same-commit release run created. Implementation felt direct once the silent broker failure path was found: the old tests explicitly pinned deleting the reservation without posting a failure, which was the bad operator experience. Confidence: medium-high on the launch classification and created-container cleanup; lower on release health because the post-push promote run failed outside the code path changed here. Surprises: `commentFailedDispatch` already had the name of the desired behavior but intentionally left no issue comment. Follow-up: inspect Forgejo run 2525 failure and decide whether deferred dispatch comments should also post a visible replacement instead of only deleting stale markers. </details>

coilyco-ops commented

2026-07-28 11:36:40 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-ward-1609 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-ward-1609` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#1609

No description provided.

Rows
Columns