coilyco-flight-deck/ward

Watch

Inspect failed promote run 2491 after bounded log reads landed #1596

New issue

Closed

opened 2026-07-28 09:07:46 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-07-28 09:07:46 +00:00

Member

Problem

After coilyco-flight-deck/ward#1594 landed the bounded fresh agent log reads change on main, the follow-up Forgejo promote run failed.

Observed from the read-only burndown surface on 2026-07-28:

run 12461 (#2491) failure  promote.yml push main "Merge remote-tracking branch 'origin/main' into issue-1594"\nrun 12459 (#2490) failure  promote.yml push main "feat(config): add runtime config drop command"\nrun 12457 (#2488) success  promote.yml push main "Merge remote-tracking branch 'origin/main'"\n```\n\nThis is adjacent to coilyco-flight-deck/ward#1595, which tracks the earlier #2490 promote failure after coilyco-flight-deck/ward#1464. Capture this separately so the #1594 landing failure is not lost.\n\n## Desired outcome\n\nDiagnose run #2491 and determine whether the failure is the same promote-path problem as #2490, a regression from #1594, or an external runner/release condition.\n\n## Acceptance\n\n- Record the exact first actionable failure from run #2491.\n- Cross-check whether it shares a root cause with #2490 / #1595.\n- If local repository evidence proves a code/config fix is needed, implement it and land through the normal Ward workflow.\n- If the failure is live-runner or external-state only, leave a clear operator checkpoint with the exact run URL, local proof state, and requested verification.\n- Do not repeatedly push to probe live CI. One locally grounded corrective push is acceptable only if the repo evidence proves it.\n- Cross-link coilyco-flight-deck/ward#1594 in the final note.\n\n## Context\n\nThis was surfaced during the autonomous Ward burndown while prioritizing UX-safe work. Keep the recovery conservative because the run is explicitly meant to avoid stability-risky changes.

## Problem After coilyco-flight-deck/ward#1594 landed the bounded fresh agent log reads change on main, the follow-up Forgejo promote run failed. Observed from the read-only burndown surface on 2026-07-28: ```text run 12461 (#2491) failure promote.yml push main "Merge remote-tracking branch 'origin/main' into issue-1594"\nrun 12459 (#2490) failure promote.yml push main "feat(config): add runtime config drop command"\nrun 12457 (#2488) success promote.yml push main "Merge remote-tracking branch 'origin/main'"\n```\n\nThis is adjacent to coilyco-flight-deck/ward#1595, which tracks the earlier #2490 promote failure after coilyco-flight-deck/ward#1464. Capture this separately so the #1594 landing failure is not lost.\n\n## Desired outcome\n\nDiagnose run #2491 and determine whether the failure is the same promote-path problem as #2490, a regression from #1594, or an external runner/release condition.\n\n## Acceptance\n\n- Record the exact first actionable failure from run #2491.\n- Cross-check whether it shares a root cause with #2490 / #1595.\n- If local repository evidence proves a code/config fix is needed, implement it and land through the normal Ward workflow.\n- If the failure is live-runner or external-state only, leave a clear operator checkpoint with the exact run URL, local proof state, and requested verification.\n- Do not repeatedly push to probe live CI. One locally grounded corrective push is acceptable only if the repo evidence proves it.\n- Cross-link coilyco-flight-deck/ward#1594 in the final note.\n\n## Context\n\nThis was surfaced during the autonomous Ward burndown while prioritizing UX-safe work. Keep the recovery conservative because the run is explicitly meant to avoid stability-risky changes.

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-28 09:07:46 +00:00

coilyco-ops commented

2026-07-28 09:13:00 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run never started. ward container reap released container engineer-codex-ward-1596 (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run never started. `ward container reap` released container `engineer-codex-ward-1596` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-28 09:20:04 +00:00

Inspect failed release run 2495 after AWS reference removal landed #1597

coilyco-ops referenced this issue from a commit

2026-07-28 09:21:23 +00:00

docs: record promote run 2491 diagnosis

coilyco-ops closed this issue

2026-07-28 09:21:23 +00:00

coilyco-ops commented

2026-07-28 09:21:55 +00:00

Author

Member

WARD-WORKFLOW: done ✅

details

workflow: merge-remote-main; review summary: skipped - the in-container review gate was intentionally skipped because the temporary ward default is pending brokered QA.

Diagnosed run https://forgejo.coilysiren.me/coilyco-flight-deck/ward/actions/runs/2491: first actionable failure was Git rejecting detached HEAD:release in Promote main to release (fast-forward only) because the destination was not a full refname; Git hinted HEAD:refs/heads/release.

Root cause matches #2490 / #1595, not a #1594 regression. Local proof: #2491 passed vet, test, lint, draft asset publish, and image alias publish before the refspec failure; #2492 succeeded after #1595 changed the refspec. Added docs/promote-run-2491.md and linked it from troubleshooting/docs index.

felt: conservative incident capture, mostly log archaeology. confidence: high. surprises: #1595 had already fixed the code path before this run started. follow-ups: let the normal promote run from this docs-only landing finish; no extra corrective push is indicated.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> workflow: merge-remote-main; review summary: skipped - the in-container review gate was intentionally skipped because the temporary ward default is pending brokered QA. Diagnosed run https://forgejo.coilysiren.me/coilyco-flight-deck/ward/actions/runs/2491: first actionable failure was Git rejecting detached `HEAD:release` in `Promote main to release (fast-forward only)` because the destination was not a full refname; Git hinted `HEAD:refs/heads/release`. Root cause matches #2490 / #1595, not a #1594 regression. Local proof: #2491 passed vet, test, lint, draft asset publish, and image alias publish before the refspec failure; #2492 succeeded after #1595 changed the refspec. Added `docs/promote-run-2491.md` and linked it from troubleshooting/docs index. felt: conservative incident capture, mostly log archaeology. confidence: high. surprises: #1595 had already fixed the code path before this run started. follow-ups: let the normal promote run from this docs-only landing finish; no extra corrective push is indicated. </details>

coilyco-ops commented

2026-07-28 09:22:04 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-ward-1596 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-ward-1596` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-28 09:32:29 +00:00

Inspect failed release run 2497 after promote-run diagnosis landed #1604

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#1596

No description provided.

Rows
Columns