coilyco-flight-deck/ward

Watch

Inspect failed release run 2495 after AWS reference removal landed #1597

New issue

Closed

opened 2026-07-28 09:20:04 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-28 09:20:04 +00:00

Member

Problem

After coilyco-flight-deck/ward#1478 landed on main at commit 29c19a7e, the promote workflow succeeded but the follow-up release workflow failed.

Observed from the read-only burndown surface on 2026-07-28:

run 12467 (#2495) failure  release.yml push release "Merge remote-tracking branch 'origin/main' into issue-1478"
run 12465 (#2494) success  promote.yml push main "Merge remote-tracking branch 'origin/main' into issue-1478"
run 12463 (#2493) failure  release.yml push release "Fix promote release ref push"
run 12462 (#2492) success  promote.yml push main "Fix promote release ref push"

This appears adjacent to the recent release/promotion recovery work, but it is separate from the earlier promote failures tracked by coilyco-flight-deck/ward#1595 and coilyco-flight-deck/ward#1596.

Desired outcome

Diagnose run #2495 and determine whether the failure is the same release-path problem as #2493, a regression from coilyco-flight-deck/ward#1478, or an external runner/release condition.

Acceptance

Record the exact first actionable failure from run #2495.
Cross-check whether it shares a root cause with #2493 or the prior promote failures #2490/#2491.
If local repository evidence proves a code/config fix is needed, implement it and land through the normal Ward workflow.
If the failure is live-runner or external-state only, leave a clear operator checkpoint with the exact run URL, local proof state, and requested verification.
Do not repeatedly push to probe live CI. One locally grounded corrective push is acceptable only if the repo evidence proves it.
Cross-link coilyco-flight-deck/ward#1478 in the final note.

Context

This was surfaced during the autonomous Ward burndown while prioritizing UX-safe work. Keep the recovery conservative because the run is explicitly meant to avoid stability-risky changes.

## Problem After coilyco-flight-deck/ward#1478 landed on main at commit 29c19a7e, the promote workflow succeeded but the follow-up release workflow failed. Observed from the read-only burndown surface on 2026-07-28: ```text run 12467 (#2495) failure release.yml push release "Merge remote-tracking branch 'origin/main' into issue-1478" run 12465 (#2494) success promote.yml push main "Merge remote-tracking branch 'origin/main' into issue-1478" run 12463 (#2493) failure release.yml push release "Fix promote release ref push" run 12462 (#2492) success promote.yml push main "Fix promote release ref push" ``` This appears adjacent to the recent release/promotion recovery work, but it is separate from the earlier promote failures tracked by coilyco-flight-deck/ward#1595 and coilyco-flight-deck/ward#1596. ## Desired outcome Diagnose run #2495 and determine whether the failure is the same release-path problem as #2493, a regression from coilyco-flight-deck/ward#1478, or an external runner/release condition. ## Acceptance - Record the exact first actionable failure from run #2495. - Cross-check whether it shares a root cause with #2493 or the prior promote failures #2490/#2491. - If local repository evidence proves a code/config fix is needed, implement it and land through the normal Ward workflow. - If the failure is live-runner or external-state only, leave a clear operator checkpoint with the exact run URL, local proof state, and requested verification. - Do not repeatedly push to probe live CI. One locally grounded corrective push is acceptable only if the repo evidence proves it. - Cross-link coilyco-flight-deck/ward#1478 in the final note. ## Context This was surfaced during the autonomous Ward burndown while prioritizing UX-safe work. Keep the recovery conservative because the run is explicitly meant to avoid stability-risky changes.

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-28 09:20:04 +00:00

coilyco-ops referenced this issue

2026-07-28 09:23:39 +00:00

Recover full host filesystem causing pre-agent Docker resource deaths #1603

coilyco-ops referenced this issue

2026-07-28 09:32:29 +00:00

Inspect failed release run 2497 after promote-run diagnosis landed #1604

coilyco-ops referenced this issue from a commit

2026-07-28 09:35:48 +00:00

Fix release draft asset checkout

coilyco-ops closed this issue

2026-07-28 09:35:48 +00:00

coilyco-ops referenced this issue

2026-07-28 09:40:28 +00:00

Recover full host filesystem causing pre-agent Docker resource deaths #1603

coilyco-ops commented

2026-07-28 09:40:35 +00:00

Author

Member

WARD-WORKFLOW: done ✅

details

workflow: merge-remote-main; review summary: in-container review gate skipped because the temporary Ward default is pending brokered QA.

Landed fix on main: promote-draft-assets in release.yml now checks out the repo before invoking repo-local scripts, with a release contract test and docs/release-run-2495.md checkpoint.

Run #2495 first actionable failure: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/actions/runs/2495 failed in promote-draft-assets / Fetch the draft assets staged on main with: bash: scripts/promote-draft-assets.sh: No such file or directory.

Cross-check: #2493 has the same missing-checkout release-path root cause. #2490/#2491 were the earlier promote refspec failures, not this one. The failure happened before any AWS, SSM, registry, tap, or Scoop credential path, so it was not caused by ward#1478 cloud credential removal.

Implementation felt straightforward once the log endpoint was found; the main surprise was the HTML job page exposing logs while the expected API log path returned 404. Confidence: high. Follow-up: #2500 superseded this push's promote run and contains this fix, so operators can watch that run rather than rerunning #2495.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> workflow: merge-remote-main; review summary: in-container review gate skipped because the temporary Ward default is pending brokered QA. Landed fix on main: promote-draft-assets in release.yml now checks out the repo before invoking repo-local scripts, with a release contract test and docs/release-run-2495.md checkpoint. Run #2495 first actionable failure: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/actions/runs/2495 failed in promote-draft-assets / Fetch the draft assets staged on main with: bash: scripts/promote-draft-assets.sh: No such file or directory. Cross-check: #2493 has the same missing-checkout release-path root cause. #2490/#2491 were the earlier promote refspec failures, not this one. The failure happened before any AWS, SSM, registry, tap, or Scoop credential path, so it was not caused by ward#1478 cloud credential removal. Implementation felt straightforward once the log endpoint was found; the main surprise was the HTML job page exposing logs while the expected API log path returned 404. Confidence: high. Follow-up: #2500 superseded this push's promote run and contains this fix, so operators can watch that run rather than rerunning #2495. </details>

coilyco-ops commented

2026-07-28 09:40:41 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-ward-1597 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-ward-1597` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-28 09:48:13 +00:00

Diagnose release run 2497

coilyco-ops referenced this issue

2026-07-28 09:52:34 +00:00

Inspect failed release run 2497 after promote-run diagnosis landed #1604