Inspect failed release run 2501 after role identity launch landed

coilyco-ops commented

2026-07-28 09:46:58 +00:00

Member

Problem

After coilyco-flight-deck/ward#1465 landed the role identity launch change, the promote workflow succeeded but the follow-up release workflow failed.

Observed from the read-only burndown surface on 2026-07-28:

run 12488 (#2501) failure  release.yml push release "merge: remote main before issue-1465 landing"
run 12484 (#2500) success  promote.yml push main "merge: remote main before issue-1465 landing"
run 12469 (#2497) failure  release.yml push release "docs: record promote run 2491 diagnosis"

This is adjacent to coilyco-flight-deck/ward#1604, which is already investigating release run #2497. Capture #2501 separately so this later failure is not lost if it has a distinct first actionable failure.

Desired outcome

Diagnose run #2501 and determine whether it shares the same root cause as #2497/#1604, #2495/#1597, or an external release runner condition.

Acceptance

Record the exact first actionable failure from run #2501.
Cross-check whether it shares a root cause with #2497/#1604 and #2495/#1597.
If it is already covered by an existing landed fix or active issue, comment with proof and close as duplicate/covered.
If local repository evidence proves a code/config fix is needed, implement it and land through the normal Ward workflow.
If the failure is live-runner or external-state only, leave a clear operator checkpoint with the exact run URL, local proof state, and requested verification.
Do not repeatedly push to probe live CI.
Cross-link coilyco-flight-deck/ward#1465 in the final note.

Context

This was surfaced during the autonomous Ward burndown while prioritizing UX-safe work. Keep recovery conservative and coordinate with coilyco-flight-deck/ward#1604 rather than duplicating risky release changes.

## Problem After coilyco-flight-deck/ward#1465 landed the role identity launch change, the promote workflow succeeded but the follow-up release workflow failed. Observed from the read-only burndown surface on 2026-07-28: ```text run 12488 (#2501) failure release.yml push release "merge: remote main before issue-1465 landing" run 12484 (#2500) success promote.yml push main "merge: remote main before issue-1465 landing" run 12469 (#2497) failure release.yml push release "docs: record promote run 2491 diagnosis" ``` This is adjacent to coilyco-flight-deck/ward#1604, which is already investigating release run #2497. Capture #2501 separately so this later failure is not lost if it has a distinct first actionable failure. ## Desired outcome Diagnose run #2501 and determine whether it shares the same root cause as #2497/#1604, #2495/#1597, or an external release runner condition. ## Acceptance - Record the exact first actionable failure from run #2501. - Cross-check whether it shares a root cause with #2497/#1604 and #2495/#1597. - If it is already covered by an existing landed fix or active issue, comment with proof and close as duplicate/covered. - If local repository evidence proves a code/config fix is needed, implement it and land through the normal Ward workflow. - If the failure is live-runner or external-state only, leave a clear operator checkpoint with the exact run URL, local proof state, and requested verification. - Do not repeatedly push to probe live CI. - Cross-link coilyco-flight-deck/ward#1465 in the final note. ## Context This was surfaced during the autonomous Ward burndown while prioritizing UX-safe work. Keep recovery conservative and coordinate with coilyco-flight-deck/ward#1604 rather than duplicating risky release changes.

coilyco-ops added the

labels

2026-07-28 09:46:58 +00:00

coilyco-ops referenced this issue from a commit

2026-07-28 10:03:32 +00:00

Diagnose release run 2501

coilyco-ops commented

2026-07-28 10:09:55 +00:00

Author

Member

WARD-WORKFLOW: done ✅

details

workflow: merge-remote-main; review summary: skipped - the in-container review gate was intentionally skipped because the temporary ward default pending brokered QA.

Recorded run #2501: first actionable failure was scripts/verify-release-assets.sh: line 5: RELEASE_TAG: missing RELEASE_TAG after checkout, draft fetch, and stable asset upload. This is not the #2495/#2497 missing-checkout root cause; it was already covered by the #1604 landed RELEASE_TAG handoff fix, with #2505 proving that boundary green. Cross-link: coilyco-flight-deck/ward#1465.

Felt: bounded and evidence-driven. Confidence: high. Surprise: aosguard could not resolve SSM creds here, so Forgejo API token verification was used. Follow-ups: none.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> workflow: merge-remote-main; review summary: skipped - the in-container review gate was intentionally skipped because the temporary ward default pending brokered QA. Recorded run #2501: first actionable failure was `scripts/verify-release-assets.sh: line 5: RELEASE_TAG: missing RELEASE_TAG` after checkout, draft fetch, and stable asset upload. This is not the #2495/#2497 missing-checkout root cause; it was already covered by the #1604 landed RELEASE_TAG handoff fix, with #2505 proving that boundary green. Cross-link: coilyco-flight-deck/ward#1465. Felt: bounded and evidence-driven. Confidence: high. Surprise: `aosguard` could not resolve SSM creds here, so Forgejo API token verification was used. Follow-ups: none. </details>

coilyco-ops commented

2026-07-28 10:10:06 +00:00

Author

Member

WARD-WORKFLOW: blocked 🛑

salvage details

An ephemeral ward container (codex mode) dispatched for this issue finished but its work was not merged to main, so cleanup preserved it on a branch before teardown and reopened the issue (a closing reference for #1605 never reached main). Recover from the salvage branch below.

Repo: coilyco-flight-deck/ward
Salvage branch: ward-salvage/ward-86b72dcf
Pull request: not opened - salvage branch has no diff against main
Reason: missing same-repo closing reference
Container uptime at reap: 13m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token)

Cleanup diagnostics

--- reap diagnostics ---
ward version:      v0.823.0
version source:    pinned via WARD_VERSION/--ward-version (v0.823.0)
HEAD:              626969b2fdb6
origin/main:       b44e0b82f3a2
ancestry:          HEAD is ALREADY on origin/main - a salvage here is a FALSE salvage (ward#504 signature)
decision gate:     missing same-repo closing reference on already-landed merge-remote-main run
reason:            missing same-repo closing reference
provenance:        present
run-owned landed:  no
working tree:      clean
container uptime:  13m (baked Forgejo PAT age proxy)
--- end reap diagnostics ---

Recover

git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/ward.git ward-salvage/ward-86b72dcf
git checkout -b ward-salvage/ward-86b72dcf FETCH_HEAD

This salvage was blocked by a missing closing reference. To recover, amend or cherry-pick the salvaged work so the landing commit message includes closes #1605, or add a small empty trailer commit with closes #1605, then land the branch.

— Codex, via ward agent

WARD-WORKFLOW: blocked 🛑 <details><summary>salvage details</summary> An ephemeral `ward container` (codex mode) dispatched for this issue finished but its work was **not merged to `main`**, so cleanup preserved it on a branch before teardown and reopened the issue (a closing reference for #1605 never reached `main`). Recover from the salvage branch below. - **Repo:** `coilyco-flight-deck/ward` - **Salvage branch:** `ward-salvage/ward-86b72dcf` - **Pull request:** not opened - salvage branch has no diff against main - **Reason:** missing same-repo closing reference - **Container uptime at reap:** 13m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token) ## Cleanup diagnostics ``` --- reap diagnostics --- ward version: v0.823.0 version source: pinned via WARD_VERSION/--ward-version (v0.823.0) HEAD: 626969b2fdb6 origin/main: b44e0b82f3a2 ancestry: HEAD is ALREADY on origin/main - a salvage here is a FALSE salvage (ward#504 signature) decision gate: missing same-repo closing reference on already-landed merge-remote-main run reason: missing same-repo closing reference provenance: present run-owned landed: no working tree: clean container uptime: 13m (baked Forgejo PAT age proxy) --- end reap diagnostics --- ``` ## Recover ```bash git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/ward.git ward-salvage/ward-86b72dcf git checkout -b ward-salvage/ward-86b72dcf FETCH_HEAD ``` This salvage was blocked by a missing closing reference. To recover, amend or cherry-pick the salvaged work so the landing commit message includes `closes #1605`, or add a small empty trailer commit with `closes #1605`, then land the branch. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-28 10:10:06 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: blocked 🛑.

ward container reap released container engineer-codex-ward-1605 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: blocked 🛑`. `ward container reap` released container `engineer-codex-ward-1605` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-28 10:12:15 +00:00

Do not reopen completed runs when reaper misses an already-pushed closing reference #1606

coilyco-ops commented

2026-07-28 10:12:32 +00:00

Author

Member

Director cleanup: closing this manually because the run completed despite the later salvage misclassification. Proof: final engineer log reports push to main at 626969b2 with commit a7cef736 carrying closes #1605; Forgejo promote #2507 succeeded; Forgejo release #2508 succeeded; the salvage branch had no diff against main. Follow-up reaper bug captured as coilyco-flight-deck/ward#1606.

Director cleanup: closing this manually because the run completed despite the later salvage misclassification. Proof: final engineer log reports push to main at 626969b2 with commit a7cef736 carrying closes #1605; Forgejo promote #2507 succeeded; Forgejo release #2508 succeeded; the salvage branch had no diff against main. Follow-up reaper bug captured as coilyco-flight-deck/ward#1606.

coilyco-ops closed this issue

2026-07-28 10:12:39 +00:00

coilyco-ops referenced this issue

2026-08-05 17:57:19 +00:00

Never salvage or reopen work already landed on main #1598

coilyco-ops referenced this issue

2026-08-05 17:57:30 +00:00

Never salvage or reopen work already landed on main #1598

coilyco-ops referenced this issue

2026-08-06 02:50:52 +00:00

Never salvage or reopen work already landed on main #1598