coilyco-flight-deck/ward

Archived

Watch

Verify ward 0.483 director recovery behavior #779

Closed

opened 2026-07-09 07:56:09 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 07:56:09 +00:00

Owner

Goal

Use a fresh ward v0.483.0 director surface to verify the recovery behavior we have been trying to get back online. This issue is intentionally a verification handoff for the new director. Do not treat results from the old v0.474.0 read-only surface as proof.

Current saga

The recovery incident had several overlapping failures:

Old director surfaces kept launching child engineers on stale ward versions. A v0.474.0 director launched children that also recorded ward v0.474.0 even after v0.480.0 was published. Tracked in ward#700.
The old surface broker stripped important forwarded flags such as --workflow pull-requests-and-merge and --details, causing recovery dispatches to fall back to workflow pr. Earlier fixes for broker forwarding landed, but they must be verified from a fresh v0.483.0 director.
Workers in workflow pr posted WARD-OUTCOME: done after opening a PR. That is the wrong state model: a PR is submitted, not done, until the change reaches main. Tracked in ward#777.
We still need proof that PR-shaped workflows can create a PR, wait for CI, and merge or hand off correctly without claiming done too early. Tracked in ward#706.
--branch <name> exists, but old bootstrap only ran git checkout -B <branch> and did not start from origin/<branch> when resuming an existing PR branch. Tracked in ward#778.

Manual director recovery already merged these green PRs after checking CI:

#772 - remove review stats command, merged to main.
#774 - remove mcporter hydration, merged to main.
#775 - remove setup and doctor release surfaces, merged to main.

Latest known release before this issue was filed: v0.483.0, published 2026-07-09T07:46:51Z. Latest known main commit was 68e468764c03c309dae71d8c104f73dc5846484e from PR #775. Its main test, release, publish-binaries, publish-kdl-read, publish-kdl-write, bump-tap-formula, and bump-scoop-manifest jobs were green.

Open PRs at filing time were:

#773 for ward#735, test failed on run 1159.
#776 for ward#737, test failed on run 1162.

Fresh v0.480.0 workers were re-dispatched for ward#735 and ward#737 from the old surface, but their results should still be treated cautiously because the old surface was not the clean v0.483 director.

Verification mission

Perform verification from the fresh v0.483.0 director only. Record findings as comments on this issue with concrete links to reservation comments, PRs, action runs, and commits.

1. Director and child version propagation

Verify the new director surface is actually running ward v0.483.0. Then dispatch at least one child engineer without an explicit --ward-version pin and confirm the reservation comment records the current expected ward version, not an old surface bootstrap version such as v0.474.0.

Pass condition: an unpinned child from the v0.483 director does not inherit an older surface WARD_VERSION.

Fail condition: a child reservation records a stale ward version. If this fails, keep this issue open and point to ward#700.

2. Broker flag forwarding

Verify the v0.483 director forwards --workflow and --details intact through the dispatch broker. Use a low-risk canary or an existing appropriate issue, but make the verification marker unmistakable.

Required check:

--workflow pull-requests-and-merge
--details "WARD-RECOVERY-VERIFY-0483: broker forwarding canary"

Pass condition: the reservation/run context shows workflow pull-requests-and-merge, and the worker seed receives the details text.

Fail condition: the forwarded broker command or reservation falls back to workflow pr, or the details are missing. If this fails, keep this issue open and point back to the broker-forwarding fixes around ward#763.

3. PR outcome semantics

Verify that PR-shaped workers no longer mark the issue done merely because a PR was opened. This applies to both pr and pull-requests-and-merge.

Pass condition: a PR-only worker reports a non-done submitted or pending state, or the director records final done only after merge to main.

Fail condition: any worker posts WARD-OUTCOME: done while the PR is still open or before the relevant commit reaches main. If this fails, keep this issue open and point to ward#777.

4. PR workflow merge behavior

Verify the intended PR workflow behavior from end to end. The verification should prove whether ward can create a PR, wait for Forgejo Actions, avoid merging failed CI, and land only after green CI.

Pass condition: a PR-shaped workflow either lands after green CI or explicitly records a non-done waiting state that the director can merge later.

Fail condition: a worker reports done before CI, ignores failing CI, or requires manual web/API merge with no director-owned policy path. If this fails, keep this issue open and point to ward#706.

5. Existing branch resume

Verify --branch <existing-branch> starts from the remote branch when origin/<branch> exists. A good practical target is a failed PR branch such as issue-735 or issue-737, if those are still open when the new director sees this.

Pass condition: the worker starts on the remote PR branch content, not a fresh branch reset from main, and the log makes the branch provenance visible.

Fail condition: the worker starts from main while merely naming the branch. If this fails, keep this issue open and point to ward#778.

6. Current open PR cleanup

Re-check open PRs after the fresh workers finish. Merge only PRs whose PR test is green and whose branch is current enough to merge cleanly. Do not merge #773 or #776 unless their failing tests have been fixed and rerun green.

Reporting format

Post a concise final comment on this issue with:

WARD-RECOVERY-VERIFY-0483: pass or WARD-RECOVERY-VERIFY-0483: fail on the first line.
Evidence links for each verification area above.
A list of any issues that remain blocking.
Whether any PRs were merged during verification, and whether main CI stayed green afterward.

This issue is a verification controller, not a broad refactor. If a behavior is still broken, file or update the specific issue and stop short of claiming recovery complete.

Codex, via ward agent

## Goal Use a fresh `ward v0.483.0` director surface to verify the recovery behavior we have been trying to get back online. This issue is intentionally a verification handoff for the new director. Do not treat results from the old `v0.474.0` read-only surface as proof. ## Current saga The recovery incident had several overlapping failures: * Old director surfaces kept launching child engineers on stale ward versions. A `v0.474.0` director launched children that also recorded `ward v0.474.0` even after `v0.480.0` was published. Tracked in `ward#700`. * The old surface broker stripped important forwarded flags such as `--workflow pull-requests-and-merge` and `--details`, causing recovery dispatches to fall back to `workflow pr`. Earlier fixes for broker forwarding landed, but they must be verified from a fresh `v0.483.0` director. * Workers in `workflow pr` posted `WARD-OUTCOME: done` after opening a PR. That is the wrong state model: a PR is submitted, not done, until the change reaches `main`. Tracked in `ward#777`. * We still need proof that PR-shaped workflows can create a PR, wait for CI, and merge or hand off correctly without claiming done too early. Tracked in `ward#706`. * `--branch <name>` exists, but old bootstrap only ran `git checkout -B <branch>` and did not start from `origin/<branch>` when resuming an existing PR branch. Tracked in `ward#778`. Manual director recovery already merged these green PRs after checking CI: * `#772` - remove review stats command, merged to main. * `#774` - remove mcporter hydration, merged to main. * `#775` - remove setup and doctor release surfaces, merged to main. Latest known release before this issue was filed: `v0.483.0`, published 2026-07-09T07:46:51Z. Latest known main commit was `68e468764c03c309dae71d8c104f73dc5846484e` from PR `#775`. Its main `test`, `release`, `publish-binaries`, `publish-kdl-read`, `publish-kdl-write`, `bump-tap-formula`, and `bump-scoop-manifest` jobs were green. Open PRs at filing time were: * `#773` for `ward#735`, test failed on run 1159. * `#776` for `ward#737`, test failed on run 1162. Fresh `v0.480.0` workers were re-dispatched for `ward#735` and `ward#737` from the old surface, but their results should still be treated cautiously because the old surface was not the clean v0.483 director. ## Verification mission Perform verification from the fresh `v0.483.0` director only. Record findings as comments on this issue with concrete links to reservation comments, PRs, action runs, and commits. ### 1. Director and child version propagation Verify the new director surface is actually running `ward v0.483.0`. Then dispatch at least one child engineer without an explicit `--ward-version` pin and confirm the reservation comment records the current expected ward version, not an old surface bootstrap version such as `v0.474.0`. Pass condition: an unpinned child from the v0.483 director does not inherit an older surface `WARD_VERSION`. Fail condition: a child reservation records a stale ward version. If this fails, keep this issue open and point to `ward#700`. ### 2. Broker flag forwarding Verify the v0.483 director forwards `--workflow` and `--details` intact through the dispatch broker. Use a low-risk canary or an existing appropriate issue, but make the verification marker unmistakable. Required check: ```text --workflow pull-requests-and-merge --details "WARD-RECOVERY-VERIFY-0483: broker forwarding canary" ``` Pass condition: the reservation/run context shows `workflow pull-requests-and-merge`, and the worker seed receives the details text. Fail condition: the forwarded broker command or reservation falls back to `workflow pr`, or the details are missing. If this fails, keep this issue open and point back to the broker-forwarding fixes around `ward#763`. ### 3. PR outcome semantics Verify that PR-shaped workers no longer mark the issue done merely because a PR was opened. This applies to both `pr` and `pull-requests-and-merge`. Pass condition: a PR-only worker reports a non-done submitted or pending state, or the director records final done only after merge to `main`. Fail condition: any worker posts `WARD-OUTCOME: done` while the PR is still open or before the relevant commit reaches `main`. If this fails, keep this issue open and point to `ward#777`. ### 4. PR workflow merge behavior Verify the intended PR workflow behavior from end to end. The verification should prove whether ward can create a PR, wait for Forgejo Actions, avoid merging failed CI, and land only after green CI. Pass condition: a PR-shaped workflow either lands after green CI or explicitly records a non-done waiting state that the director can merge later. Fail condition: a worker reports done before CI, ignores failing CI, or requires manual web/API merge with no director-owned policy path. If this fails, keep this issue open and point to `ward#706`. ### 5. Existing branch resume Verify `--branch <existing-branch>` starts from the remote branch when `origin/<branch>` exists. A good practical target is a failed PR branch such as `issue-735` or `issue-737`, if those are still open when the new director sees this. Pass condition: the worker starts on the remote PR branch content, not a fresh branch reset from `main`, and the log makes the branch provenance visible. Fail condition: the worker starts from `main` while merely naming the branch. If this fails, keep this issue open and point to `ward#778`. ### 6. Current open PR cleanup Re-check open PRs after the fresh workers finish. Merge only PRs whose PR test is green and whose branch is current enough to merge cleanly. Do not merge `#773` or `#776` unless their failing tests have been fixed and rerun green. ## Reporting format Post a concise final comment on this issue with: * `WARD-RECOVERY-VERIFY-0483: pass` or `WARD-RECOVERY-VERIFY-0483: fail` on the first line. * Evidence links for each verification area above. * A list of any issues that remain blocking. * Whether any PRs were merged during verification, and whether main CI stayed green afterward. This issue is a verification controller, not a broad refactor. If a behavior is still broken, file or update the specific issue and stop short of claiming recovery complete.  Codex, via `ward agent`

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-09 07:56:09 +00:00

coilyco-ops commented

2026-07-09 07:59:52 +00:00

Author

Owner

WARD-RECOVERY-VERIFY-0483: in progress

Director-side evidence from this fresh surface:

Director binary is ward v0.483.0 (ward --version, 2026-07-09T07:57Z).
Unpinned child dispatch: ward#780, reservation at 2026-07-09T07:58:09Z records ward v0.483.0, harness codex, branch issue-780, workflow pull-requests-and-merge. Evidence: #780 (issue thread has the reservation comment).
Broker forwarding evidence: dispatch output printed ward agent engineer coilyco-flight-deck/ward#780 --harness codex --workflow pull-requests-and-merge --details WARD-RECOVERY-VERIFY-0483: broker forwarding canary; ward#780 issue body also carries the marker for the worker seed.
Existing branch resume canary: ward#782, dispatched with hidden --branch issue-735, --workflow patch-only, and details WARD-RECOVERY-VERIFY-0483: branch resume canary; reservation at 2026-07-09T07:59:08Z records ward v0.483.0, branch issue-735, workflow patch-only. Evidence: #782
Open PRs rechecked before dispatch: #773 for ward#735 and #776 for ward#737 were still open and not merged. Both source issues already had active v0.480.0 reservations, so this director did not force duplicate workers into those issue branches.

Remaining reconciliation after child runs finish:

Confirm ward#780 opens a PR, waits for CI, and does not claim done before the PR boundary is actually satisfied.
Confirm ward#782 posts the branch provenance check showing whether HEAD started from origin/issue-735.
Re-check PR #773, PR #776, and any PR from ward#780; merge only if green and policy-eligible.
Post the final required WARD-RECOVERY-VERIFY-0483: pass or fail comment after those outcomes are visible.

No PRs were merged by this director pass.

Codex, via ward agent

WARD-RECOVERY-VERIFY-0483: in progress Director-side evidence from this fresh surface: * Director binary is `ward v0.483.0` (`ward --version`, 2026-07-09T07:57Z). * Unpinned child dispatch: `ward#780`, reservation at 2026-07-09T07:58:09Z records ward `v0.483.0`, harness `codex`, branch `issue-780`, workflow `pull-requests-and-merge`. Evidence: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/780#issuecomment-unknown (issue thread has the reservation comment). * Broker forwarding evidence: dispatch output printed `ward agent engineer coilyco-flight-deck/ward#780 --harness codex --workflow pull-requests-and-merge --details WARD-RECOVERY-VERIFY-0483: broker forwarding canary`; `ward#780` issue body also carries the marker for the worker seed. * Existing branch resume canary: `ward#782`, dispatched with hidden `--branch issue-735`, `--workflow patch-only`, and details `WARD-RECOVERY-VERIFY-0483: branch resume canary`; reservation at 2026-07-09T07:59:08Z records ward `v0.483.0`, branch `issue-735`, workflow `patch-only`. Evidence: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/782 * Open PRs rechecked before dispatch: `#773` for `ward#735` and `#776` for `ward#737` were still open and not merged. Both source issues already had active `v0.480.0` reservations, so this director did not force duplicate workers into those issue branches. Remaining reconciliation after child runs finish: * Confirm `ward#780` opens a PR, waits for CI, and does not claim done before the PR boundary is actually satisfied. * Confirm `ward#782` posts the branch provenance check showing whether HEAD started from `origin/issue-735`. * Re-check PR `#773`, PR `#776`, and any PR from `ward#780`; merge only if green and policy-eligible. * Post the final required `WARD-RECOVERY-VERIFY-0483: pass` or `fail` comment after those outcomes are visible. No PRs were merged by this director pass.  Codex, via `ward agent`

coilyco-ops commented

2026-07-09 18:16:34 +00:00

Author

Owner

Validation pass: this was a point-in-time v0.483 recovery verification controller. Its own thread already recorded the in-progress evidence, and the failures it identified now live in specific implementation tickets: ward#700, ward#777, ward#778, and the later PR workflow recovery work. The director lane is now on newer ward versions, so this old verification controller is no longer an actionable headless ticket.

— Codex, via director surface

coilyco-ops closed this issue

2026-07-09 18:16:35 +00:00