coilyco-flight-deck/ward

Fork 0

Repair current-main broker cleanup failure after PR #1426 #1428

New issue

Closed

opened 2026-07-15 12:25:35 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-15 12:25:35 +00:00

Member

Problem

QA closed PR #1426 unmerged because its green PR check was stale relative to current main after PR #1425 landed. The branch still merged cleanly, but the combined result fails the focused broker cleanup regression.

Evidence from the director surface on 2026-07-15:

base: origin/main = 7b7e4f5966e4d3a4d34bf3fb41be598db4df876b (PR #1425 merged)
head: PR #1426 = 6e2edc14c42d8d507a3ae41ce3cb53400bd85952
command: go test ./cmd/ward -run TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns -count=20
--- FAIL: TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns
agent_dispatch_broker_test.go:1790: remove redacted dispatch artifact dir .../.ward/agent-logs-redacted/dispatch: unlinkat .../coilyco-flight-deck-ward-795: directory not empty

PR #1426 tried to remove the raw and redacted dispatch artifact roots explicitly, but after #1425 adds whole temporary .ward cleanup, the explicit redacted RemoveAll can still race with a non-empty dispatch artifact directory and fail the test.

Acceptance

Start from current main including PR #1425.
Repair TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns so it does not fail tempdir cleanup or explicit artifact cleanup for raw or redacted dispatch logs.
Preserve the intent from #1422/#1425: broker launch env restoration is still tested and dispatch artifacts do not leak into tempdir teardown.
Add or adjust coverage so this failure is caught on the current-main combined state, not only on the pre-#1425 PR base.
Confirm go test ./cmd/ward -run TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns -count=20 passes, and confirm the relevant PR check or promote gate is green.

Related: #1423, #1426, #1425, run 8530 / #2252.

## Problem QA closed PR #1426 unmerged because its green PR check was stale relative to current `main` after PR #1425 landed. The branch still merged cleanly, but the combined result fails the focused broker cleanup regression. Evidence from the director surface on 2026-07-15: ```text base: origin/main = 7b7e4f5966e4d3a4d34bf3fb41be598db4df876b (PR #1425 merged) head: PR #1426 = 6e2edc14c42d8d507a3ae41ce3cb53400bd85952 command: go test ./cmd/ward -run TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns -count=20 --- FAIL: TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns agent_dispatch_broker_test.go:1790: remove redacted dispatch artifact dir .../.ward/agent-logs-redacted/dispatch: unlinkat .../coilyco-flight-deck-ward-795: directory not empty ``` PR #1426 tried to remove the raw and redacted dispatch artifact roots explicitly, but after #1425 adds whole temporary `.ward` cleanup, the explicit redacted `RemoveAll` can still race with a non-empty dispatch artifact directory and fail the test. ## Acceptance - Start from current `main` including PR #1425. - Repair `TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns` so it does not fail tempdir cleanup or explicit artifact cleanup for raw or redacted dispatch logs. - Preserve the intent from #1422/#1425: broker launch env restoration is still tested and dispatch artifacts do not leak into tempdir teardown. - Add or adjust coverage so this failure is caught on the current-main combined state, not only on the pre-#1425 PR base. - Confirm `go test ./cmd/ward -run TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns -count=20` passes, and confirm the relevant PR check or promote gate is green. Related: #1423, #1426, #1425, run 8530 / #2252.

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 12:25:35 +00:00

coilyco-ops added the

headless

labels

2026-07-15 12:25:35 +00:00

coilyco-ops commented

2026-07-15 12:25:57 +00:00

Author

Member

DISPATCH-DEFERRED: capacity blocked. Attempted warded engineer coilyco-flight-deck/ward#1428 --harness codex --workflow pull-request-and-merge at 2026-07-15T12:25Z from the director surface. Ward rejected the launch because coilyco-flight-deck/ward already has 3 active engineers at the repo limit. Broker evidence: repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3). Dispatch log: /Users/kai/.ward/agent-logs/dispatch/20260715T122540Z-director-codex-jp85-coilyco-flight-deck-ward-1428.log. Retry this current-main broker cleanup follow-up as soon as a slot opens.

DISPATCH-DEFERRED: capacity blocked. Attempted `warded engineer coilyco-flight-deck/ward#1428 --harness codex --workflow pull-request-and-merge` at 2026-07-15T12:25Z from the director surface. Ward rejected the launch because coilyco-flight-deck/ward already has 3 active engineers at the repo limit. Broker evidence: repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3). Dispatch log: /Users/kai/.ward/agent-logs/dispatch/20260715T122540Z-director-codex-jp85-coilyco-flight-deck-ward-1428.log. Retry this current-main broker cleanup follow-up as soon as a slot opens.

coilyco-ops referenced this issue

2026-07-15 12:27:50 +00:00

Repair failed promote run after PR #1421 #1423

coilyco-ops commented

2026-07-15 12:27:58 +00:00

Author

Member

Additional post-merge evidence after PR #1426 landed anyway:

current main: cbc42f3b0c0b821346176b782bacfb843ac94af1
command: go test ./cmd/ward -run TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns -count=20
--- FAIL: TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns (0.00s)
agent_dispatch_broker_test.go:1739: runHostDispatchBrokerRequest returned early: {logPath:.../.ward/agent-logs/dispatch/.../console.log err:<nil>}
FAIL github.com/coilyco-flight-deck/ward/cmd/ward 20.059s

So #1428 should now be treated as the post-merge repair for #1426, not just a pre-merge stale-base guard. The earlier dispatch attempt is still capacity-blocked at 3/3 active engineers.

Additional post-merge evidence after PR #1426 landed anyway: ```text current main: cbc42f3b0c0b821346176b782bacfb843ac94af1 command: go test ./cmd/ward -run TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns -count=20 --- FAIL: TestRunHostDispatchBrokerRequestClearsBrokerEnvWhileLaunchRuns (0.00s) agent_dispatch_broker_test.go:1739: runHostDispatchBrokerRequest returned early: {logPath:.../.ward/agent-logs/dispatch/.../console.log err:<nil>} FAIL github.com/coilyco-flight-deck/ward/cmd/ward 20.059s ``` So #1428 should now be treated as the post-merge repair for #1426, not just a pre-merge stale-base guard. The earlier dispatch attempt is still capacity-blocked at 3/3 active engineers.

coilyco-ops commented

2026-07-15 12:28:44 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1428 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T12:28:43Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1428 · branch issue-1428 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1428 · ward v0.710.0 · dispatched 2026-07-15T12:28:36Z
Comment thread: 2 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-15T12:25:57Z), @coilyco-ops (2026-07-15T12:27:58Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1428` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T12:28:43Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1428` · branch `issue-1428` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1428` · ward `v0.710.0` · dispatched `2026-07-15T12:28:36Z` - **Comment thread:** 2 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-15T12:25:57Z), @coilyco-ops (2026-07-15T12:27:58Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 12:36:11 +00:00

Repair current-main broker cleanup failure after PR #1426 #1434

coilyco-ops closed this issue

2026-07-15 12:38:03 +00:00

coilyco-ops commented

2026-07-15 12:38:14 +00:00

Author

Member

WARDED_WORKFLOW: #1434

details

workflow: pull-request-and-merge; review summary: skipped (review gate intentionally skipped because the temporary ward default is pending brokered QA)

Implemented the narrow broker cleanup fix by removing the race-prone dispatch subtree RemoveAll calls and keeping the env-restoration assertion on current main.

confidence: high
surprises: Forgejo Actions took a short while to report, but the run finished green.
follow-ups: none

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1434 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped (review gate intentionally skipped because the temporary ward default is pending brokered QA) Implemented the narrow broker cleanup fix by removing the race-prone dispatch subtree RemoveAll calls and keeping the env-restoration assertion on current main. confidence: high surprises: Forgejo Actions took a short while to report, but the run finished green. follow-ups: none </details>