coilyco-flight-deck/ward

Fork 0

Clear ghost failed-before-start rows after successful brokered runs #1380

New issue

Closed

opened 2026-07-15 10:08:23 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-15 10:08:23 +00:00

Member

Problem

ward agent list can keep showing an old brokered engineer run as failed before container start even after the run actually completed, pushed to main, and reached the workflow boundary.

Current evidence from the QA director surface on 2026-07-15:

ward agent list --json reports coilyco-flight-deck/ward#1243 as status=failed, phase=failed before container start, container engineer-codex-ward-1243, age about 4h48m.
ward agent stop coilyco-flight-deck/ward#1243 --print refuses it as a ghost launch record: no running container exists and it is not stoppable through ward agent stop.
ward agent logs coilyco-flight-deck/ward#1243 --tail 80 uses the drained console log with outcome pushed-to-main, shows PR #1322 merged, go test ./... passed, and the reaper reached the workflow boundary.
The linked issue #1243 is closed and PR #1322 is merged.

That leaves dispatch/list health with a stale failed-before-start row for a successful historical run, which makes the operations surface noisier and weakens trust in the list/health readouts.

Expected behavior

A run that has a drained log outcome showing a successful workflow boundary should not remain counted or rendered as a failed-before-start launch intent. The stale local reservation or dispatch artifact should be pruned, reclassified as cleanup-only with clear remediation, or otherwise excluded from the failed operational count.

Acceptance criteria

Add a regression test for a stale reservation/dispatch artifact where the drained engineer log outcome is successful (pushed-to-main or equivalent workflow-boundary success) and no running container exists.
ward agent list --json must not report that row as status=failed / phase=failed before container start.
ward agent dispatch-health --repo coilyco-flight-deck/ward --json must not surface that successful historical run as a current failed launch.
The cleanup/remediation path for true ghost launches remains clear and does not hide real failed-before-start runs.

Follow-up found during QA of #1243 / PR #1322.

## Problem `ward agent list` can keep showing an old brokered engineer run as `failed before container start` even after the run actually completed, pushed to main, and reached the workflow boundary. Current evidence from the QA director surface on 2026-07-15: * `ward agent list --json` reports `coilyco-flight-deck/ward#1243` as `status=failed`, `phase=failed before container start`, container `engineer-codex-ward-1243`, age about 4h48m. * `ward agent stop coilyco-flight-deck/ward#1243 --print` refuses it as a ghost launch record: no running container exists and it is not stoppable through `ward agent stop`. * `ward agent logs coilyco-flight-deck/ward#1243 --tail 80` uses the drained console log with `outcome pushed-to-main`, shows PR #1322 merged, `go test ./...` passed, and the reaper reached the workflow boundary. * The linked issue #1243 is closed and PR #1322 is merged. That leaves dispatch/list health with a stale failed-before-start row for a successful historical run, which makes the operations surface noisier and weakens trust in the list/health readouts. ## Expected behavior A run that has a drained log outcome showing a successful workflow boundary should not remain counted or rendered as a failed-before-start launch intent. The stale local reservation or dispatch artifact should be pruned, reclassified as cleanup-only with clear remediation, or otherwise excluded from the failed operational count. ## Acceptance criteria * Add a regression test for a stale reservation/dispatch artifact where the drained engineer log outcome is successful (`pushed-to-main` or equivalent workflow-boundary success) and no running container exists. * `ward agent list --json` must not report that row as `status=failed` / `phase=failed before container start`. * `ward agent dispatch-health --repo coilyco-flight-deck/ward --json` must not surface that successful historical run as a current failed launch. * The cleanup/remediation path for true ghost launches remains clear and does not hide real failed-before-start runs. ## Related Follow-up found during QA of #1243 / PR #1322.

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 10:08:23 +00:00

coilyco-ops added the

coherence-core

headless

labels

2026-07-15 10:08:23 +00:00

coilyco-ops commented

2026-07-15 10:08:46 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1380 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T10:08:45Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1380 · branch issue-1380 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1380 · ward v0.710.0 · dispatched 2026-07-15T10:08:40Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1380` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T10:08:45Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1380` · branch `issue-1380` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1380` · ward `v0.710.0` · dispatched `2026-07-15T10:08:40Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-15 10:09:32 +00:00

Keep the host dispatch broker alive after brokered dispatch failures #1243

coilyco-ops referenced this issue from a commit

2026-07-15 10:17:54 +00:00

Fix stale failed-before-start rows after successful runs

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 10:19:24 +00:00

Clear ghost failed-before-start rows after successful brokered runs #1384

coilyco-ops closed this issue

2026-07-15 10:20:23 +00:00

coilyco-ops commented

2026-07-15 10:26:29 +00:00

Author

Member

QA follow-up: PASS. PR #1384 merged at 659252ca21 and release run 8454 succeeded.

Evidence checked:

PR test run 8450 succeeded; promote run 8452 succeeded; release run 8454 succeeded.
Focused code review confirms stale reservation rows are pruned only when the newest drained engineer metadata for that issue has a successful workflow outcome (pushed-to-main / normalized workflow-boundary success).
The existing failed-before-start path remains visible and excluded from active counts when no successful drain supersedes it.
dispatch-health --repo coilyco-flight-deck/ward --json currently reports failed_before_start=0, failed=0, backpressure=false.
Focused local verification passed with broker forwarding disabled for the test shell: go test ./cmd/ward -run 'TestAgentList(PrunesFailedBeforeStartRowAfterSuccessfulDrain|KeepsFailedBeforeStartRowsVisibleButExcluded|MarksStalePrelaunchLaunchesCleanupNeeded|MarksPartialLaunchWhenReservationMarkerIsMissing|PrunesStaleReservationCacheEntry)|TestRunAgentLogsIssueScopedArchiveEmptyExplainsSelectedSource|TestRunAgentLogs(ReadsDispatchArtifact|ReadsArchivedContainer|LiveDockerEmpty)' -count=1.\n\nThe issue criteria are met; no follow-up issue needed.

QA follow-up: PASS. PR #1384 merged at 659252ca21452bfe5f763d9174433dc528bd4d18 and release run 8454 succeeded. Evidence checked: - PR test run 8450 succeeded; promote run 8452 succeeded; release run 8454 succeeded. - Focused code review confirms stale reservation rows are pruned only when the newest drained engineer metadata for that issue has a successful workflow outcome (`pushed-to-main` / normalized workflow-boundary success). - The existing failed-before-start path remains visible and excluded from active counts when no successful drain supersedes it. - `dispatch-health --repo coilyco-flight-deck/ward --json` currently reports `failed_before_start=0`, `failed=0`, `backpressure=false`. - Focused local verification passed with broker forwarding disabled for the test shell: `go test ./cmd/ward -run 'TestAgentList(PrunesFailedBeforeStartRowAfterSuccessfulDrain|KeepsFailedBeforeStartRowsVisibleButExcluded|MarksStalePrelaunchLaunchesCleanupNeeded|MarksPartialLaunchWhenReservationMarkerIsMissing|PrunesStaleReservationCacheEntry)|TestRunAgentLogsIssueScopedArchiveEmptyExplainsSelectedSource|TestRunAgentLogs(ReadsDispatchArtifact|ReadsArchivedContainer|LiveDockerEmpty)' -count=1`.\n\nThe issue criteria are met; no follow-up issue needed.