coilyco-flight-deck/ward

Fork 0

Engineer dispatch smoke-test-dies on KAI-SERVER and a death can orphan the issue - failover re-dispatch is not deterministic #595

New issue

Closed

opened 2026-07-04 04:51:01 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-07-04 04:51:01 +00:00

Owner

Summary

Engineer dispatch is smoke-test-dying on KAI-SERVER (the ward#222/#264 failure mode, live and recurring), and - the new and worse part - a smoke-test death does not reliably trigger a re-dispatch, so the carried issue is silently orphaned: it shows a reservation-then-release in its thread, no engineer is running, and no work happens, yet nothing signals that the run never started.

Evidence (one surface session, two dispatches, 2026-07-04)

Both engineer runs started on KAI-SERVER and smoke-test-died within ~7-8s. One got a lucky failover, one did not:

ward#593 - reserved on KAI-SERVER at 04:24:44, released 04:24:52: "container exited without launching the agent (smoke-test death, ward#222/#264)". A second attempt landed on KAI-DESKTOP-TOWER at 04:29:43 and completed with a real WARD-OUTCOME: done. Survived by failover.
ward#594 - reserved on KAI-SERVER at 04:39:45, released 04:39:52, same smoke-test death. No retry. No second host picked it up. The issue is orphaned: open, reservation released, zero work done, no running container.

So whether a dispatch survives is a coin-flip on whether a retry happens to land on a healthy host, and when it does not, the failure is invisible from the issue state.

Two distinct defects

KAI-SERVER smoke-test death (recurrence of ward#222/#264). Containers exit before launching the agent on that host. This is the trigger, and it is live right now, not historical.
Failover orphaning gap (new). ward container reap releases the reservation on a smoke-test death ("a plain retry no longer needs --force") but does not re-queue the work. Re-dispatch, when it happens, is not part of the death-handling path - #593 got one, #594 did not. The result is an orphaned issue that reads as "was dispatched" while nothing is running and nothing will run. A smoke-test death should either deterministically re-dispatch (to a healthy host, or the same host after remediation) or leave a loud, machine-detectable "run never started - needs re-dispatch" marker on the issue, not a quiet reservation-release that looks benign.

Impact

Silent work loss. An operator (or the director heartbeat) sees a reservation comment and assumes the run is in flight. The orphaned issue never lands and never signals failure. #594's catalog work simply did not happen and would have sat indefinitely undetected.

Proposed fix

Fix or drain KAI-SERVER so it stops smoke-test-dying (ward#222/#264), and/or stop routing first attempts to a host that is failing this check.
Make death-handling own the retry: a smoke-test death re-dispatches deterministically rather than relying on an incidental second attempt, and an issue that cannot be re-dispatched carries an explicit orphaned/needs-redispatch marker.

ward#222 / ward#264 - the smoke-test death mechanism this recurs.
ward#577 - advisor dispatch flakiness in the same host-run surface (adjacent dispatch reliability).
ward#594 - the orphaned run (needs re-dispatch). ward#593 - the one that survived via failover.

Filed from the surface session that dispatched both runs.

## Summary Engineer dispatch is smoke-test-dying on **KAI-SERVER** (the ward#222/#264 failure mode, live and recurring), and - the new and worse part - **a smoke-test death does not reliably trigger a re-dispatch, so the carried issue is silently orphaned**: it shows a reservation-then-release in its thread, no engineer is running, and no work happens, yet nothing signals that the run never started. ## Evidence (one surface session, two dispatches, 2026-07-04) Both engineer runs started on **KAI-SERVER** and smoke-test-died within ~7-8s. One got a lucky failover, one did not: - **ward#593** - reserved on KAI-SERVER at 04:24:44, released 04:24:52: "container exited without launching the agent (smoke-test death, ward#222/#264)". A second attempt landed on **KAI-DESKTOP-TOWER** at 04:29:43 and completed with a real `WARD-OUTCOME: done`. Survived by failover. - **ward#594** - reserved on KAI-SERVER at 04:39:45, released 04:39:52, same smoke-test death. **No retry. No second host picked it up.** The issue is **orphaned**: open, reservation released, zero work done, no running container. So whether a dispatch survives is a coin-flip on whether a retry happens to land on a healthy host, and when it does not, the failure is invisible from the issue state. ## Two distinct defects 1. **KAI-SERVER smoke-test death (recurrence of ward#222/#264).** Containers exit before launching the agent on that host. This is the trigger, and it is live right now, not historical. 2. **Failover orphaning gap (new).** `ward container reap` releases the reservation on a smoke-test death ("a plain retry no longer needs --force") but **does not re-queue the work**. Re-dispatch, when it happens, is not part of the death-handling path - #593 got one, #594 did not. The result is an orphaned issue that reads as "was dispatched" while nothing is running and nothing will run. A smoke-test death should either deterministically re-dispatch (to a healthy host, or the same host after remediation) or leave a loud, machine-detectable "run never started - needs re-dispatch" marker on the issue, not a quiet reservation-release that looks benign. ## Impact Silent work loss. An operator (or the director heartbeat) sees a reservation comment and assumes the run is in flight. The orphaned issue never lands and never signals failure. #594's catalog work simply did not happen and would have sat indefinitely undetected. ## Proposed fix - Fix or drain KAI-SERVER so it stops smoke-test-dying (ward#222/#264), and/or stop routing first attempts to a host that is failing this check. - Make death-handling own the retry: a smoke-test death re-dispatches deterministically rather than relying on an incidental second attempt, and an issue that cannot be re-dispatched carries an explicit orphaned/needs-redispatch marker. ## Related - ward#222 / ward#264 - the smoke-test death mechanism this recurs. - ward#577 - advisor dispatch flakiness in the same host-run surface (adjacent dispatch reliability). - ward#594 - the orphaned run (needs re-dispatch). ward#593 - the one that survived via failover. Filed from the surface session that dispatched both runs.

coilyco-ops added the

consult

labels

2026-07-04 08:21:28 +00:00

coilysiren referenced this issue

2026-07-06 18:58:25 +00:00

ward agent advisor fails to launch with exit status 125 (engineer dispatch works; advisor-only tailnet/~/.aws mount likely) #604

coilyco-ops added

headless

and removed

consult

labels

2026-07-07 22:45:47 +00:00

coilyco-ops referenced this issue

2026-07-07 22:46:03 +00:00

Fix residual reap landing path that can push work without closing the carried issue #515

coilyco-ops commented

2026-07-07 22:46:03 +00:00

Member

Cleared consult→headless (2026-07-07), but held one step behind #515 - both edit cmd/ward/container_reap.go, so firing concurrently would collide. Dispatch when #515 lands. Scope for this run = defect 2 only (the failover-orphan gap): a smoke-test death must own its retry - deterministic re-dispatch to a healthy host + a loud machine-detectable needs-redispatch marker on the issue, and don't route first attempts to a host currently failing the smoke check. Defect 1 (physically fix/drain KAI-SERVER's smoke-test death, ward#222/#264) is a host-ops tail, not this container run.

Cleared consult→headless (2026-07-07), but **held one step behind #515** - both edit cmd/ward/container_reap.go, so firing concurrently would collide. Dispatch when #515 lands. Scope for this run = **defect 2 only** (the failover-orphan gap): a smoke-test death must own its retry - deterministic re-dispatch to a healthy host + a loud machine-detectable needs-redispatch marker on the issue, and don't route first attempts to a host currently failing the smoke check. **Defect 1** (physically fix/drain KAI-SERVER's smoke-test death, ward#222/#264) is a host-ops tail, not this container run.

coilyco-ops commented

2026-07-07 23:51:41 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-595 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-07T23:51:37Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#595 · branch issue-595 · driver claude · workflow direct-main
Run: engineer-claude-ward-595 · ward v0.422.0 · dispatched 2026-07-07T23:51:37Z
Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-07T22:46:03Z)

Issue body as seeded:

## Summary

Engineer dispatch is smoke-test-dying on **KAI-SERVER** (the ward#222/#264 failure mode, live and recurring), and - the new and worse part - **a smoke-test death does not reliably trigger a re-dispatch, so the carried issue is silently orphaned**: it shows a reservation-then-release in its thread, no engineer is running, and no work happens, yet nothing signals that the run never started.

## Evidence (one surface session, two dispatches, 2026-07-04)

Both engineer runs started on **KAI-SERVER** and smoke-test-died within ~7-8s. One got a lucky failover, one did not:

- **ward#593** - reserved on KAI-SERVER at 04:24:44, released 04:24:52: "container exited without launching the agent (smoke-test death, ward#222/#264)". A second attempt landed on **KAI-DESKTOP-TOWER** at 04:29:43 and completed with a real `WARD-OUTCOME: done`. Survived by failover.
- **ward#594** - reserved on KAI-SERVER at 04:39:45, released 04:39:52, same smoke-test death. **No retry. No second host picked it up.** The issue is **orphaned**: open, reservation released, zero work done, no running container.

So whether a dispatch survives is a coin-flip on whether a retry happens to land on a healthy host, and when it does not, the failure is invisible from the issue state.

## Two distinct defects

1. **KAI-SERVER smoke-test death (recurrence of ward#222/#264).** Containers exit before launching the agent on that host. This is the trigger, and it is live right now, not historical.

2. **Failover orphaning gap (new).** `ward container reap` releases the reservation on a smoke-test death ("a plain retry no longer needs --force") but **does not re-queue the work**. Re-dispatch, when it happens, is not part of the death-handling path - #593 got one, #594 did not. The result is an orphaned issue that reads as "was dispatched" while nothing is running and nothing will run. A smoke-test death should either deterministically re-dispatch (to a healthy host, or the same host after remediation) or

… (truncated to 2000 chars; full body is on this issue)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.422.0).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-595` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-07T23:51:37Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#595` · branch `issue-595` · driver `claude` · workflow `direct-main` - **Run:** `engineer-claude-ward-595` · ward `v0.422.0` · dispatched `2026-07-07T23:51:37Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-07T22:46:03Z) **Issue body as seeded:** ``` ## Summary Engineer dispatch is smoke-test-dying on **KAI-SERVER** (the ward#222/#264 failure mode, live and recurring), and - the new and worse part - **a smoke-test death does not reliably trigger a re-dispatch, so the carried issue is silently orphaned**: it shows a reservation-then-release in its thread, no engineer is running, and no work happens, yet nothing signals that the run never started. ## Evidence (one surface session, two dispatches, 2026-07-04) Both engineer runs started on **KAI-SERVER** and smoke-test-died within ~7-8s. One got a lucky failover, one did not: - **ward#593** - reserved on KAI-SERVER at 04:24:44, released 04:24:52: "container exited without launching the agent (smoke-test death, ward#222/#264)". A second attempt landed on **KAI-DESKTOP-TOWER** at 04:29:43 and completed with a real `WARD-OUTCOME: done`. Survived by failover. - **ward#594** - reserved on KAI-SERVER at 04:39:45, released 04:39:52, same smoke-test death. **No retry. No second host picked it up.** The issue is **orphaned**: open, reservation released, zero work done, no running container. So whether a dispatch survives is a coin-flip on whether a retry happens to land on a healthy host, and when it does not, the failure is invisible from the issue state. ## Two distinct defects 1. **KAI-SERVER smoke-test death (recurrence of ward#222/#264).** Containers exit before launching the agent on that host. This is the trigger, and it is live right now, not historical. 2. **Failover orphaning gap (new).** `ward container reap` releases the reservation on a smoke-test death ("a plain retry no longer needs --force") but **does not re-queue the work**. Re-dispatch, when it happens, is not part of the death-handling path - #593 got one, #594 did not. The result is an orphaned issue that reads as "was dispatched" while nothing is running and nothing will run. A smoke-test death should either deterministically re-dispatch (to a healthy host, or the same host after remediation) or ``` … (truncated to 2000 chars; full body is on this issue) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.422.0). </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-08 00:04:33 +00:00

fix(agent): pre-launch death owns its retry, not a silent orphan

coilyco-ops closed this issue

2026-07-08 00:04:33 +00:00

coilyco-ops commented

2026-07-08 00:05:11 +00:00

Member

WARD-OUTCOME: done - pre-launch smoke-test deaths now own their retry (deterministic bounded re-dispatch) and leave a loud, machine-detectable needs-redispatch marker instead of a silent orphan.

Scoped to defect 2 as the thread asked, so this run never touched container_reap.go (the #515 collision file) - the whole fix lives in agent_reserve.go (the release comment) and agent_director.go (the reconcile pass), which sidestepped the dependency entirely.

The satisfying part was that the fix fell straight out of existing doctrine: agent-director-dispatch.md already argues that a launch/infra failure which consumed no autonomous run must defer, not park terminally. The orphan bug was that exact rule missing on the reconcile side - a pre-launch death (container detached, then died at the ward#222 gate) was parked terminal failed instead of re-queued. So this is less a new mechanism than closing a gap the codebase already knew the shape of. Classification came out as a clean pure reconcileNoOutcome/prelaunchDeathRelease pair, easy to unit-test.

What fought back was purely the comment-discipline hook - two rounds of trimming doc-comments to the 2-line / 90-char cap. No logic churn.

Fairly confident in the two code paths (bounded retry + loud marker, both tested). One honest rough edge worth a follow-up: the thread also asked to not route first attempts to a host currently failing the smoke check. ward's director dispatches engineers on its own host with no cross-host health-aware router, so I couldn't implement true "route to a healthy host" here - the bounded-park-with-loud-marker is the backstop (a sick host parks blocked loudly rather than looping or orphaning, and the marker lets another host pick it up). A host-side pre-dispatch smoke probe (mirroring ward doctor's Ollama probe) would be the real fix and is worth its own issue. KAI-SERVER's physical remediation stays host-ops, as scoped.

WARD-OUTCOME: done - pre-launch smoke-test deaths now own their retry (deterministic bounded re-dispatch) and leave a loud, machine-detectable needs-redispatch marker instead of a silent orphan. Scoped to defect 2 as the thread asked, so this run never touched `container_reap.go` (the #515 collision file) - the whole fix lives in `agent_reserve.go` (the release comment) and `agent_director.go` (the reconcile pass), which sidestepped the dependency entirely. The satisfying part was that the fix fell straight out of existing doctrine: `agent-director-dispatch.md` already argues that a launch/infra failure which consumed no autonomous run must **defer, not park terminally**. The orphan bug was that exact rule missing on the *reconcile* side - a pre-launch death (container detached, then died at the ward#222 gate) was parked terminal `failed` instead of re-queued. So this is less a new mechanism than closing a gap the codebase already knew the shape of. Classification came out as a clean pure `reconcileNoOutcome`/`prelaunchDeathRelease` pair, easy to unit-test. What fought back was purely the comment-discipline hook - two rounds of trimming doc-comments to the 2-line / 90-char cap. No logic churn. Fairly confident in the two code paths (bounded retry + loud marker, both tested). One honest rough edge worth a follow-up: the thread also asked to **not route first attempts to a host currently failing the smoke check**. ward's director dispatches engineers on its *own* host with no cross-host health-aware router, so I couldn't implement true "route to a healthy host" here - the bounded-park-with-loud-marker is the backstop (a sick host parks `blocked` loudly rather than looping or orphaning, and the marker lets another host pick it up). A host-side pre-dispatch smoke probe (mirroring `ward doctor`'s Ollama probe) would be the real fix and is worth its own issue. KAI-SERVER's physical remediation stays host-ops, as scoped.

coilyco-ops referenced this issue

2026-07-08 22:19:00 +00:00

Make agent workflow defaults configurable and default engineers to PRs #707