ward agent: smoke-test death (ward#222) leaks the issue reservation, blocking retry without --force #264

New issue

Closed

opened 2026-06-24 07:25:02 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-06-24 07:25:02 +00:00

Member

Summary

When a headless agent container dies at the auth smoke test (ward#222), it does not release the remote issue reservation. The stale hold then blocks any retry of that issue until someone passes --force.

How it showed up

A batch of ward agent headless containers died, all at the same point:

ward-container: auth smoke test: probing claude before launch (ward#222)
ward-container: fatal: auth smoke test: claude -p did not respond within 90s -
  credentials are unusable in-container (ward#222).
ward-container: reaping: salvage residual work before teardown
ward container reap: nothing to reap (tree clean, HEAD on origin/main)

Root cause of the batch death was API saturation, not credentials: pass/fail was interleaved in the same window (some containers exited 0 while neighbors timed out), and the failure is a 90s hang rather than a fast 401. Many containers firing claude -p at once pushed some smoke-test probes past the 90s ceiling.

The containers reaped clean (zero work done, HEAD still on origin/main), yet the issue stayed reserved:

ward: ward agent headless: issue coilyco-bridge/deploy#9 is already reserved
  remotely (by @coilyco-ops at 2026-06-24T06:46:28Z); wait for it to finish or
  pass --force to override

That reservation was still held 33 minutes after the container died with nothing to show for it.

Impact

Every rate-limit / smoke-test death leaves a stale lock. Retrying the affected issues requires --force per issue, which also bypasses the genuine-concurrency safety check. The recovery path for a transient, expected failure (API saturation) is the same blunt instrument used to reclaim a foreign hold.

Suggested fix

The reaper already runs on the smoke-test-death path ("reaping: salvage residual work before teardown"). When it reaps clean AND the container never got past the smoke test (did no work), it should release the issue reservation it acquired, so an ordinary (non---force) retry just works. A reservation should not outlive a container that did nothing.

Optionally: treat the smoke-test timeout as retryable in-place with backoff rather than a fatal exit, so transient saturation self-heals without leaving the lock at all.

Repro

Dispatch enough ward agent headless containers concurrently to saturate the API so some smoke tests exceed 90s.
Observe a container exit at the ward#222 smoke test, reaping clean.
Re-run ward agent headless <same-issue> without --force.
It refuses: issue already reserved remotely, despite no live container.

## Summary When a headless agent container dies at the **auth smoke test** (ward#222), it does not release the remote issue reservation. The stale hold then blocks any retry of that issue until someone passes `--force`. ## How it showed up A batch of `ward agent headless` containers died, all at the same point: ``` ward-container: auth smoke test: probing claude before launch (ward#222) ward-container: fatal: auth smoke test: claude -p did not respond within 90s - credentials are unusable in-container (ward#222). ward-container: reaping: salvage residual work before teardown ward container reap: nothing to reap (tree clean, HEAD on origin/main) ``` Root cause of the batch death was API saturation, not credentials: pass/fail was interleaved in the same window (some containers exited 0 while neighbors timed out), and the failure is a 90s hang rather than a fast 401. Many containers firing `claude -p` at once pushed some smoke-test probes past the 90s ceiling. The containers reaped clean (zero work done, HEAD still on origin/main), yet the issue stayed reserved: ``` ward: ward agent headless: issue coilyco-bridge/deploy#9 is already reserved remotely (by @coilyco-ops at 2026-06-24T06:46:28Z); wait for it to finish or pass --force to override ``` That reservation was still held 33 minutes after the container died with nothing to show for it. ## Impact Every rate-limit / smoke-test death leaves a stale lock. Retrying the affected issues requires `--force` per issue, which also bypasses the genuine-concurrency safety check. The recovery path for a transient, expected failure (API saturation) is the same blunt instrument used to reclaim a foreign hold. ## Suggested fix The reaper already runs on the smoke-test-death path ("reaping: salvage residual work before teardown"). When it reaps clean AND the container never got past the smoke test (did no work), it should release the issue reservation it acquired, so an ordinary (non-`--force`) retry just works. A reservation should not outlive a container that did nothing. Optionally: treat the smoke-test timeout as retryable in-place with backoff rather than a fatal exit, so transient saturation self-heals without leaving the lock at all. ## Repro 1. Dispatch enough `ward agent headless` containers concurrently to saturate the API so some smoke tests exceed 90s. 2. Observe a container exit at the ward#222 smoke test, reaping clean. 3. Re-run `ward agent headless <same-issue>` without `--force`. 4. It refuses: issue already reserved remotely, despite no live container.

coilyco-ops commented

2026-06-24 08:42:42 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container ward-ward-issue-264-claude-80cdd128 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-24T08:42:41Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `ward-ward-issue-264-claude-80cdd128` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-24T08:42:41Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`