ward agent: docker daemon restart orphans remote reservations (drain-on-exit waiter dies with the container, never releases the hold) #555

New issue

Closed

opened 2026-07-03 07:21:24 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-03 07:21:24 +00:00

Member

What happened

A docker restart (daemon-level) SIGKILLed/SIGTERMed all live engineer-* and session-* containers (exit 143/137). The host-side drain-on-exit waiter pids that release a run's reservation on clean exit were killed at the same time, so the remote Forgejo reservations were never released.

Result: every interrupted issue (observed: coilyco-flight-deck/ward#220, #289, #376, #411) stayed reserved by @coilyco-ops with the original dispatch timestamp. Re-dispatch failed hard:

reservation remote acquire failed ... is already reserved remotely (by @coilyco-ops at <original ts>); wait for it to finish or pass --force to override

The precheck correctly reported no live local holder, but there is no reconciliation that turns 'no live holder + container gone' into 'release the remote reservation'. The only recovery was --force on each issue (which the flag help does sanction: 'reclaim a stale or foreign hold').

The gap

The drain/reap path assumes the host-side waiter outlives the container. A daemon restart (or any event that kills waiter + container together) breaks that assumption and leaves orphaned remote holds. There is no host-side reconciler that, on ward startup or on demand, sweeps reservations whose container no longer exists and releases them.

Suggested fixes (pick)

Reconcile on startup / periodically: for each remote reservation held by this host's identity, if no matching live container exists, release it (liveness-based reclaim instead of manual --force).
A ward agent reservations surface: list (local + remote, with liveness) and release <ref>, so orphans are visible and clearable without --force on a blind guess.
Reap-driven release: make reservation release idempotent and also fire it from a supervisor that watches container liveness, not only from the per-run waiter.

Recovery used this time

ward agent coilyco-flight-deck/ward#<N> --force for each of #220/#289/#376/#411; all four re-dispatched and are running. Stale local reservation files for the exited-0/1 containers (#251/#294/#373/#387/#423/#494/#501/#508/#546 etc.) are also lingering under ~/.ward/agent-reservations/ and want the same reconciler.

Origin: found while recovering from an accidental Docker restart on kai-desktop-tower, 2026-07-03.

## What happened A `docker restart` (daemon-level) SIGKILLed/SIGTERMed all live `engineer-*` and `session-*` containers (exit 143/137). The host-side `drain-on-exit waiter` pids that release a run's reservation on clean exit were killed at the same time, so **the remote Forgejo reservations were never released**. Result: every interrupted issue (observed: `coilyco-flight-deck/ward#220, #289, #376, #411`) stayed reserved by `@coilyco-ops` with the *original* dispatch timestamp. Re-dispatch failed hard: ``` reservation remote acquire failed ... is already reserved remotely (by @coilyco-ops at <original ts>); wait for it to finish or pass --force to override ``` The precheck correctly reported `no live local holder`, but there is no reconciliation that turns 'no live holder + container gone' into 'release the remote reservation'. The only recovery was `--force` on each issue (which the flag help does sanction: 'reclaim a stale or foreign hold'). ## The gap The drain/reap path assumes the host-side waiter outlives the container. A daemon restart (or any event that kills waiter + container together) breaks that assumption and leaves orphaned remote holds. There is no host-side reconciler that, on ward startup or on demand, sweeps reservations whose container no longer exists and releases them. ## Suggested fixes (pick) - **Reconcile on startup / periodically**: for each remote reservation held by this host's identity, if no matching live container exists, release it (liveness-based reclaim instead of manual `--force`). - **A `ward agent reservations` surface**: `list` (local + remote, with liveness) and `release <ref>`, so orphans are visible and clearable without `--force` on a blind guess. - **Reap-driven release**: make reservation release idempotent and also fire it from a supervisor that watches container liveness, not only from the per-run waiter. ## Recovery used this time `ward agent coilyco-flight-deck/ward#<N> --force` for each of #220/#289/#376/#411; all four re-dispatched and are running. Stale *local* reservation files for the exited-0/1 containers (#251/#294/#373/#387/#423/#494/#501/#508/#546 etc.) are also lingering under `~/.ward/agent-reservations/` and want the same reconciler. Origin: found while recovering from an accidental Docker restart on kai-desktop-tower, 2026-07-03.

coilyco-ops added the

consult

labels

2026-07-03 17:30:25 +00:00