Drain closed agent-run logs shortly after container exit, not only at keep-10 eviction #510

New issue

Closed

opened 2026-07-02 04:58:41 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-07-02 04:58:41 +00:00

Owner

Infrastructure logdy watches ~/.ward/agent-logs/ and the operator requirement is now explicit: newly closed ward agent runs should appear there shortly after the container stops, not only when the exited container ages out of the keep-10 retention window.

Current behavior:

ward drains console.log, transcript.jsonl, and meta.json to ~/.ward/agent-logs/<container>/ only during the keep-10 stale-container sweep, right before docker rm of older exited containers
infrastructure logdy correctly watches that tree, but therefore cannot show a just-closed run until much later

Required outcome:

shortly after a ward agent container exits, its drained run artifacts become available under ~/.ward/agent-logs/<container>/
the keep-10 retention policy for exited containers can remain, but observability for closed runs must not wait on eviction from that window

Acceptance criteria:

design and implement the earliest safe host-side point to drain a just-closed run's console/transcript/meta into ~/.ward/agent-logs/<container>/
preserve the current safety properties: no secret spill, best-effort drain, and no destructive interaction with live containers
make the behavior idempotent if the later stale-container sweep sees a run that was already drained
add or update tests around drain timing/order and duplicate-drain handling
update ward observability / cleanup docs to describe the new timing
link the infrastructure consumer issue: coilyco-flight-deck/infrastructure#424

Infrastructure logdy watches `~/.ward/agent-logs/` and the operator requirement is now explicit: newly closed `ward agent` runs should appear there shortly after the container stops, not only when the exited container ages out of the keep-10 retention window. Current behavior: - ward drains `console.log`, `transcript.jsonl`, and `meta.json` to `~/.ward/agent-logs/<container>/` only during the keep-10 stale-container sweep, right before `docker rm` of older exited containers - infrastructure logdy correctly watches that tree, but therefore cannot show a just-closed run until much later Required outcome: - shortly after a `ward agent` container exits, its drained run artifacts become available under `~/.ward/agent-logs/<container>/` - the keep-10 retention policy for exited containers can remain, but observability for closed runs must not wait on eviction from that window Acceptance criteria: - design and implement the earliest safe host-side point to drain a just-closed run's console/transcript/meta into `~/.ward/agent-logs/<container>/` - preserve the current safety properties: no secret spill, best-effort drain, and no destructive interaction with live containers - make the behavior idempotent if the later stale-container sweep sees a run that was already drained - add or update tests around drain timing/order and duplicate-drain handling - update ward observability / cleanup docs to describe the new timing - link the infrastructure consumer issue: coilyco-flight-deck/infrastructure#424

coilyco-ops commented

2026-07-02 04:58:52 +00:00

Member

🔒 Reserved by ward agent --driver goose — container engineer-goose-ward-510 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T04:58:49Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Goose, via ward agent

🔒 Reserved by `ward agent --driver goose` — container `engineer-goose-ward-510` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T04:58:49Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Goose, via `ward agent`

coilyco-ops added the

headless

labels

2026-07-02 07:04:13 +00:00

coilyco-ops commented

2026-07-02 16:58:18 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-510 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T16:58:16Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-510` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T16:58:16Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 17:17:54 +00:00

feat(agent): drain closed run logs shortly after exit, not only at eviction

coilysiren closed this issue

2026-07-02 17:17:54 +00:00

coilysiren commented

2026-07-02 17:18:24 +00:00

Author

Owner

WARD-OUTCOME: done - detached container drain-exit waiter + full-set idempotent sweep land run logs shortly after exit; merged to main.

The shape of this one clicked once I accepted the real constraint: a detached ward agent run is fire-and-forget, so there is genuinely no host process alive to notice the container exit. That ruled out the tempting "just watch it" answers and pointed at a detached grandchild waiter blocking on docker wait. Spawning something that outlives its parent (Setsid + Release + nulled stdio) felt slightly heavier than a normal ward change, so I leaned on the keep-10 sweep as a second, idempotent drain point - which also happens to be exactly what acceptance criterion 3 asks for.

What fought back was entirely the pre-commit suite, not the logic: the ≤2-line comment-block rule and the 80-line/4000-char doc caps. Both target docs were already sitting right at the char cap, so every sentence I added had to be paid for by cutting one elsewhere - I ended up splitting the detail into a new docs/drain-timing.md. A subtle gotcha: a //-only line does NOT break a comment block for the linter, so blank lines had to be real blanks.

Confidence is high on the sweep/idempotency half (unit-tested, and I drove the full docker wait -> drain -> marker path end-to-end with a fake docker, including the secret-allowlist and the skip-on-marker case). The one thing untested by a real run is the detached spawn on a live Mac host - the argv/marker logic is unit-tested, but nobody has watched an actual orphaned waiter drain a real container yet. If it ever misbehaves, the sweep still backstops it, so the failure mode is "slightly later", not "lost". Possible follow-up: prune stale .drained/ sentinels for containers that no longer exist (today they are cleared on removal, which covers the normal path).

WARD-OUTCOME: done - detached `container drain-exit` waiter + full-set idempotent sweep land run logs shortly after exit; merged to main. The shape of this one clicked once I accepted the real constraint: a detached `ward agent` run is fire-and-forget, so there is genuinely no host process alive to notice the container exit. That ruled out the tempting "just watch it" answers and pointed at a detached grandchild waiter blocking on `docker wait`. Spawning something that outlives its parent (Setsid + Release + nulled stdio) felt slightly heavier than a normal ward change, so I leaned on the keep-10 sweep as a second, idempotent drain point - which also happens to be exactly what acceptance criterion 3 asks for. What fought back was entirely the pre-commit suite, not the logic: the ≤2-line comment-block rule and the 80-line/4000-char doc caps. Both target docs were already sitting right at the char cap, so every sentence I added had to be paid for by cutting one elsewhere - I ended up splitting the detail into a new `docs/drain-timing.md`. A subtle gotcha: a `//`-only line does NOT break a comment block for the linter, so blank lines had to be real blanks. Confidence is high on the sweep/idempotency half (unit-tested, and I drove the full `docker wait -> drain -> marker` path end-to-end with a fake docker, including the secret-allowlist and the skip-on-marker case). The one thing untested by a real run is the detached spawn on a live Mac host - the argv/marker logic is unit-tested, but nobody has watched an actual orphaned waiter drain a real container yet. If it ever misbehaves, the sweep still backstops it, so the failure mode is "slightly later", not "lost". Possible follow-up: prune stale `.drained/` sentinels for containers that no longer exist (today they are cleared on removal, which covers the normal path).

coilysiren referenced this issue

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283