ward agent: reap engineer containers idle >1h (no idle-killer exists today) #376

New issue

Closed

opened 2026-06-26 07:43:47 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-06-26 07:43:47 +00:00

Member

Problem

There is no idle-killer for warded containers. A ward agent engineer carry is fire-and-forget and detached, and an engineer container exits cleanly when its work is done (the fleet is full of Exited (0) engineer/issue containers). So an engineer that is still Up but has gone silent for a long stretch is not "working quietly" - it is wedged, holding a container slot, and sometimes spinning a core for nothing.

Today the only cleanup is the in-binary reaping of already-Exited containers on the next ward agent invocation. Nothing stops a still-running-but-wedged engineer. They are caught only by a human noticing and hand-running docker stop. We did exactly that earlier today to clear orphans.

Why log-staleness is a sound idle signal

The container entrypoint converts the headless claude stream-json feed into concise stdout log lines (entrypoint headless-progress block, ~L667). So container-log staleness is a true activity proxy for a headless engineer: a working agent streams a line per tool call / text event, and a wedged one goes silent. Combined with "engineers exit when done," an engineer idle past a threshold is unambiguously stuck and safe to stop.

Proposal

Add a host-side idle-reaper as a real ward feature (authored here, fleet rollout via infrastructure/ansible, not ward setup):

A ward agent reap verb (and/or a standing launchd-managed daemon) that periodically (~every 5m) sweeps docker ps --filter label=ward=true --filter label=ward.role=engineer.
For each, compute idle = now - last stdout log timestamp (docker logs --tail 1 --timestamps).
docker stop any engineer whose idle exceeds the threshold.
Consider a CPU guard (only stop if idle and low CPU) so a legit long quiet build/test is not killed mid-work - engineers can commit/push, unlike architects, so a false kill has a real cost. Worth deciding during impl.

Scope decision (this issue)

Target: ward.role=engineer only.
Idle threshold: 1h.
Leave architect / director / advisor untouched - those are interactive and idle by design (sitting at a prompt waiting for input is normal, not wedged). A separate, longer-grace policy for interactive roles can be a follow-up if the orphan-architect pattern recurs.

Notes for implementation

Containers already carry clean selector labels: ward=true, ward.role, ward.driver, ward.repo, ward.issue, ward.machine. No name-regex needed.
/var/run/docker.sock is already mounted into every warded container, and there is a TTL concept in flight (WARD_SUBSTRATE_TTL), so a self-watchdog-in-entrypoint variant is also feasible - but the host-side reaper is preferred (it can also catch a fully-wedged PID1 that a self-watchdog cannot).
Authoring vs rollout: the verb/logic is authored in ward, the fleet rollout (launchd timer / converge) is an ansible role in infrastructure, per the agentic-os authoring/rollout split.

Filed from a session that hand-cleaned 3 orphaned (architect) containers and confirmed the engineer-exits-when-done + log-staleness model against the live fleet.

## Problem There is no idle-killer for warded containers. A `ward agent engineer` carry is fire-and-forget and detached, and an engineer container **exits cleanly when its work is done** (the fleet is full of `Exited (0)` engineer/issue containers). So an engineer that is still `Up` but has gone silent for a long stretch is not "working quietly" - it is **wedged**, holding a container slot, and sometimes spinning a core for nothing. Today the only cleanup is the in-binary reaping of already-`Exited` containers on the next `ward agent` invocation. Nothing stops a still-running-but-wedged engineer. They are caught only by a human noticing and hand-running `docker stop`. We did exactly that earlier today to clear orphans. ## Why log-staleness is a sound idle signal The container entrypoint converts the headless `claude` `stream-json` feed into concise stdout log lines (entrypoint headless-progress block, ~L667). So **container-log staleness is a true activity proxy** for a headless engineer: a working agent streams a line per tool call / text event, and a wedged one goes silent. Combined with "engineers exit when done," an engineer idle past a threshold is unambiguously stuck and safe to stop. ## Proposal Add a host-side idle-reaper as a real ward feature (authored here, fleet rollout via infrastructure/ansible, not `ward setup`): - A `ward agent reap` verb (and/or a standing launchd-managed daemon) that periodically (~every 5m) sweeps `docker ps --filter label=ward=true --filter label=ward.role=engineer`. - For each, compute idle = `now - last stdout log timestamp` (`docker logs --tail 1 --timestamps`). - `docker stop` any engineer whose idle exceeds the threshold. - Consider a CPU guard (only stop if idle **and** low CPU) so a legit long quiet build/test is not killed mid-work - engineers can commit/push, unlike architects, so a false kill has a real cost. Worth deciding during impl. ## Scope decision (this issue) - **Target: `ward.role=engineer` only.** - **Idle threshold: 1h.** - **Leave `architect` / `director` / `advisor` untouched** - those are interactive and idle by design (sitting at a prompt waiting for input is normal, not wedged). A separate, longer-grace policy for interactive roles can be a follow-up if the orphan-architect pattern recurs. ## Notes for implementation - Containers already carry clean selector labels: `ward=true`, `ward.role`, `ward.driver`, `ward.repo`, `ward.issue`, `ward.machine`. No name-regex needed. - `/var/run/docker.sock` is already mounted into every warded container, and there is a TTL concept in flight (`WARD_SUBSTRATE_TTL`), so a self-watchdog-in-entrypoint variant is also feasible - but the host-side reaper is preferred (it can also catch a fully-wedged PID1 that a self-watchdog cannot). - Authoring vs rollout: the verb/logic is authored in ward, the fleet rollout (launchd timer / converge) is an ansible role in infrastructure, per the agentic-os authoring/rollout split. Filed from a session that hand-cleaned 3 orphaned (architect) containers and confirmed the engineer-exits-when-done + log-staleness model against the live fleet.

coilysiren added the

interactive

labels

2026-07-01 04:29:35 +00:00

coilyco-ops commented

2026-07-03 07:02:48 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-376 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T07:02:46Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-376` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T07:02:46Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-03 07:19:19 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-376 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T07:19:17Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-376` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T07:19:17Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-03 07:35:33 +00:00

feat(agent): add `ward agent reap` host-side idle-killer for wedged engineers (closes #376)

coilysiren closed this issue

2026-07-03 07:35:33 +00:00

coilysiren commented

2026-07-03 07:36:10 +00:00

Owner

WARD-OUTCOME: done - ward agent reap host-side idle-killer landed on main (559d6ed), stops running engineers idle past --idle (default 1h) with a --max-cpu guard.

Honestly this one went smoothly - the issue did most of the design work for me. The labels (ward=true, ward.role=engineer) meant no name-regex, and the existing dockerLogsCombined / exited-sweep code gave me the exact shape to mirror, so the docker plumbing was copy-the-pattern rather than invent-it.

The one real decision was the CPU guard. I settled on "the guard only ever spares" - a busy container above --max-cpu is spared as a live build/test, but an unreadable docker stats does NOT disable the reap. That keeps a broken stats probe from silently switching the whole idle-killer off, which felt like the safer failure mode given a false kill on an engineer that can commit/push has a real cost. I also added a start-time fallback so a fully-wedged PID1 that never logged a line is still caught, not just a mid-run silence.

What fought back: nothing in the code, but the documentation-layout char caps were brutal - FEATURES.md was ~20 chars under its 12500 cap before I touched it, so fitting the new entry meant trimming two neighboring mega-bullets. A bit of collateral prose-tightening, but nothing lost.

Confidence is high on the pure logic (unit-tested: timestamp parse, CPU parse, verdict matrix, skew clamp) and the wiring (help renders, roster correctly skips it as a meta verb). The one thing I could NOT exercise here is a real docker stop against a live wedged engineer - no docker daemon in this container - so the end-to-end sweep is verified only against the "no daemon / empty fleet" paths. Worth a human eyeballing the first real sweep on the fleet.

Follow-ups worth filing if wanted: the ansible rollout (launchd timer or --interval daemon) is explicitly out of scope here and still needs authoring in infrastructure; and the longer-grace interactive-role policy the issue mentions, if the orphan-architect pattern recurs.

WARD-OUTCOME: done - `ward agent reap` host-side idle-killer landed on main (559d6ed), stops running engineers idle past --idle (default 1h) with a --max-cpu guard. Honestly this one went smoothly - the issue did most of the design work for me. The labels (`ward=true`, `ward.role=engineer`) meant no name-regex, and the existing `dockerLogsCombined` / exited-sweep code gave me the exact shape to mirror, so the docker plumbing was copy-the-pattern rather than invent-it. The one real decision was the CPU guard. I settled on "the guard only ever spares" - a busy container above --max-cpu is spared as a live build/test, but an unreadable `docker stats` does NOT disable the reap. That keeps a broken stats probe from silently switching the whole idle-killer off, which felt like the safer failure mode given a false kill on an engineer that can commit/push has a real cost. I also added a start-time fallback so a fully-wedged PID1 that never logged a line is still caught, not just a mid-run silence. What fought back: nothing in the code, but the `documentation-layout` char caps were brutal - FEATURES.md was ~20 chars under its 12500 cap before I touched it, so fitting the new entry meant trimming two neighboring mega-bullets. A bit of collateral prose-tightening, but nothing lost. Confidence is high on the pure logic (unit-tested: timestamp parse, CPU parse, verdict matrix, skew clamp) and the wiring (help renders, roster correctly skips it as a meta verb). The one thing I could NOT exercise here is a real `docker stop` against a live wedged engineer - no docker daemon in this container - so the end-to-end sweep is verified only against the "no daemon / empty fleet" paths. Worth a human eyeballing the first real sweep on the fleet. Follow-ups worth filing if wanted: the ansible rollout (launchd timer or `--interval` daemon) is explicitly out of scope here and still needs authoring in infrastructure; and the longer-grace interactive-role policy the issue mentions, if the orphan-architect pattern recurs.