ward agent headless: silenced docker pull + no timeout makes a slow/mid-push registry an invisible startup hang #322

New issue

Closed

opened 2026-06-25 08:37:40 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-06-25 08:37:40 +00:00

Member

Problem

A detached ward agent headless run silences all docker pull output (ward#306, runDockerSilenced with silenceStderr=true in pullAgentImage / agent.go). It also has no pull timeout. So when the registry is slow - most commonly when a freshly-built :latest is still propagating into the Forgejo registry - the run prints the carrying ... line and then hangs with zero feedback: no progress bar, no "pulling", no error, no timeout. It looks permanently wedged at startup, and the operator has no way to attribute the stall to the pull.

Hit live on 2026-06-25: the agentic-os:latest image was rebuilt at 08:22:26Z, and two warded ...ward#320 dispatches at 08:25:50Z (sibling) and 08:28:34Z (manual) both hung on the silenced pull while the new image was mid-push. Both got past pre-flight and reserveIssue (reservation comments posted) but never reached docker run. Re-dispatching ~8 min later (image fully propagated) launched instantly.

Why it bites

The whole point of silencing the pull (ward#306) is to keep a detached launch quiet. But "quiet on success" became "indistinguishable from a hang on a slow registry". The operator cannot tell a wedged pull from a wedged anything-else.

Possible fixes (pick one or more)

Print a single attributable line before the silenced pull, e.g. ... : pulling <image> (silenced; this can stall on a mid-push registry), so a stall has a name.
Add a pull timeout (context deadline) so a stuck pull fails loudly with a clear "registry pull timed out" instead of hanging forever; the run already tolerates a failed pull by falling back to the local image.
For detached runs, emit a periodic heartbeat while the pull is in flight.

Acceptance

A detached ward agent headless dispatch hitting a slow/unreachable registry produces an attributable signal (a named line and/or a bounded-time error), not an indefinite silent hang.

Found while diagnosing a silent-startup-hang on ward#320 (the dispatch itself succeeded on retry).

## Problem A detached `ward agent headless` run silences all `docker pull` output (ward#306, `runDockerSilenced` with `silenceStderr=true` in `pullAgentImage` / `agent.go`). It also has **no pull timeout**. So when the registry is slow - most commonly when a freshly-built `:latest` is still propagating into the Forgejo registry - the run prints the `carrying ...` line and then hangs with **zero feedback**: no progress bar, no "pulling", no error, no timeout. It looks permanently wedged at startup, and the operator has no way to attribute the stall to the pull. Hit live on 2026-06-25: the `agentic-os:latest` image was rebuilt at 08:22:26Z, and two `warded ...ward#320` dispatches at 08:25:50Z (sibling) and 08:28:34Z (manual) both hung on the silenced pull while the new image was mid-push. Both got past pre-flight and `reserveIssue` (reservation comments posted) but never reached `docker run`. Re-dispatching ~8 min later (image fully propagated) launched instantly. ## Why it bites The whole point of silencing the pull (ward#306) is to keep a detached launch quiet. But "quiet on success" became "indistinguishable from a hang on a slow registry". The operator cannot tell a wedged pull from a wedged anything-else. ## Possible fixes (pick one or more) 1. Print a single attributable line before the silenced pull, e.g. `... : pulling <image> (silenced; this can stall on a mid-push registry)`, so a stall has a name. 2. Add a pull timeout (context deadline) so a stuck pull fails loudly with a clear "registry pull timed out" instead of hanging forever; the run already tolerates a failed pull by falling back to the local image. 3. For detached runs, emit a periodic heartbeat while the pull is in flight. ## Acceptance - A detached `ward agent headless` dispatch hitting a slow/unreachable registry produces an attributable signal (a named line and/or a bounded-time error), not an indefinite silent hang. Found while diagnosing a silent-startup-hang on ward#320 (the dispatch itself succeeded on retry).

coilysiren commented

2026-06-25 08:39:12 +00:00

Owner

I pick #1 and #3

coilyco-ops commented

2026-06-25 08:39:41 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container ward-ward-issue-322-claude-74c48442 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-25T08:39:39Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `ward-ward-issue-322-claude-74c48442` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-25T08:39:39Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-06-25 08:49:27 +00:00

feat(agent): name + heartbeat the silenced detached pull (ward#322)

coilysiren closed this issue

2026-06-25 08:49:27 +00:00

coilysiren commented

2026-06-25 08:50:21 +00:00

Owner

Landed #1 and #3 as you picked: a named line before the silenced pull, plus a 30s still pulling heartbeat on a goroutine that drains on stop so no stray line lands after the pull returns.

The Go was the easy part here. Small, and clean under -race. What actually fought back was the doc budget: docs/agent-flags.md was already sitting at 3971/4000 chars, so the new note simply didn't fit and I ended up condensing unrelated --new-tab / --details prose to claw back room. The other snag was the container itself. trufflehog wasn't on PATH, so the pre-commit gate refused the commit. I pulled the release binary into ~/go/bin and prepended PATH rather than reach for --no-verify.

Fairly confident in the behavior. One honest edge worth naming: since we deliberately skipped the timeout (#2), a genuinely unreachable registry still hangs. It just hangs out loud now, ticking with a named signal instead of dead silence. If a hard bound ever feels worth it, that's the natural follow-up.

Claude (she/her), via ward agent

Landed #1 and #3 as you picked: a named line before the silenced pull, plus a 30s `still pulling` heartbeat on a goroutine that drains on stop so no stray line lands after the pull returns. The Go was the easy part here. Small, and clean under `-race`. What actually fought back was the doc budget: `docs/agent-flags.md` was already sitting at 3971/4000 chars, so the new note simply didn't fit and I ended up condensing unrelated `--new-tab` / `--details` prose to claw back room. The other snag was the container itself. trufflehog wasn't on PATH, so the pre-commit gate refused the commit. I pulled the release binary into `~/go/bin` and prepended PATH rather than reach for `--no-verify`. Fairly confident in the behavior. One honest edge worth naming: since we deliberately skipped the timeout (#2), a genuinely unreachable registry still hangs. It just hangs out loud now, ticking with a named signal instead of dead silence. If a hard bound ever feels worth it, that's the natural follow-up. - Claude (she/her), via `ward agent`

coilyco-ops referenced this issue

2026-06-25 09:22:47 +00:00

Extract the small reusable helpers to cli-guard (junk-scan, settings-installer, hook tables, attribution, flock, semver, owner-trust, issue-ref) #269