ward-container: headless agents silently hang when in-container claude -p can't auth (Keychain creds seeding)

coilyco-ops commented

2026-06-19 07:52:24 +00:00

Owner

Symptom

All 4 ward agent claude headless containers sat idle for 15-26 min producing zero output. Container logs stop dead at launching claude as uid 1000; no further progress ever streams.

Diagnosis (from host-side probing of a live container)

The in-container claude (2.1.178) is wedged in its own startup, before the first model turn - no session transcript written, /workspace clean, no ESTABLISHED :443 socket, the jq log-streamer blocked on pipe_read of an empty pipe.

Root cause is auth: in-container claude -p cannot authenticate with the seeded credentials. It produces zero bytes and exits 0 on a trivial prompt (even --output-format json --debug yields 0 stdout / 0 stderr), and hangs forever when stdin is the entrypoint's open pipe instead of /dev/null. Same root, two surface symptoms (hang vs silent no-op) depending on stdin.

Ruled out:

Infra - DNS resolves, egress to api.anthropic.com returns 405 in 0.08s from inside, no OOM/die events, default bridge net.
Onboarding - ~/.claude.json shows completed onboarding for the account.
Corrupt creds file - structure is normal (claudeAiOauth + nested mcpOAuth), 1488 bytes, all expected keys present. The token inside is just not usable in the container.

Likely culprit: on macOS hosts the credentials live in the Keychain, not a .credentials.json file (host has no such file). ward-container must synthesize /home/ubuntu/.claude/.credentials.json from the Keychain, and that synthesized token is unusable in the container (expired / wrong field / refresh failing silently).

Two bugs to fix

Silent-hang design bug (highest value). A headless agent that cannot start should crash the container loudly, not idle forever. Add a pre-launch auth smoke test in the entrypoint (e.g. claude -p "ping" --output-format json with stdin < /dev/null and a short timeout); if it returns empty / non-zero, abort the container with a clear error instead of launching the real agent into a silent hang. Also reap on no-output-within-timeout.
Credential seeding. Investigate the macOS Keychain -> .credentials.json synthesis; the seeded token is not usable inside the container. Verify token validity at seed time and fail seeding if a smoke test fails.

Secondary: the agent invocation leaves stdin as an open pipe, so a non-functional claude -p hangs instead of exiting. Redirect agent stdin from /dev/null so failures surface fast.

Affected run

4 containers wedged: ward#218, cli-guard#157, agentic-os-kai#687, agentic-os#260 (issue 218 was launched non-headless - plain claude, no -p --output-format stream-json). No work lost - all /workspace trees clean.

## Symptom All 4 `ward agent claude` headless containers sat **idle for 15-26 min** producing zero output. Container logs stop dead at `launching claude as uid 1000`; no further progress ever streams. ## Diagnosis (from host-side probing of a live container) The in-container `claude` (2.1.178) is wedged in its **own startup, before the first model turn** - no session transcript written, `/workspace` clean, no ESTABLISHED `:443` socket, the `jq` log-streamer blocked on `pipe_read` of an empty pipe. Root cause is **auth**: in-container `claude -p` cannot authenticate with the seeded credentials. It produces **zero bytes and exits 0** on a trivial prompt (even `--output-format json --debug` yields 0 stdout / 0 stderr), and **hangs forever** when stdin is the entrypoint's open pipe instead of `/dev/null`. Same root, two surface symptoms (hang vs silent no-op) depending on stdin. Ruled out: - **Infra** - DNS resolves, egress to api.anthropic.com returns 405 in 0.08s from inside, no OOM/die events, default bridge net. - **Onboarding** - `~/.claude.json` shows completed onboarding for the account. - **Corrupt creds file** - structure is normal (`claudeAiOauth` + nested `mcpOAuth`), 1488 bytes, all expected keys present. The token inside is just not usable in the container. Likely culprit: on **macOS hosts the credentials live in the Keychain, not a `.credentials.json` file** (host has no such file). ward-container must *synthesize* `/home/ubuntu/.claude/.credentials.json` from the Keychain, and that synthesized token is unusable in the container (expired / wrong field / refresh failing silently). ## Two bugs to fix 1. **Silent-hang design bug (highest value).** A headless agent that cannot start should **crash the container loudly**, not idle forever. Add a pre-launch **auth smoke test** in the entrypoint (e.g. `claude -p "ping" --output-format json` with stdin `< /dev/null` and a short timeout); if it returns empty / non-zero, abort the container with a clear error instead of launching the real agent into a silent hang. Also reap on no-output-within-timeout. 2. **Credential seeding.** Investigate the macOS Keychain -> `.credentials.json` synthesis; the seeded token is not usable inside the container. Verify token validity at seed time and fail seeding if a smoke test fails. Secondary: the agent invocation leaves stdin as an open pipe, so a non-functional `claude -p` hangs instead of exiting. Redirect agent stdin from `/dev/null` so failures surface fast. ## Affected run 4 containers wedged: ward#218, cli-guard#157, agentic-os-kai#687, agentic-os#260 (issue 218 was launched non-headless - plain `claude`, no `-p --output-format stream-json`). No work lost - all `/workspace` trees clean.

coilyco-ops commented

2026-06-19 07:59:11 +00:00

Author

Owner

Canary relaunch confirms the bug is deterministic

After killing the 4 wedged containers, relaunched a single canary: ward agent claude headless --force coilyco-flight-deck/agentic-os#260 (fresh image pull, fresh clone, creds re-seeded from the current host Keychain - same host token that works for the interactive session right now).

Result: identical wedge. claude (-p --verbose --output-format stream-json) launched, ran 4+ min at <1% CPU in S (sleeping), produced zero stream-json output (container log frozen at the launching claude as uid 1000 line, never grew past 30 lines). Killed it.

Takeaways:

The failure is deterministic, not a transient/expiry-timing fluke - a fresh seed from a known-good host token reproduces it.
So no relaunch will stick until the seeding/auth path is fixed. The pre-launch auth smoke test (bug 1) is the priority: it would have aborted this canary in seconds with a clear error instead of another 4-min silent hang.

Note: killing a wedged container leaves a stale remote reservation on the issue (held by @coilyco-ops); the next launch needs --force to reclaim it. Worth auto-releasing the hold on container death/reap.

## Canary relaunch confirms the bug is deterministic After killing the 4 wedged containers, relaunched a single canary: `ward agent claude headless --force coilyco-flight-deck/agentic-os#260` (fresh image pull, fresh clone, creds re-seeded from the current host Keychain - same host token that works for the interactive session right now). Result: **identical wedge.** claude (`-p --verbose --output-format stream-json`) launched, ran 4+ min at <1% CPU in `S (sleeping)`, produced **zero** stream-json output (container log frozen at the `launching claude as uid 1000` line, never grew past 30 lines). Killed it. Takeaways: - The failure is **deterministic**, not a transient/expiry-timing fluke - a fresh seed from a known-good host token reproduces it. - So **no relaunch will stick until the seeding/auth path is fixed.** The pre-launch auth smoke test (bug 1) is the priority: it would have aborted this canary in seconds with a clear error instead of another 4-min silent hang. Note: killing a wedged container leaves a **stale remote reservation** on the issue (held by @coilyco-ops); the next launch needs `--force` to reclaim it. Worth auto-releasing the hold on container death/reap.

coilysiren referenced this issue from a commit

2026-06-19 09:24:43 +00:00

fix(container): fail loud when a headless agent can't authenticate (ward#222)

coilyco-ops commented

2026-06-19 09:28:34 +00:00

Author

Owner

Fixes landed in v0.94.0 + live repro result

All three fixes landed on main (e74d918) and released as v0.94.0:

Pre-launch auth smoke test (entrypoint, headless-claude only, WARD_SMOKE_TEST_SKIP=1 bypass).
One-shot (headless/ask) agent stdin pinned to /dev/null.
Host-side claudeCredsHealth warning (empty / no-token / expired).

Live repro (fresh headless dispatch on v0.94.0)

Dispatched ward agent claude headless coilyco-flight-deck/agentic-os#260:

No host-side creds-health warning at dispatch -> the host OAuth token is currently valid (non-empty, has access token, not expired).
Smoke test fired and passed: auth smoke test: claude responded, auth OK.
Agent then launched and worked normally (ToolSearch / WebFetch / Bash / Read streaming) - no wedge. (Canary killed after the verdict; it was a repro, not a real #260 dispatch.)

What this says about the seeding root cause

The wedge is not currently reproducible - the same creds path now authenticates fine. The original failure (~07:11Z) was almost certainly a transient unusable credential (token expired / mid-rotation in that window; the host has since refreshed it through normal use), not a persistent Keychain->.credentials.json synthesis bug. The earlier symptom (claude -p emitting zero bytes + exit 0 rather than an auth error) is consistent with that.

Net: the silent-hang class is now defended - any recurrence aborts loudly with a clear message + host-side expiry hint, instead of idling for 25 min. Leaving this open only to confirm the host-side expiry warning fires correctly the next time a token genuinely expires; the deterministic fixes are done.

## Fixes landed in v0.94.0 + live repro result All three fixes landed on main (`e74d918`) and released as **v0.94.0**: - Pre-launch auth smoke test (entrypoint, headless-claude only, `WARD_SMOKE_TEST_SKIP=1` bypass). - One-shot (headless/ask) agent stdin pinned to `/dev/null`. - Host-side `claudeCredsHealth` warning (empty / no-token / expired). ## Live repro (fresh headless dispatch on v0.94.0) Dispatched `ward agent claude headless coilyco-flight-deck/agentic-os#260`: - **No** host-side creds-health warning at dispatch -> the host OAuth token is currently valid (non-empty, has access token, not expired). - Smoke test fired and **passed**: `auth smoke test: claude responded, auth OK`. - Agent then launched and **worked normally** (ToolSearch / WebFetch / Bash / Read streaming) - no wedge. (Canary killed after the verdict; it was a repro, not a real #260 dispatch.) ## What this says about the seeding root cause The wedge is **not currently reproducible** - the same creds path now authenticates fine. The original failure (~07:11Z) was almost certainly a **transient unusable credential** (token expired / mid-rotation in that window; the host has since refreshed it through normal use), not a persistent Keychain->`.credentials.json` synthesis bug. The earlier symptom (claude `-p` emitting zero bytes + exit 0 rather than an auth error) is consistent with that. Net: the silent-hang class is now **defended** - any recurrence aborts loudly with a clear message + host-side expiry hint, instead of idling for 25 min. Leaving this open only to confirm the host-side expiry warning fires correctly the next time a token genuinely expires; the deterministic fixes are done.

coilyco-ops added the

labels

2026-06-24 04:15:49 +00:00

coilyco-ops commented

2026-06-24 04:34:13 +00:00

Author

Owner

Fresh repro 2026-06-24: two findings

1. work mode bypasses the smoke test and hangs silently. ward agent claude work <ref> (run detached, as a fan-out dispatcher does) launches claude in interactive mode; in a TTY-less container it sits at launching claude as uid 1000 producing zero output indefinitely (repro: 14 containers, 11-24 min, no progress). The v0.94 smoke-test + stdin->/dev/null guards only fire on the headless path, so work has no protection and reproduces the original silent-hang. Suggest work either refuses a detached/no-TTY run or inherits the same smoke-test + stdin pin.

2. The seeding is failing again on headless. A fresh ward agent claude headless --force coilyco-flight-deck/ward#177 fired the smoke test and it failed: claude -p did not respond within 90s - credentials are unusable in-container - despite the host OAuth token being timestamp-valid (expiresAt ~6h out, refreshToken present, subscriptionType max). So the silent-hang defense works (aborts loudly, clean reap), but the underlying OAuth snapshot is unusable in-container again. Refreshing the host login is the prescribed workaround; worth deciding whether Max OAuth snapshots have stopped working headless-in-container (vs needing an explicit API-key path), since timestamp-validity is not a sufficient health signal.

## Fresh repro 2026-06-24: two findings **1. `work` mode bypasses the smoke test and hangs silently.** `ward agent claude work <ref>` (run detached, as a fan-out dispatcher does) launches claude in **interactive** mode; in a TTY-less container it sits at `launching claude as uid 1000` producing zero output indefinitely (repro: 14 containers, 11-24 min, no progress). The v0.94 smoke-test + stdin->/dev/null guards only fire on the `headless` path, so `work` has no protection and reproduces the original silent-hang. Suggest `work` either refuses a detached/no-TTY run or inherits the same smoke-test + stdin pin. **2. The seeding is failing again on `headless`.** A fresh `ward agent claude headless --force coilyco-flight-deck/ward#177` fired the smoke test and it **failed**: `claude -p did not respond within 90s - credentials are unusable in-container` - despite the host OAuth token being timestamp-valid (expiresAt ~6h out, refreshToken present, subscriptionType max). So the silent-hang defense works (aborts loudly, clean reap), but the underlying OAuth snapshot is unusable in-container again. Refreshing the host login is the prescribed workaround; worth deciding whether Max OAuth snapshots have stopped working headless-in-container (vs needing an explicit API-key path), since timestamp-validity is not a sufficient health signal.

coilyco-ops commented

2026-06-24 17:27:28 +00:00

Author

Owner

Re-triage (goose-triage method, claude-macos-kais-macbook-pro-2 as the judgment engine, 2026-06-24)

P1 - Headless containers sit idle on an auth wedge, blocks the dispatch path. Diagnosed, near-term.
interactive - Live auth/dispatch bug, agent investigates and pauses for the human verify.

**Re-triage** (goose-triage method, claude-macos-kais-macbook-pro-2 as the judgment engine, 2026-06-24) - **P1** - Headless containers sit idle on an auth wedge, blocks the dispatch path. Diagnosed, near-term. - **interactive** - Live auth/dispatch bug, agent investigates and pauses for the human verify.

coilyco-ops commented

2026-06-24 17:29:48 +00:00

Author

Owner

Re-triage (goose-triage method, claude-macos-kais-macbook-pro-2 as the judgment engine, 2026-06-24)

P1 - Headless containers sit idle on an auth wedge, blocks the dispatch path. Diagnosed, near-term.
interactive - Live auth/dispatch bug, agent investigates and pauses for the human verify.

**Re-triage** (goose-triage method, claude-macos-kais-macbook-pro-2 as the judgment engine, 2026-06-24) - **P1** - Headless containers sit idle on an auth wedge, blocks the dispatch path. Diagnosed, near-term. - **interactive** - Live auth/dispatch bug, agent investigates and pauses for the human verify.

coilyco-ops commented

2026-07-01 23:59:33 +00:00

Author

Owner

Fleet-readiness note: surfaced early for tonight's burndown session (silent auth hangs can eat the whole session window). Run right after #301. Recorded by Claude Code (Fable) during the 2026-07-01 ward launch triage session with Kai.

coilyco-ops referenced this issue

2026-07-02 00:01:42 +00:00

meta: jul 1st evening sprint - launch burndown session 1 #492

coilysiren commented

2026-07-02 02:20:05 +00:00

Owner

Fleet-health canary result - tonight's burndown (ward#492)

The session's first headless dispatch (engineer-claude-ward-301, 02:11Z via the TCP broker, current :latest image, host kais-macbook-pro-2) ran clean end to end: seeded, authenticated, worked, pushed 893d70f, closed its issue in ~6 minutes. Datapoints against this issue's two failure classes, observed from inside that run:

No silent hang - the in-container claude auth path worked first try on the current host credential snapshot. The v0.94.0 smoke-test defense did not need to fire.
Cred seeding healthy - /etc/ward-git-credentials came up 0640 root:agent, readable by the dropped uid-1000 agent, store helper wired to the bot. Committer on the push is coilyco-ops@coilysiren.me, so attribution is correct.

Headless dispatch is healthy for tonight's session window. Leaving this open per its interactive triage label - the work-mode smoke-test bypass (2026-06-24 comment) and the 'timestamp-valid but unusable snapshot' question are still unresolved and stay with this ticket.

Recorded by Claude Code (Fable) during the ward#492 burndown session.

## Fleet-health canary result - tonight's burndown (ward#492) The session's first headless dispatch (`engineer-claude-ward-301`, 02:11Z via the TCP broker, current :latest image, host kais-macbook-pro-2) ran clean end to end: seeded, authenticated, worked, pushed `893d70f`, closed its issue in ~6 minutes. Datapoints against this issue's two failure classes, observed from inside that run: * **No silent hang** - the in-container claude auth path worked first try on the current host credential snapshot. The v0.94.0 smoke-test defense did not need to fire. * **Cred seeding healthy** - `/etc/ward-git-credentials` came up 0640 root:agent, readable by the dropped uid-1000 agent, store helper wired to the bot. Committer on the push is `coilyco-ops@coilysiren.me`, so attribution is correct. Headless dispatch is healthy for tonight's session window. Leaving this open per its interactive triage label - the `work`-mode smoke-test bypass (2026-06-24 comment) and the 'timestamp-valid but unusable snapshot' question are still unresolved and stay with this ticket. Recorded by Claude Code (Fable) during the ward#492 burndown session.

coilysiren referenced this issue

2026-07-02 07:10:55 +00:00

meta: jul 1st evening sprint - launch burndown session 1 #492

coilysiren referenced this issue

2026-07-05 02:37:43 +00:00

Dispatch UX: make a failed run self-documenting on the tracker (verbose reservation comment + enriched release comment + docker-log backstop) #609

coilysiren closed this issue

2026-07-06 20:03:23 +00:00

coilyco-ops referenced this issue

2026-07-08 22:19:00 +00:00

Make agent workflow defaults configurable and default engineers to PRs #707