coilyco-flight-deck/ward

Fork 0

fix(agent): fail fast when docker is snap-confined (private /tmp breaks --env-file and -v mounts) #557

New issue

Closed

opened 2026-07-03 07:59:07 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-03 07:59:07 +00:00

Member

Symptom

A warded director -> read-only surface bring-up on kai-server dies at docker run with:

docker: open /tmp/ward-forgejo-env-2612683732: no such file or directory
exit status 125

The container never launches. The same wall blocks every container mode (surface, interactive, headless dispatch) on this host.

Root cause

kai-server's docker is the snap package (/snap/bin/docker, canonical rev 3505). Snap confinement gives the docker CLI a private /tmp and only the home interface connected (snap connections docker shows home connected, removable-media not). So when ward stages a host-/tmp file and hands the path to the snap docker CLI, that CLI enters its own empty /tmp and the open fails.

Ward hands docker four host-/tmp paths that snap docker cannot reach:

--env-file - cmd/ward/container.go:255 (os.CreateTemp("", "ward-forgejo-env-*")). This is the one that fired.
-v container-assets mount - cmd/ward/container.go:404 (os.MkdirTemp("", containerAssetsPrefix+"*")).
dispatch clone dir - cmd/ward/git_clone.go clones to /tmp/$TMPDIR, then bind-mounts it.
broker bridge socket - cmd/ward/container_bootstrap.go:438 (const bridge = "/tmp/docker-agent.sock").

Ward assumes the docker CLI shares the host filesystem view. That holds for docker-ce and Docker-Desktop-on-Mac, but not for snap-confined docker on Linux.

Proposed fix (this issue)

Add a preflight that detects snap-confined docker and fails fast with a remediation message, instead of the raw mid-launch ENOENT. Detection signals: resolved docker binary under /snap/, or docker info reporting a snap install. Message should name the cause (snap private /tmp) and the fix (install docker-ce from Docker's apt repo, or connect an interface that exposes host paths - which snap docker cannot do for /tmp).

This converts a cryptic exit-125 into an actionable error and protects any Linux host that happens to have snap docker.

Out of scope / follow-up

Relocating ward's docker-facing temp staging under a docker-visible dir (non-hidden $HOME subdir) would let snap docker work, but the home interface excludes dot-dirs and there are four distinct touchpoints (env-file, assets, clone, unix socket), so fail-fast + "use docker-ce" is the right v1. The host-side remediation for kai-server (swap snap docker -> docker-ce via ansible) is tracked in coilyco-flight-deck/infrastructure.

Evidence

command -v docker -> /snap/bin/docker; snap list docker -> 29.3.1 rev 3505 canonical**.
snap connections docker -> only home connected.
No docker-ce present (/usr/bin/docker, /usr/bin/dockerd absent), so it is not a PATH shadow.

Diagnosed from a director surface on kai-server, 2026-07-03.

## Symptom A `warded director` -> read-only surface bring-up on kai-server dies at `docker run` with: ``` docker: open /tmp/ward-forgejo-env-2612683732: no such file or directory exit status 125 ``` The container never launches. The same wall blocks every container mode (surface, interactive, headless dispatch) on this host. ## Root cause kai-server's `docker` is the **snap package** (`/snap/bin/docker`, canonical rev 3505). Snap confinement gives the docker CLI a **private `/tmp`** and only the `home` interface connected (`snap connections docker` shows `home` connected, `removable-media` not). So when ward stages a host-`/tmp` file and hands the path to the snap docker CLI, that CLI enters its own empty `/tmp` and the open fails. Ward hands docker four host-`/tmp` paths that snap docker cannot reach: - **`--env-file`** - `cmd/ward/container.go:255` (`os.CreateTemp("", "ward-forgejo-env-*")`). This is the one that fired. - **`-v` container-assets mount** - `cmd/ward/container.go:404` (`os.MkdirTemp("", containerAssetsPrefix+"*")`). - **dispatch clone dir** - `cmd/ward/git_clone.go` clones to `/tmp`/`$TMPDIR`, then bind-mounts it. - **broker bridge socket** - `cmd/ward/container_bootstrap.go:438` (`const bridge = "/tmp/docker-agent.sock"`). Ward assumes the docker CLI shares the host filesystem view. That holds for docker-ce and Docker-Desktop-on-Mac, but not for snap-confined docker on Linux. ## Proposed fix (this issue) Add a **preflight that detects snap-confined docker and fails fast with a remediation message**, instead of the raw mid-launch ENOENT. Detection signals: resolved docker binary under `/snap/`, or `docker info` reporting a snap install. Message should name the cause (snap private /tmp) and the fix (install docker-ce from Docker's apt repo, or connect an interface that exposes host paths - which snap docker cannot do for /tmp). This converts a cryptic exit-125 into an actionable error and protects any Linux host that happens to have snap docker. ## Out of scope / follow-up Relocating ward's docker-facing temp staging under a docker-visible dir (non-hidden `$HOME` subdir) would let snap docker work, but the `home` interface excludes dot-dirs and there are four distinct touchpoints (env-file, assets, clone, unix socket), so fail-fast + "use docker-ce" is the right v1. The host-side remediation for kai-server (swap snap docker -> docker-ce via ansible) is tracked in coilyco-flight-deck/infrastructure. ## Evidence - `command -v docker` -> `/snap/bin/docker`; `snap list docker` -> `29.3.1 rev 3505 canonical**`. - `snap connections docker` -> only `home` connected. - No docker-ce present (`/usr/bin/docker`, `/usr/bin/dockerd` absent), so it is not a PATH shadow. Diagnosed from a director surface on kai-server, 2026-07-03.

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-07-03 07:59:43 +00:00

feat(ansible): provision docker-ce on kai-server, replacing snap docker (private /tmp blocks warded) #450

coilyco-ops added the

headless

labels

2026-07-03 17:30:29 +00:00

coilysiren referenced this issue

2026-07-03 17:54:25 +00:00

warded advisor dispatch 125s on host docker run (research mode) while engineer runs succeed same session #577

coilyco-ops referenced this issue

2026-07-03 18:08:33 +00:00

warded advisor dispatch 125s on host docker run (research mode) while engineer runs succeed same session #577

coilyco-ops commented

2026-07-04 08:31:34 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-557 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T08:31:28Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-557` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T08:31:28Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-04 08:43:26 +00:00

fix(agent): fail fast when docker is snap-confined instead of a raw exit-125

coilysiren closed this issue

2026-07-04 08:43:26 +00:00

coilysiren commented

2026-07-04 08:44:04 +00:00

Owner

WARD-OUTCOME: done - added a snap-confined-docker fail-fast preflight at the shared createAgentContainer launch chokepoint.

This one was cleaner than I expected going in, mostly because so much of the groundwork already existed. What surprised me: #569/#574 had already relocated the env-file and assets dir to $HOME for exactly this snap /tmp problem, so the specific ENOENT this issue quotes was already dead. I paused on that — was the fail-fast still wanted, or superseded? Two things settled it: the remaining touchpoints (the ward git clone /tmp checkout and the /tmp/docker-agent.sock broker bridge) genuinely still break on snap docker, and the issue itself explicitly picks "fail-fast + use docker-ce" over chasing every touchpoint. So I honored the issue.

The non-obvious bit was placement: the reported symptom is the director surface bring-up, which calls createAgentContainer directly and bypasses launchAgentContainer (where the sibling blocked() check lives). Putting the guard in createAgentContainer covers all four modes (engineer/task/advisor/director) in one spot. Detection is path-based (resolved binary under /snap, or a PATH shim whose symlink chain ends at the snap wrapper) rather than shelling out to docker info — deterministic and unit-testable.

Confidence is high on the pure detection (covered by tests) and the wiring compiles/vets/tests clean. The one thing I could not exercise here is the real failure on an actual snap host — I have no snap docker in this container — so that path is verified by construction and unit test, not live. Only rough edge worth a follow-up: the two remaining /tmp touchpoints are now guarded-against rather than fixed; if someone ever wants snap docker to actually work, relocating those is the tracked follow-up (and the host-side docker-ce swap lives in infrastructure).

WARD-OUTCOME: done - added a snap-confined-docker fail-fast preflight at the shared createAgentContainer launch chokepoint. This one was cleaner than I expected going in, mostly because so much of the groundwork already existed. What surprised me: #569/#574 had already relocated the env-file and assets dir to `$HOME` for exactly this snap `/tmp` problem, so the *specific* ENOENT this issue quotes was already dead. I paused on that — was the fail-fast still wanted, or superseded? Two things settled it: the remaining touchpoints (the `ward git clone` /tmp checkout and the `/tmp/docker-agent.sock` broker bridge) genuinely still break on snap docker, and the issue itself explicitly picks "fail-fast + use docker-ce" over chasing every touchpoint. So I honored the issue. The non-obvious bit was placement: the reported symptom is the *director surface* bring-up, which calls `createAgentContainer` directly and bypasses `launchAgentContainer` (where the sibling `blocked()` check lives). Putting the guard in `createAgentContainer` covers all four modes (engineer/task/advisor/director) in one spot. Detection is path-based (resolved binary under `/snap`, or a PATH shim whose symlink chain ends at the `snap` wrapper) rather than shelling out to `docker info` — deterministic and unit-testable. Confidence is high on the pure detection (covered by tests) and the wiring compiles/vets/tests clean. The one thing I could not exercise here is the real failure on an actual snap host — I have no snap docker in this container — so that path is verified by construction and unit test, not live. Only rough edge worth a follow-up: the two remaining /tmp touchpoints are now guarded-against rather than fixed; if someone ever wants snap docker to actually *work*, relocating those is the tracked follow-up (and the host-side docker-ce swap lives in infrastructure).