engineer launch fails under snap docker: --env-file staged in /tmp is invisible to confined docker client

coilyco-ops commented

2026-07-03 17:41:41 +00:00

Owner

Symptom

On kai-server (native Linux), ward agent engineer <ref> fails at container launch:

ward agent engineer --driver claude: wrote launch env file for coilyco-flight-deck/agentic-os#322
docker: open /tmp/ward-forgejo-env-3189451037: no such file or directory
exit status 125

Everything up to docker run works: ref resolves, reservation acquired (local + remote comment), prompt seeded, image pulled. Only the final docker run dies.

Root cause

docker on this host is the snap package (/snap/bin/docker). Snap confinement gives the docker client a private /tmp namespace, so any path ward hands it under the host /tmp is invisible. Ward stages two things there:

the Forgejo-token --env-file /tmp/ward-forgejo-env-* (the one that errors first)
the container assets bind-mount -v /tmp/ward-container-assets-*:/opt/ward:ro (would fail next)

Minimal proof: host writes and cats /tmp/ward-snap-probe.N; docker run --env-file /tmp/ward-snap-probe.N ... returns the identical open ...: no such file or directory.

Fix options

ward-side (portable): stage launch assets under $HOME (e.g. ~/.ward/launch/<container>/) instead of /tmp. Snap docker can read $HOME by default, and this makes ward robust to a confined docker regardless of packaging. Preferred.
host-side: replace snap docker with docker-ce from the apt repo. Belongs in infra/ansible fleet convergence, not ward, but would unblock kai-server immediately.

Discovered running a smoke-test engineer dispatch (agentic-os#322). No container ever launched; nothing landed.

## Symptom On kai-server (native Linux), `ward agent engineer <ref>` fails at container launch: ``` ward agent engineer --driver claude: wrote launch env file for coilyco-flight-deck/agentic-os#322 docker: open /tmp/ward-forgejo-env-3189451037: no such file or directory exit status 125 ``` Everything up to `docker run` works: ref resolves, reservation acquired (local + remote comment), prompt seeded, image pulled. Only the final `docker run` dies. ## Root cause `docker` on this host is the **snap** package (`/snap/bin/docker`). Snap confinement gives the docker client a **private `/tmp` namespace**, so any path ward hands it under the host `/tmp` is invisible. Ward stages two things there: - the Forgejo-token `--env-file /tmp/ward-forgejo-env-*` (the one that errors first) - the container assets bind-mount `-v /tmp/ward-container-assets-*:/opt/ward:ro` (would fail next) Minimal proof: host writes and cats `/tmp/ward-snap-probe.N`; `docker run --env-file /tmp/ward-snap-probe.N ...` returns the identical `open ...: no such file or directory`. ## Fix options 1. **ward-side (portable):** stage launch assets under `$HOME` (e.g. `~/.ward/launch/<container>/`) instead of `/tmp`. Snap docker can read $HOME by default, and this makes ward robust to a confined docker regardless of packaging. Preferred. 2. **host-side:** replace snap docker with docker-ce from the apt repo. Belongs in infra/ansible fleet convergence, not ward, but would unblock kai-server immediately. Discovered running a smoke-test engineer dispatch (agentic-os#322). No container ever launched; nothing landed.

coilyco-ops commented

2026-07-03 17:46:19 +00:00

Author

Owner

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-574 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T17:46:13Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

The pre-flight judged this issue GO for an unattended run. Its justification:

pre-flight read (GO)

Context to front-load: Before my first edit in the fresh coilyco-flight-deck/ward clone, I will READ: (1) the docker-launch codepath that stages --env-file /tmp/ward-forgejo-env-* and -v /tmp/ward-container-assets-*:/opt/ward:ro — grep for ward-forgejo-env and ward-container-assets to find both temp-staging sites and their cleanup/teardown logic; (2) docs/agent.md and docs/agent-preflight.md for the engineer dispatch + headless launch flow; (3) .agents/skills/tooling-ward-agent/SKILL.md for ref-normalization and dispatch conventions; (4) .ward/ward.yaml for the build/test/lint verbs I must route through, plus existing test patterns around launch-env staging. I will read each in the clone, not discover them lazily.

The fix is well-diagnosed and self-contained: move launch-asset staging from /tmp to ~/.ward/launch/<container>/ (option 1, explicitly preferred) so snap-confined docker can read it under $HOME. Main risk/unknown is completeness and hygiene rather than feasibility — I must catch both staging sites (env-file and assets bind-mount), preserve secure file permissions on the Forgejo-token env-file (0600, since $HOME is longer-lived than /tmp), and get cleanup right so I don't leak per-container dirs; a secondary unknown is whether the container-assets mount has snap-specific path expectations I can only confirm by reading the code. None of that needs a human — it's verifiable by reading the launch code and running ward's test suite in the clone.

GO

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-574` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T17:46:13Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). The pre-flight judged this issue **GO** for an unattended run. Its justification: <details><summary>pre-flight read (GO)</summary> **Context to front-load:** Before my first edit in the fresh `coilyco-flight-deck/ward` clone, I will READ: (1) the docker-launch codepath that stages `--env-file /tmp/ward-forgejo-env-*` and `-v /tmp/ward-container-assets-*:/opt/ward:ro` — grep for `ward-forgejo-env` and `ward-container-assets` to find both temp-staging sites and their cleanup/teardown logic; (2) `docs/agent.md` and `docs/agent-preflight.md` for the engineer dispatch + headless launch flow; (3) `.agents/skills/tooling-ward-agent/SKILL.md` for ref-normalization and dispatch conventions; (4) `.ward/ward.yaml` for the build/test/lint verbs I must route through, plus existing test patterns around launch-env staging. I will read each in the clone, not discover them lazily. The fix is well-diagnosed and self-contained: move launch-asset staging from `/tmp` to `~/.ward/launch/<container>/` (option 1, explicitly preferred) so snap-confined docker can read it under `$HOME`. Main risk/unknown is completeness and hygiene rather than feasibility — I must catch *both* staging sites (env-file and assets bind-mount), preserve secure file permissions on the Forgejo-token env-file (0600, since `$HOME` is longer-lived than `/tmp`), and get cleanup right so I don't leak per-container dirs; a secondary unknown is whether the container-assets mount has snap-specific path expectations I can only confirm by reading the code. None of that needs a human — it's verifiable by reading the launch code and running ward's test suite in the clone. GO </details>  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-03 17:47:41 +00:00

Author

Owner

Cross-link: this is already known and being fixed the other way. The chosen remediation is a host swap - replace kai-server's snap docker with docker-ce so the CLI shares the host filesystem view (infrastructure#450, ward touchpoints in ward#557). The ansible docker role is authored, committed, and gated at kai-server; it just has not been converged onto the host yet (snap docker is still live).

So the $HOME-staging idea here is not the primary fix - it is optional defense-in-depth that would make ward robust to any confined docker regardless of packaging. Keep or close against ward#557 as you prefer; demoting from bug to hardening.

Cross-link: this is already known and being fixed the other way. The chosen remediation is a **host swap** - replace kai-server's snap docker with docker-ce so the CLI shares the host filesystem view (infrastructure#450, ward touchpoints in ward#557). The ansible `docker` role is authored, committed, and gated at kai-server; it just has not been converged onto the host yet (snap docker is still live). So the $HOME-staging idea here is **not the primary fix** - it is optional defense-in-depth that would make ward robust to any confined docker regardless of packaging. Keep or close against ward#557 as you prefer; demoting from bug to hardening.

coilysiren referenced this issue from a commit

2026-07-03 17:51:56 +00:00

fix(agent): stage container assets in $HOME, not /tmp, for snap docker (closes #574)

coilysiren closed this issue

2026-07-03 17:51:56 +00:00

coilysiren referenced this issue from a commit

2026-07-03 17:51:57 +00:00

Merge issue-574: stage container assets in $HOME for snap docker (closes #574)

coilysiren commented

2026-07-03 17:52:28 +00:00

Owner

WARD-OUTCOME: done - assets bind-mount now stages under $HOME (launchStagingDir), completing the snap-docker /tmp-invisibility fix begun in ward#569.

The satisfying part: ward#569 had already done the hard diagnostic work and even named this second wall in its own commit body ("if a snap daemon also cannot see /tmp that is a separate wall to file"). So this was less an investigation and more collecting on a debt the previous fix flagged. The env-file is client-read, the assets dir is daemon-resolved, but under snap both live behind the same private-/tmp confinement, so the fix is symmetric: one shared launchStagingDir ($HOME else $TMPDIR) now feeds both.

What fought back, mildly: the code-comment discipline hook, which caps contiguous comment blocks at 2 lines and lines at 90 chars. My first-draft explanatory comments were too chatty and I trimmed them three times, pushing the durable "why" into docs/container-env.md where it belongs. Good forcing function, annoying in the moment.

Confidence is high on the code path (tests cover $HOME staging, the TTL sweep, and the snap-hidden-dir guard; build/vet/test green). The one thing I could NOT do from this container is exercise it against a real snap docker daemon - the actual bind-mount resolution only proves out on kai-server. Worth a smoke-test engineer dispatch there to confirm the launch now clears docker run. No follow-ups I'd file; the fix is small and mirrors the reviewed ward#569 pattern exactly.

WARD-OUTCOME: done - assets bind-mount now stages under $HOME (launchStagingDir), completing the snap-docker /tmp-invisibility fix begun in ward#569. The satisfying part: ward#569 had already done the hard diagnostic work and even named this second wall in its own commit body ("if a snap daemon also cannot see /tmp that is a separate wall to file"). So this was less an investigation and more collecting on a debt the previous fix flagged. The env-file is client-read, the assets dir is daemon-resolved, but under snap both live behind the same private-/tmp confinement, so the fix is symmetric: one shared launchStagingDir ($HOME else $TMPDIR) now feeds both. What fought back, mildly: the code-comment discipline hook, which caps contiguous comment blocks at 2 lines and lines at 90 chars. My first-draft explanatory comments were too chatty and I trimmed them three times, pushing the durable "why" into docs/container-env.md where it belongs. Good forcing function, annoying in the moment. Confidence is high on the code path (tests cover $HOME staging, the TTL sweep, and the snap-hidden-dir guard; build/vet/test green). The one thing I could NOT do from this container is exercise it against a real snap docker daemon - the actual bind-mount resolution only proves out on kai-server. Worth a smoke-test engineer dispatch there to confirm the launch now clears docker run. No follow-ups I'd file; the fix is small and mirrors the reviewed ward#569 pattern exactly.

coilyco-ops commented

2026-07-03 19:17:37 +00:00

Author

Owner

Re-elevating this - it is NOT just snap hardening, it is the fix that lets a sandboxed agent run warded engineers with no workarounds.

Confirmed live on kai-server (now on docker-ce, snap fully gone): a warded engineer run works, BUT only when launched with the Bash sandbox disabled. In the default sandbox, ward stages the token --env-file and the ward-container-assets-* bind-mount under a sandboxed /tmp that the host docker daemon cannot see, so docker run dies with open /tmp/ward-forgejo-env-...: no such file - the SAME error class snap gave, for a different reason.

So the snap private-/tmp confinement and the agent-sandbox /tmp isolation are one root cause: ward assumes its /tmp is the docker client's /tmp. The host-swap fixed only the snap half. A fresh agent (new session, no memory of the workaround) will re-hit this and likely misdiagnose it as 'docker is broken again'.

Fix as originally proposed: stage launch assets under $HOME (e.g. ~/.ward/launch/<container>/), which both the snap client and a sandboxed agent can read. Then warded runs are self-service from any harness - no disable-sandbox, no sg-docker dance.

Priority: this is the difference between 'warded runs work if you know two tricks' and 'warded runs just work'. Bumping from hardening to a real fix.

Re-elevating this - it is NOT just snap hardening, it is the fix that lets a **sandboxed agent** run warded engineers with no workarounds. Confirmed live on kai-server (now on docker-ce, snap fully gone): a warded engineer run works, BUT only when launched with the Bash sandbox disabled. In the default sandbox, ward stages the token `--env-file` and the `ward-container-assets-*` bind-mount under a sandboxed `/tmp` that the host docker daemon cannot see, so `docker run` dies with `open /tmp/ward-forgejo-env-...: no such file` - the SAME error class snap gave, for a different reason. So the snap private-`/tmp` confinement and the agent-sandbox `/tmp` isolation are one root cause: **ward assumes its `/tmp` is the docker client's `/tmp`.** The host-swap fixed only the snap half. A fresh agent (new session, no memory of the workaround) will re-hit this and likely misdiagnose it as 'docker is broken again'. Fix as originally proposed: stage launch assets under `$HOME` (e.g. `~/.ward/launch/<container>/`), which both the snap client and a sandboxed agent can read. Then warded runs are self-service from any harness - no disable-sandbox, no sg-docker dance. Priority: this is the difference between 'warded runs work if you know two tricks' and 'warded runs just work'. Bumping from hardening to a real fix.

coilysiren reopened this issue

2026-07-03 19:19:45 +00:00

coilyco-ops commented

2026-07-03 19:24:46 +00:00

Author

Owner

Correction to my previous comment - I over-claimed the sandbox angle. Measured on kai-server (docker-ce): a /tmp-staged --env-file AND a /tmp bind-mount both work fine from inside the agent's default Bash sandbox. This particular sandbox does NOT isolate /tmp from the host docker daemon, so stock ward (staging in /tmp) already works in-sandbox on docker-ce. The disable-sandbox 'trick' I cited was a snap-era artifact, not a real requirement here.

So the accurate scope of this issue is narrower than my re-elevation implied: /tmp staging only breaks under a docker client whose /tmp genuinely differs from ward's - snap-confined docker (private /tmp), and any harness that DOES isolate /tmp (not this one). It is defense-in-depth that makes ward independent of the docker client's /tmp view, not a fix that kai-server (now on docker-ce) needs to run warded agents.

Value call is therefore: worth doing if any fleet host still runs snap docker or a /tmp-isolating harness; not urgent for kai-server. I have the change implemented (stage under ~/.ward/launch) but am holding it pending that call.

Correction to my previous comment - I over-claimed the sandbox angle. Measured on kai-server (docker-ce): a /tmp-staged `--env-file` AND a /tmp bind-mount both work fine from inside the agent's default Bash sandbox. This particular sandbox does NOT isolate /tmp from the host docker daemon, so stock ward (staging in /tmp) already works in-sandbox on docker-ce. The disable-sandbox 'trick' I cited was a snap-era artifact, not a real requirement here. So the accurate scope of this issue is narrower than my re-elevation implied: /tmp staging only breaks under a docker client whose /tmp genuinely differs from ward's - **snap-confined docker** (private /tmp), and any harness that DOES isolate /tmp (not this one). It is defense-in-depth that makes ward independent of the docker client's /tmp view, not a fix that kai-server (now on docker-ce) needs to run warded agents. Value call is therefore: worth doing if any fleet host still runs snap docker or a /tmp-isolating harness; not urgent for kai-server. I have the change implemented (stage under ~/.ward/launch) but am holding it pending that call.

coilyco-ops commented

2026-07-03 19:30:12 +00:00

Author

Owner

Already fixed and landed - closing. Commit 40b7eeb ("fix(agent): stage container assets in $HOME, not /tmp, for snap docker", closes #574) implemented exactly this: a launchStagingDir() under $HOME with launchEnvFilePrefix, a stale-env-file sweep, and docs/container-env.md. The closes #574 trailer did not auto-close the issue on the direct-main push, so closing manually.

Landed by a warded engineer agent (claude-linux-...-she-her) at 2026-07-03 17:51Z - dogfood: a warded run fixed the warded-launch path.

Note: the fix landed AFTER v0.349.0, so a host still on that installed ward + snap docker needs ward upgrade to get it. kai-server is on docker-ce now, where /tmp works regardless, so it is unaffected either way (measured: /tmp env-file + bind-mount both cross the sandbox->daemon boundary on docker-ce).

Already fixed and landed - closing. Commit 40b7eeb ("fix(agent): stage container assets in $HOME, not /tmp, for snap docker", closes #574) implemented exactly this: a `launchStagingDir()` under $HOME with `launchEnvFilePrefix`, a stale-env-file sweep, and docs/container-env.md. The `closes #574` trailer did not auto-close the issue on the direct-main push, so closing manually. Landed by a warded engineer agent (claude-linux-...-she-her) at 2026-07-03 17:51Z - dogfood: a warded run fixed the warded-launch path. Note: the fix landed AFTER v0.349.0, so a host still on that installed ward + snap docker needs `ward upgrade` to get it. kai-server is on docker-ce now, where /tmp works regardless, so it is unaffected either way (measured: /tmp env-file + bind-mount both cross the sandbox->daemon boundary on docker-ce).

coilyco-ops closed this issue

2026-07-03 19:30:13 +00:00

coilysiren referenced this issue

2026-07-04 08:29:52 +00:00

ward agent engineer: launch env file removed before docker run (exit 125, no container) #569

Rows
Columns

engineer launch fails under snap docker: --env-file staged in /tmp is invisible to confined docker client #574

Symptom

Root cause

Fix options