ward agent engineer: launch env file removed before docker run (exit 125, no container)

coilyco-ops commented

2026-07-03 17:15:03 +00:00

Member

Symptom

A live ward agent engineer dispatch dies at the docker run step:

ward agent engineer --driver claude: wrote launch env file for coilyco-flight-deck/infrastructure#452
docker: open /tmp/ward-forgejo-env-834139975: no such file or directory
exit status 125

Repro: ward agent engineer coilyco-flight-deck/infrastructure#452 --workflow patch-only (ward v0.349.0, native Linux / KAI-SERVER). --print renders the full plan fine, so resolution and prompt-seeding are healthy. The break is only on a real launch.

Root cause

ward logs wrote launch env file, then hands docker run --env-file /tmp/ward-forgejo-env-<rand>, but by exec time nothing exists at /tmp/ward-forgejo-env-* (verified - the glob is empty right after the failure). The env file is written and then removed before docker run consumes it. Most likely a defer os.Remove(envFile) firing before the exec (or in the wrong scope), or a write-path vs pass-path mismatch. Note the placeholder in --print reads <ward-forgejo-token-envfile> while the real path is /tmp/ward-forgejo-env-<rand> - worth confirming both sides reference the same variable.

Impact

No container is ever created, so engineer dispatch is fully broken on this host. No cleanup of the reservation happens on this failure path - see the companion issue.

Fix direction

Keep the env file alive until docker run has started (the container reads it at start, then it can be removed). Move any cleanup to after the exec, or hold the file for the container lifetime and reap it with the container.

## Symptom A live `ward agent engineer` dispatch dies at the `docker run` step: ``` ward agent engineer --driver claude: wrote launch env file for coilyco-flight-deck/infrastructure#452 docker: open /tmp/ward-forgejo-env-834139975: no such file or directory exit status 125 ``` Repro: `ward agent engineer coilyco-flight-deck/infrastructure#452 --workflow patch-only` (ward v0.349.0, native Linux / KAI-SERVER). `--print` renders the full plan fine, so resolution and prompt-seeding are healthy. The break is only on a real launch. ## Root cause ward logs `wrote launch env file`, then hands `docker run` `--env-file /tmp/ward-forgejo-env-<rand>`, but by exec time nothing exists at `/tmp/ward-forgejo-env-*` (verified - the glob is empty right after the failure). The env file is written and then removed before `docker run` consumes it. Most likely a `defer os.Remove(envFile)` firing before the exec (or in the wrong scope), or a write-path vs pass-path mismatch. Note the placeholder in `--print` reads `<ward-forgejo-token-envfile>` while the real path is `/tmp/ward-forgejo-env-<rand>` - worth confirming both sides reference the same variable. ## Impact No container is ever created, so engineer dispatch is fully broken on this host. No cleanup of the reservation happens on this failure path - see the companion issue. ## Fix direction Keep the env file alive until `docker run` has started (the container reads it at start, then it can be removed). Move any cleanup to after the exec, or hold the file for the container lifetime and reap it with the container.

coilyco-ops referenced this issue

2026-07-03 17:15:17 +00:00

ward agent engineer: reservation (local hold + forge road-block comment) not rolled back when docker run fails #570

coilyco-ops commented

2026-07-03 17:17:49 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-569 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T17:17:44Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-569` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T17:17:44Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilyco-ops added the

headless

P1

labels

2026-07-03 17:30:38 +00:00

coilysiren referenced this issue from a commit

2026-07-03 17:30:58 +00:00

fix(agent): write docker --env-file in $HOME, not /tmp, for snap docker (closes #569)

coilysiren closed this issue

2026-07-03 17:30:58 +00:00

coilysiren commented

2026-07-03 17:31:33 +00:00

Owner

WARD-OUTCOME: done - --env-file now lands top-level in $HOME so a snap docker CLI can read it; engineer dispatch no longer dies at exit 125.

The reported diagnosis (a defer os.Remove firing early) turned out to be a red herring, which is the part that fought back. defer cleanupEnv() in launchAgentContainer is correctly scoped - it only fires on function return, after docker run has already consumed the file - so "glob empty right after the failure" was just that deferred cleanup running on the error path, not a premature delete. Once I stopped chasing lifetime and looked at the exact host (KAI-SERVER, snap docker, same box as ward#540), the real shape fell out: the docker CLI reads --env-file client-side, and a snap-confined CLI runs under a private /tmp, so a host /tmp path reads back as "no such file". A visibility bug wearing a lifetime bug`s clothes.

Fix is small and I am fairly confident in it: write the file in $HOME (snaps default home` interface exposes non-hidden $HOME files, never the hidden ~/.ward), fall back to $TMPDIR, plus a past-TTL orphan sweep since $HOME is never OS-reaped. Unit tests + full suite + the pre-commit catalog are green.

One honest caveat I could not verify from this container (no snap docker here): the container assets dir is also bind-mounted from /tmp, but that source is resolved daemon-side, and the snap docker daemon (docker-support interface) is far less confined than the CLI - so it should be fine, and the fact that the failure was purely --env-file supports that. If a snap daemon also cannot see /tmp for bind mounts, that is a distinct next wall worth its own issue.

WARD-OUTCOME: done - `--env-file` now lands top-level in $HOME so a snap docker CLI can read it; engineer dispatch no longer dies at exit 125. The reported diagnosis (a `defer os.Remove` firing early) turned out to be a red herring, which is the part that fought back. `defer cleanupEnv()` in launchAgentContainer is correctly scoped - it only fires on function return, after `docker run` has already consumed the file - so "glob empty right after the failure" was just that deferred cleanup running on the error path, not a premature delete. Once I stopped chasing lifetime and looked at the exact host (KAI-SERVER, snap docker, same box as ward#540), the real shape fell out: the docker CLI reads `--env-file` client-side, and a snap-confined CLI runs under a private /tmp, so a host /tmp path reads back as "no such file". A visibility bug wearing a lifetime bug`s clothes. Fix is small and I am fairly confident in it: write the file in $HOME (snap`s default `home` interface exposes non-hidden $HOME files, never the hidden ~/.ward), fall back to $TMPDIR, plus a past-TTL orphan sweep since $HOME is never OS-reaped. Unit tests + full suite + the pre-commit catalog are green. One honest caveat I could not verify from this container (no snap docker here): the container **assets** dir is also bind-mounted from /tmp, but that source is resolved daemon-side, and the snap docker daemon (docker-support interface) is far less confined than the CLI - so it should be fine, and the fact that the failure was purely `--env-file` supports that. If a snap daemon also cannot see /tmp for bind mounts, that is a distinct next wall worth its own issue.

coilysiren referenced this issue from a commit

2026-07-03 17:50:28 +00:00

fix(agent): roll reservation back when a launch dies before the container is up

coilyco-ops commented

2026-07-04 08:26:20 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-569 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T08:26:14Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-569` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T08:26:14Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-04 08:26:25 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-569 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T08:26:14Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-569` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T08:26:14Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-04 08:26:31 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-569 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T08:26:14Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-569` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T08:26:14Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren commented

2026-07-04 08:29:52 +00:00

Owner

WARD-OUTCOME: done - #569 was already fixed and closed by 6e02891 (--env-file in $HOME for snap docker); this re-dispatch found nothing left to land.

Honest retrospective: this one was already done before I arrived. The env-file fix landed on main as commit 6e02891 ("write docker --env-file in $HOME, not /tmp, for snap docker (closes #569)") on 2026-07-03, the issue closed at 17:30Z that same day, and the companion assets-dir wall (#574) followed in 40b7eeb. My fresh clone's branch has zero diff from main — the code already stages the --env-file under launchStagingDir() ($HOME, never the snap-invisible hidden ~/.ward, falling back to $TMPDIR), with an orphan sweep since $HOME is never OS-reaped, plus unit coverage in container_test.go.

The interesting part is the diagnosis in the issue body was a red herring, and the prior run caught it: the failure wasn't a defer os.Remove firing early (that defer is correctly scoped to function return, after docker run consumes the file). It was visibility — a snap-confined docker CLI reads --env-file client-side under a private /tmp, so a host /tmp path reads back as "no such file". Moving to $HOME (which snap's home interface exposes) cleared it.

So I did not manufacture a commit for an already-solved, already-closed issue — that would just be noise on a green tree. I confirmed go build ./... compiles and the env-file/staging tests pass. Confidence is high that #569 is genuinely resolved. The one open thread is the same caveat the prior run flagged: if a snap daemon also can't see /tmp for some other bind mount down the line, that's a distinct wall — but #574 already covered the known assets-dir case, so nothing to file here.

WARD-OUTCOME: done - #569 was already fixed and closed by 6e02891 (`--env-file` in $HOME for snap docker); this re-dispatch found nothing left to land. Honest retrospective: this one was already done before I arrived. The env-file fix landed on `main` as commit 6e02891 ("write docker --env-file in $HOME, not /tmp, for snap docker (closes #569)") on 2026-07-03, the issue closed at 17:30Z that same day, and the companion assets-dir wall (#574) followed in 40b7eeb. My fresh clone's branch has **zero diff from main** — the code already stages the `--env-file` under `launchStagingDir()` ($HOME, never the snap-invisible hidden `~/.ward`, falling back to $TMPDIR), with an orphan sweep since $HOME is never OS-reaped, plus unit coverage in container_test.go. The interesting part is the diagnosis in the issue body was a red herring, and the prior run caught it: the failure wasn't a `defer os.Remove` firing early (that defer is correctly scoped to function return, after `docker run` consumes the file). It was **visibility** — a snap-confined docker CLI reads `--env-file` client-side under a private /tmp, so a host /tmp path reads back as "no such file". Moving to $HOME (which snap's `home` interface exposes) cleared it. So I did not manufacture a commit for an already-solved, already-closed issue — that would just be noise on a green tree. I confirmed `go build ./...` compiles and the env-file/staging tests pass. Confidence is high that #569 is genuinely resolved. The one open thread is the same caveat the prior run flagged: if a snap *daemon* also can't see /tmp for some other bind mount down the line, that's a distinct wall — but #574 already covered the known assets-dir case, so nothing to file here.

Rows
Columns

ward agent engineer: launch env file removed before docker run (exit 125, no container) #569

Symptom

Root cause

Impact

Fix direction