#597 - advisor dispatch dies with raw exit 125 when the ward-tailnet docker network is absent - preflight it - coilyco-flight-deck/ward

coilyco-ops commented

2026-07-04 07:34:29 +00:00

Owner

The advisor role's default guardfile set (tailnet + AWS observe) attaches the container to the docker network 'ward-tailnet'. On a host where that network does not exist (e.g. kai-desktop-tower, Docker Desktop on Windows, 2026-07-04), dispatch fails with the raw docker error 'failed to set up container networking: network ward-tailnet not found' and exit 125 - after the image pull and drain sweep already ran.

Cost: a prior session (a warded container session working coilyco-bridge/deploy#31) hit persistent 125s on the same dispatch and burned several retries misdiagnosing it as a generic container-start failure, since nothing surfaced the missing-network cause or the fallback.

Ask: preflight the network before pull/run when the role's guardfile set needs it, and fail with a one-line actionable error naming the missing network and the '--no-tailnet' fallback (or auto-degrade to isolated with a warning). Related: ward#174 (dispatch self-service), ward#586 (cred expiry warning pattern - same 'tell the operator what will bite them' shape).

Reopened: regression confirmed live (2026-07-05)

Advisor dispatch is broken again from a director surface host. Two consecutive attempts from an eco-ops read-only director session:

warded advisor coilyco-flight-deck/cli-guard#190 "<design question>"
-> ward agent: docker network "ward-tailnet" not found - the advisor role joins
   the tailnet through it; create the network on this host, or re-run with
   --no-tailnet to dispatch isolated (ward#597)

warded advisor coilyco-flight-deck/cli-guard#190 --no-tailnet "<design question>"
-> ward agent: docker network "ward-tailnet" not found ... (ward#597)

The second attempt passed --no-tailnet and still failed with the identical error. So the isolation opt-out this issue's fix was supposed to deliver is not honored: the tailnet-network preflight fires before --no-tailnet is applied (or ignores it entirely). Net effect: the advisor role cannot launch at all from a host without the ward-tailnet network, even isolated, which blocks all design/research dispatch from the director surface.

Engineer dispatch is unaffected (engineer does not join the tailnet) - ward#547 and ward#612 engineers dispatched fine from the same session.

Ask: honor --no-tailnet (skip the network join and its preflight entirely), and when tailnet IS requested but the network is absent, either create it or fail with an actionable message that does not also swallow the isolation opt-out.

Steer from Kai (2026-07-06): tailnet is ALWAYS default

Reframe the fix. Kai wants every agent on the tailnet by default — not "honor --no-tailnet as an escape." So the bug is not that the isolation opt-out is ignored; it is that the ward-tailnet docker network is absent on the host and the launch hard-fails instead of ensuring it exists.

Correct behavior:

Every dispatched role (engineer, director, advisor) joins ward-tailnet by default.
The launcher creates / ensures the ward-tailnet network as part of bring-up (idempotent), rather than erroring docker network "ward-tailnet" not found. A missing network is a provisioning step, not a launch failure.
--no-tailnet remains a rare explicit opt-out, but it is not the resolution here — the default path must just work.

Failing-launch evidence I have

Two dispatch attempts from this eco-ops director surface, both failed identically at the broker before the container came up:

warded advisor coilyco-flight-deck/cli-guard#190 "<design q>"
-> ward agent: docker network "ward-tailnet" not found - the advisor role joins
   the tailnet through it; create the network on this host, or re-run with
   --no-tailnet to dispatch isolated (ward#597)

warded advisor coilyco-flight-deck/cli-guard#190 --no-tailnet "<design q>"
-> ward agent: docker network "ward-tailnet" not found ... (ward#597)   # --no-tailnet ignored

Note: engineer dispatches from the same session succeeded (engineer does not join the tailnet), so the failure is specific to the tailnet-joining path. That is the fuller inline broker error I have — the deeper per-launch log is host-side at C:\Users\firem\.ward\agent-logs\dispatch\...advisor...log on KAI-DESKTOP-TOWER, which I cannot read from this container. If you want the full stack, that log is the place.

The advisor role's default guardfile set (tailnet + AWS observe) attaches the container to the docker network 'ward-tailnet'. On a host where that network does not exist (e.g. kai-desktop-tower, Docker Desktop on Windows, 2026-07-04), dispatch fails with the raw docker error 'failed to set up container networking: network ward-tailnet not found' and exit 125 - after the image pull and drain sweep already ran. Cost: a prior session (a warded container session working coilyco-bridge/deploy#31) hit persistent 125s on the same dispatch and burned several retries misdiagnosing it as a generic container-start failure, since nothing surfaced the missing-network cause or the fallback. Ask: preflight the network before pull/run when the role's guardfile set needs it, and fail with a one-line actionable error naming the missing network and the '--no-tailnet' fallback (or auto-degrade to isolated with a warning). Related: ward#174 (dispatch self-service), ward#586 (cred expiry warning pattern - same 'tell the operator what will bite them' shape). --- ## Reopened: regression confirmed live (2026-07-05) Advisor dispatch is broken again from a director surface host. Two consecutive attempts from an eco-ops read-only director session: ``` warded advisor coilyco-flight-deck/cli-guard#190 "<design question>" -> ward agent: docker network "ward-tailnet" not found - the advisor role joins the tailnet through it; create the network on this host, or re-run with --no-tailnet to dispatch isolated (ward#597) warded advisor coilyco-flight-deck/cli-guard#190 --no-tailnet "<design question>" -> ward agent: docker network "ward-tailnet" not found ... (ward#597) ``` The second attempt passed **`--no-tailnet`** and still failed with the identical error. So the isolation opt-out this issue's fix was supposed to deliver is not honored: the tailnet-network preflight fires **before** `--no-tailnet` is applied (or ignores it entirely). Net effect: **the advisor role cannot launch at all from a host without the `ward-tailnet` network**, even isolated, which blocks all design/research dispatch from the director surface. Engineer dispatch is unaffected (engineer does not join the tailnet) - ward#547 and ward#612 engineers dispatched fine from the same session. Ask: honor `--no-tailnet` (skip the network join and its preflight entirely), and when tailnet IS requested but the network is absent, either create it or fail with an actionable message that does not also swallow the isolation opt-out. --- ## Steer from Kai (2026-07-06): tailnet is ALWAYS default Reframe the fix. Kai wants **every agent on the tailnet by default** — not "honor `--no-tailnet` as an escape." So the bug is not that the isolation opt-out is ignored; it is that the **`ward-tailnet` docker network is absent on the host and the launch hard-fails instead of ensuring it exists**. Correct behavior: - Every dispatched role (engineer, director, advisor) joins `ward-tailnet` by default. - The launcher **creates / ensures the `ward-tailnet` network** as part of bring-up (idempotent), rather than erroring `docker network "ward-tailnet" not found`. A missing network is a provisioning step, not a launch failure. - `--no-tailnet` remains a rare explicit opt-out, but it is not the resolution here — the default path must just work. ## Failing-launch evidence I have Two dispatch attempts from this eco-ops director surface, both failed identically at the broker before the container came up: ``` warded advisor coilyco-flight-deck/cli-guard#190 "<design q>" -> ward agent: docker network "ward-tailnet" not found - the advisor role joins the tailnet through it; create the network on this host, or re-run with --no-tailnet to dispatch isolated (ward#597) warded advisor coilyco-flight-deck/cli-guard#190 --no-tailnet "<design q>" -> ward agent: docker network "ward-tailnet" not found ... (ward#597) # --no-tailnet ignored ``` Note: engineer dispatches from the same session succeeded (engineer does not join the tailnet), so the failure is specific to the tailnet-joining path. That is the fuller inline broker error I have — the deeper per-launch log is host-side at `C:\Users\firem\.ward\agent-logs\dispatch\...advisor...log` on KAI-DESKTOP-TOWER, which I cannot read from this container. If you want the full stack, that log is the place.

coilyco-ops added the

labels

2026-07-04 08:21:30 +00:00

coilyco-ops commented

2026-07-04 08:33:53 +00:00

Author

Owner

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-597 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T08:33:48Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-597` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T08:33:48Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-04 08:49:01 +00:00

fix(agent): preflight the ward-tailnet network before pull so a missing net fails fast

coilysiren closed this issue

2026-07-04 08:49:01 +00:00

coilysiren commented

2026-07-04 08:49:42 +00:00

Owner

WARD-OUTCOME: done - preflightTailnet now gates every --ts-sidecar dispatch on the ward-tailnet network before the pull/sweep, with a one-line missing-network error naming --no-tailnet.

Honest retro: the fix itself was small, but the diagnosis was the interesting part. A preflightTailnetProxy check already existed - it just (a) ran inside createAgentContainer, after the pull and drain sweep had already burned, and (b) conflated "network absent" with "proxy box unattached" behind one mac-proxy message. The real sting was that the advisor research path (agent_reply.go) calls docker create directly and never routes through createAgentContainer, so it had no guard at all - that's the path that produced the raw 125. Splitting the check, gating it on plan.TSSidecar, and hoisting it ahead of the pull in all four dispatch paths covered it.

What fought back: the repo's code-comments pre-commit hook (2-line contiguous blocks, 90-char cap) rejected my first two commit attempts, so I trimmed the prose down and pushed the durable detail into docs/agent-ts-sidecar.md instead. Also had to watch gocyclo - my extra branch tipped runAgentAsk over the threshold, so I factored the shared preflight+sweep+pull into a prelaunchDispatch helper, which netted lint back to the exact baseline (50 findings, no new ones).

Confidence: high on the logic and the unit test (missing-net vs unattached-box vs no-op non-tailnet plan all covered). The one thing untested end-to-end is the live behavior on an actual network-less Docker Desktop host, since this container can't reproduce that. Merged cleanly with the concurrent snap-docker exit-125 fix that landed on main mid-run - same "name the cause, don't emit a bare 125" spirit, no overlap. No follow-ups I'd file.

WARD-OUTCOME: done - preflightTailnet now gates every --ts-sidecar dispatch on the ward-tailnet network before the pull/sweep, with a one-line missing-network error naming --no-tailnet. Honest retro: the fix itself was small, but the diagnosis was the interesting part. A `preflightTailnetProxy` check already existed - it just (a) ran *inside* `createAgentContainer`, after the pull and drain sweep had already burned, and (b) conflated "network absent" with "proxy box unattached" behind one mac-proxy message. The real sting was that the advisor *research* path (`agent_reply.go`) calls `docker create` directly and never routes through `createAgentContainer`, so it had no guard at all - that's the path that produced the raw 125. Splitting the check, gating it on `plan.TSSidecar`, and hoisting it ahead of the pull in all four dispatch paths covered it. What fought back: the repo's `code-comments` pre-commit hook (2-line contiguous blocks, 90-char cap) rejected my first two commit attempts, so I trimmed the prose down and pushed the durable detail into docs/agent-ts-sidecar.md instead. Also had to watch gocyclo - my extra branch tipped `runAgentAsk` over the threshold, so I factored the shared preflight+sweep+pull into a `prelaunchDispatch` helper, which netted lint back to the exact baseline (50 findings, no new ones). Confidence: high on the logic and the unit test (missing-net vs unattached-box vs no-op non-tailnet plan all covered). The one thing untested end-to-end is the live behavior on an actual network-less Docker Desktop host, since this container can't reproduce that. Merged cleanly with the concurrent snap-docker exit-125 fix that landed on main mid-run - same "name the cause, don't emit a bare 125" spirit, no overlap. No follow-ups I'd file.

coilysiren reopened this issue

2026-07-05 22:08:17 +00:00

coilyco-ops commented

2026-07-06 01:43:17 +00:00

Author

Owner

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-597 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-06T01:43:14Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#597 · branch issue-597 · driver claude · workflow direct-main
Run: engineer-claude-ward-597 · ward v0.406.0 · dispatched 2026-07-06T01:43:14Z
Comment thread: 1 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilysiren (2026-07-04T08:49:42Z)
- stripped: @coilyco-ops (2026-07-04T08:33:53Z)

Issue body as seeded:

The advisor role's default guardfile set (tailnet + AWS observe) attaches the container to the docker network 'ward-tailnet'. On a host where that network does not exist (e.g. kai-desktop-tower, Docker Desktop on Windows, 2026-07-04), dispatch fails with the raw docker error 'failed to set up container networking: network ward-tailnet not found' and exit 125 - after the image pull and drain sweep already ran.

Cost: a prior session (a warded container session working coilyco-bridge/deploy#31) hit persistent 125s on the same dispatch and burned several retries misdiagnosing it as a generic container-start failure, since nothing surfaced the missing-network cause or the fallback.

Ask: preflight the network before pull/run when the role's guardfile set needs it, and fail with a one-line actionable error naming the missing network and the '--no-tailnet' fallback (or auto-degrade to isolated with a warning). Related: ward#174 (dispatch self-service), ward#586 (cred expiry warning pattern - same 'tell the operator what will bite them' shape).

---

## Reopened: regression confirmed live (2026-07-05)

Advisor dispatch is broken again from a director surface host. Two consecutive attempts from an eco-ops read-only director session:

` ` `
warded advisor coilyco-flight-deck/cli-guard#190 "<design question>"
-> ward agent: docker network "ward-tailnet" not found - the advisor role joins
   the tailnet through it; create the network on this host, or re-run with
   --no-tailnet to dispatch isolated (ward#597)

warded advisor coilyco-flight-deck/cli-guard#190 --no-tailnet "<design question>"
-> ward agent: docker network "ward-tailnet" not found ... (ward#597)
` ` `

The second attempt passed **`--no-tailnet`** and still failed with the identical error. So the isolation opt-out this issue's fix was supposed to deliver is not honored: the tailnet-network preflight fires **before** `--no-tailnet` is applied (or ignores it entirely). Net effect: **the advisor role cannot launch at all

… (truncated to 2000 chars; full body is on this issue)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.406.0).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-597` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-06T01:43:14Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#597` · branch `issue-597` · driver `claude` · workflow `direct-main` - **Run:** `engineer-claude-ward-597` · ward `v0.406.0` · dispatched `2026-07-06T01:43:14Z` - **Comment thread:** 1 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilysiren (2026-07-04T08:49:42Z) - stripped: @coilyco-ops (2026-07-04T08:33:53Z) **Issue body as seeded:** ``` The advisor role's default guardfile set (tailnet + AWS observe) attaches the container to the docker network 'ward-tailnet'. On a host where that network does not exist (e.g. kai-desktop-tower, Docker Desktop on Windows, 2026-07-04), dispatch fails with the raw docker error 'failed to set up container networking: network ward-tailnet not found' and exit 125 - after the image pull and drain sweep already ran. Cost: a prior session (a warded container session working coilyco-bridge/deploy#31) hit persistent 125s on the same dispatch and burned several retries misdiagnosing it as a generic container-start failure, since nothing surfaced the missing-network cause or the fallback. Ask: preflight the network before pull/run when the role's guardfile set needs it, and fail with a one-line actionable error naming the missing network and the '--no-tailnet' fallback (or auto-degrade to isolated with a warning). Related: ward#174 (dispatch self-service), ward#586 (cred expiry warning pattern - same 'tell the operator what will bite them' shape). --- ## Reopened: regression confirmed live (2026-07-05) Advisor dispatch is broken again from a director surface host. Two consecutive attempts from an eco-ops read-only director session: ` ` ` warded advisor coilyco-flight-deck/cli-guard#190 "<design question>" -> ward agent: docker network "ward-tailnet" not found - the advisor role joins the tailnet through it; create the network on this host, or re-run with --no-tailnet to dispatch isolated (ward#597) warded advisor coilyco-flight-deck/cli-guard#190 --no-tailnet "<design question>" -> ward agent: docker network "ward-tailnet" not found ... (ward#597) ` ` ` The second attempt passed **`--no-tailnet`** and still failed with the identical error. So the isolation opt-out this issue's fix was supposed to deliver is not honored: the tailnet-network preflight fires **before** `--no-tailnet` is applied (or ignores it entirely). Net effect: **the advisor role cannot launch at all ``` … (truncated to 2000 chars; full body is on this issue) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.406.0). </details>  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-06 01:57:06 +00:00

fix(agent): create the ward-tailnet network on dispatch instead of 125ing when it is absent (closes #597)

coilysiren closed this issue

2026-07-06 01:57:06 +00:00

coilysiren commented

2026-07-06 01:57:52 +00:00

Owner

WARD-OUTCOME: done - preflightTailnet now creates the ward-tailnet network on dispatch (idempotent) instead of 125ing; unattached mac-proxy box downgraded from hard-fail to warning. Landed on main as 4cf59d5.

The Kai steer reframed this cleanly: the earlier fix had made "network absent" an actionable error, and the reopen showed that was exactly backwards for a tailnet-is-default world. So the change was mostly a sign flip in one function - turn the missing-network branch from return error into ensureTailnetNetwork (create + idempotent re-inspect on race), and turn the box-unattached branch from a second hard error into a warning that matches the existing #586/#332 "tell the operator what will bite them" pattern already living two functions up in the same file.

What fought back a little: the --no-tailnet ignored symptom in the reopen was a red herring under the new framing - I deliberately did NOT chase it, since the steer says the default path must just work and --no-tailnet stays a rare opt-out. The only real subtlety was idempotency without stderr: docker's "already exists" goes to stderr and I capture stdout, so I settled on create-then-reinspect to tell a benign race apart from a real daemon failure.

Confidence is high on the unit level (create-succeeds / create-fails / box-unattached all covered, full suite + pre-commit green). The one thing unit tests can't reach is the live end-to-end on kai-desktop-tower itself - I can't run Docker Desktop from this container, so the actual "advisor launches from the director surface" verification is still Kai's to confirm. The ts-sidecar doc already carried an open "live end-to-end lands as a follow-up" note for the same reason, so this rides along with that. No follow-up issue felt warranted beyond that existing live-fire gap.

WARD-OUTCOME: done - preflightTailnet now creates the ward-tailnet network on dispatch (idempotent) instead of 125ing; unattached mac-proxy box downgraded from hard-fail to warning. Landed on main as 4cf59d5. The Kai steer reframed this cleanly: the earlier fix had made "network absent" an actionable error, and the reopen showed that was exactly backwards for a tailnet-is-default world. So the change was mostly a sign flip in one function - turn the missing-network branch from `return error` into `ensureTailnetNetwork` (create + idempotent re-inspect on race), and turn the box-unattached branch from a second hard error into a warning that matches the existing #586/#332 "tell the operator what will bite them" pattern already living two functions up in the same file. What fought back a little: the `--no-tailnet ignored` symptom in the reopen was a red herring under the new framing - I deliberately did NOT chase it, since the steer says the default path must just work and --no-tailnet stays a rare opt-out. The only real subtlety was idempotency without stderr: docker's "already exists" goes to stderr and I capture stdout, so I settled on create-then-reinspect to tell a benign race apart from a real daemon failure. Confidence is high on the unit level (create-succeeds / create-fails / box-unattached all covered, full suite + pre-commit green). The one thing unit tests can't reach is the live end-to-end on kai-desktop-tower itself - I can't run Docker Desktop from this container, so the actual "advisor launches from the director surface" verification is still Kai's to confirm. The ts-sidecar doc already carried an open "live end-to-end lands as a follow-up" note for the same reason, so this rides along with that. No follow-up issue felt warranted beyond that existing live-fire gap.

coilysiren referenced this issue

2026-07-06 18:58:25 +00:00

advisor ref-mode returns docker exit 125 through the dispatch broker (TTY/attach in non-interactive host-forward context) #605

coilyco-ops referenced this issue

2026-07-08 22:19:00 +00:00

Make agent workflow defaults configurable and default engineers to PRs #707

Rows
Columns

advisor dispatch dies with raw exit 125 when the ward-tailnet docker network is absent - preflight it #597

Reopened: regression confirmed live (2026-07-05)

Steer from Kai (2026-07-06): tailnet is ALWAYS default

Failing-launch evidence I have