agent-proxy director: dispatch broker unusable in container (proto v0 vs warded v1; $WARD_DISPATCH_BROKER_SOCK is an empty dir) #381

New issue

Closed

opened 2026-06-26 09:51:10 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-06-26 09:51:10 +00:00

Owner

Blocker

From the agent-proxy read-only director seat (ward v0.204.0), no sibling dispatch works - every warded engineer ... --driver goose fails. Two compounding faults:

$WARD_DISPATCH_BROKER_SOCK points at an empty directory, not a socket. It is set to /run/ward/dispatch-broker.sock, but inside the container that path is an empty dir (drwxr-xr-x root root, 0 entries) - the classic Docker symptom of bind-mounting a source path that does not exist on the host. So warded dials it and gets connect: permission denied (it is a directory).
The only real socket, /run/ward/broker.sock, speaks an older protocol. Overriding WARD_DISPATCH_BROKER_SOCK=/run/ward/broker.sock gets past the dial but then fails with broker: unsupported protocol version 0 (want 1). So the host broker that is mounted is a version behind what this container's warded expects.

Net: the host either did not mount the v1 dispatch broker at the advertised env path, or mounted a v0 broker at a different path. Both must be reconciled for in-container dispatch to function.

Impact

The whole agent-proxy phase-1 build (#2) was decomposed into 13 self-chaining goose-engineer issues (coilyco-flight-deck/agent-proxy#5 .. #17); each issue's final step dispatches the next, so the entire reliability proxy builds from one kickoff dispatch. That kickoff cannot fire until this broker is fixed. The decomposition itself is landed and durable - only the dispatch is blocked.

Repro

WARD_DISPATCH_BROKER_SOCK=/run/ward/broker.sock warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose
# -> ward: dispatch broker: broker: unsupported protocol version 0 (want 1)
ls -la /run/ward/   # broker.sock (srw-rw---- root:ubuntu) + dispatch-broker.sock/ (empty dir)

Suggested fix

Mount the v1 dispatch broker socket at $WARD_DISPATCH_BROKER_SOCK (currently an empty dir), or align the container's warded with the v0 broker that is mounted. Related: ward#318 (dispatch-only token), ward#330 (tower tailnet route).

Workaround until fixed

Kick the chain off from a working host context: warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose - it self-chains #5 -> #17.

## Blocker From the `agent-proxy` read-only director seat (ward v0.204.0), **no sibling dispatch works** - every `warded engineer ... --driver goose` fails. Two compounding faults: 1. **$WARD_DISPATCH_BROKER_SOCK points at an empty directory, not a socket.** It is set to `/run/ward/dispatch-broker.sock`, but inside the container that path is an empty dir (`drwxr-xr-x root root`, 0 entries) - the classic Docker symptom of bind-mounting a source path that does not exist on the host. So warded dials it and gets `connect: permission denied` (it is a directory). 2. **The only real socket, `/run/ward/broker.sock`, speaks an older protocol.** Overriding `WARD_DISPATCH_BROKER_SOCK=/run/ward/broker.sock` gets past the dial but then fails with `broker: unsupported protocol version 0 (want 1)`. So the host broker that *is* mounted is a version behind what this container's warded expects. Net: the host either did not mount the v1 dispatch broker at the advertised env path, or mounted a v0 broker at a different path. Both must be reconciled for in-container dispatch to function. ## Impact The whole `agent-proxy` phase-1 build (#2) was decomposed into 13 self-chaining goose-engineer issues (`coilyco-flight-deck/agent-proxy#5` .. `#17`); each issue's final step dispatches the next, so the entire reliability proxy builds from **one** kickoff dispatch. That kickoff cannot fire until this broker is fixed. The decomposition itself is landed and durable - only the dispatch is blocked. ## Repro ``` WARD_DISPATCH_BROKER_SOCK=/run/ward/broker.sock warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose # -> ward: dispatch broker: broker: unsupported protocol version 0 (want 1) ls -la /run/ward/ # broker.sock (srw-rw---- root:ubuntu) + dispatch-broker.sock/ (empty dir) ``` ## Suggested fix Mount the v1 dispatch broker socket at $WARD_DISPATCH_BROKER_SOCK (currently an empty dir), or align the container's warded with the v0 broker that is mounted. Related: ward#318 (dispatch-only token), ward#330 (tower tailnet route). ## Workaround until fixed Kick the chain off from a working host context: `warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose` - it self-chains #5 -> #17.

coilysiren commented

2026-07-01 03:22:06 +00:00

Author

Owner

The dispatch path is unblocked. ward#382 (the fix issue) is closed: the host dispatch broker no longer bind-mounts a host unix socket (which landed as an empty dir under Docker Desktop / linuxkit) - it serves over TCP on the docker gateway now (host.docker.internal:<port>, guarded by a per-launch token; WARD_DISPATCH_BROKER_ADDR). The transport move landed in ward#391; this pass added the two fail-loud papercuts #382 asked for (a clear "did not answer at <addr>" on an unreachable broker, and a wrong broker hint when a dial reaches the credential broker's proto-version refusal) plus the docs/broker.md note distinguishing the two brokers.

From a fresh read-only director surface the agent-proxy chain can now be kicked off, e.g. warded coilyco-flight-deck/agent-proxy#5 --driver goose.

The dispatch path is **unblocked**. ward#382 (the fix issue) is closed: the host dispatch broker no longer bind-mounts a host unix socket (which landed as an empty dir under Docker Desktop / linuxkit) - it serves over **TCP on the docker gateway** now (`host.docker.internal:<port>`, guarded by a per-launch token; `WARD_DISPATCH_BROKER_ADDR`). The transport move landed in ward#391; this pass added the two fail-loud papercuts #382 asked for (a clear "did not answer at `<addr>`" on an unreachable broker, and a `wrong broker` hint when a dial reaches the credential broker's proto-version refusal) plus the `docs/broker.md` note distinguishing the two brokers. From a fresh read-only director surface the `agent-proxy` chain can now be kicked off, e.g. `warded coilyco-flight-deck/agent-proxy#5 --driver goose`.

coilysiren added the

interactive

labels

2026-07-01 04:29:34 +00:00

coilyco-ops commented

2026-07-01 23:59:34 +00:00

Member

Fleet-readiness note: surfaced for tonight's burndown session - only matters if runs use the dispatch broker; skip if engineers run plain warded. Recorded by Claude Code (Fable) during the 2026-07-01 ward launch triage session with Kai.

coilyco-ops referenced this issue

2026-07-02 00:01:42 +00:00

meta: jul 1st evening sprint - launch burndown session 1 #492

coilysiren commented

2026-07-02 02:14:26 +00:00

Author

Owner

Live verification - broker works end to end, closing

Verified from tonight's burndown session (ward#492), from exactly the seat that filed this: a read-only director surface container. The agent dispatched warded coilyco-flight-deck/ward#301 --no-preflight --repo coilyco-flight-deck/agentic-os and the full chain worked:

dial - warded reached the TCP broker at $WARD_DISPATCH_BROKER_ADDR (no socket path involved, so fault 1, the empty-dir bind mount, is gone by design)
forward - the broker relayed ward agent engineer coilyco-flight-deck/ward#301 --driver claude --repo coilyco-flight-deck/agentic-os --no-preflight to host ward (host run log ~/.ward/agent-logs/dispatch/20260702T021132Z-session-claude-fcb1cdfc-coilyco-flight-deck-ward-301.log)
launch - container engineer-claude-ward-301 came up on kais-macbook-pro-2.local and posted its reservation on ward#301 at 2026-07-02T02:11:36Z, attributed to coilyco-ops
no proto mismatch - fault 2 (v0 vs v1) did not reproduce on the TCP path

The ward#382 + ward#391 transport move is confirmed working in live use. The agent-proxy kickoff chain (#5-#17) this was blocking is now unblocked, but that is outside tonight's scope and stays with its own tracker.

Recorded by Claude Code (Fable) during the ward#492 burndown session.

## Live verification - broker works end to end, closing Verified from tonight's burndown session (ward#492), from exactly the seat that filed this: a read-only director surface container. The agent dispatched `warded coilyco-flight-deck/ward#301 --no-preflight --repo coilyco-flight-deck/agentic-os` and the full chain worked: * dial - warded reached the TCP broker at $WARD_DISPATCH_BROKER_ADDR (no socket path involved, so fault 1, the empty-dir bind mount, is gone by design) * forward - the broker relayed `ward agent engineer coilyco-flight-deck/ward#301 --driver claude --repo coilyco-flight-deck/agentic-os --no-preflight` to host ward (host run log `~/.ward/agent-logs/dispatch/20260702T021132Z-session-claude-fcb1cdfc-coilyco-flight-deck-ward-301.log`) * launch - container `engineer-claude-ward-301` came up on kais-macbook-pro-2.local and posted its reservation on ward#301 at 2026-07-02T02:11:36Z, attributed to coilyco-ops * no proto mismatch - fault 2 (v0 vs v1) did not reproduce on the TCP path The ward#382 + ward#391 transport move is confirmed working in live use. The agent-proxy kickoff chain (#5-#17) this was blocking is now unblocked, but that is outside tonight's scope and stays with its own tracker. Recorded by Claude Code (Fable) during the ward#492 burndown session.

coilysiren closed this issue

2026-07-02 02:14:26 +00:00

coilysiren referenced this issue

2026-07-02 07:10:55 +00:00

meta: jul 1st evening sprint - launch burndown session 1 #492