coilyco-flight-deck/ward

Fork 0

Fix: host dispatch-broker unix socket does not bind-mount into containers under Docker Desktop / linuxkit (lands as empty dir) #382

New issue

Closed

opened 2026-06-26 09:57:26 +00:00 by coilysiren · 5 comments

coilysiren commented

2026-06-26 09:57:26 +00:00

Owner

Summary (for the engineer: this is a real, reproducible infra bug - read it all before touching code)

In a read-only director surface container, sibling dispatch (warded engineer ...) is 100% broken because the host dispatch broker's unix socket never arrives in the container. The mount target /run/ward/dispatch-broker.sock shows up as an empty directory, not a socket. This blocks the entire agent-proxy build chain (coilyco-flight-deck/agent-proxy#5..#17); tracked as the blocker in ward#381 - this issue is the fix.

Exact symptoms (reproduced)

$ echo $WARD_DISPATCH_BROKER_SOCK
/run/ward/dispatch-broker.sock
$ ls -la /run/ward/
srw-rw---- 1 root ubuntu  0  /run/ward/broker.sock            # the CREDENTIAL broker (works)
drwxr-xr-x 2 root root   40  /run/ward/dispatch-broker.sock/  # the DISPATCH broker: an EMPTY DIR, not a socket

$ warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose
ward: dispatch broker unavailable: dial /run/ward/dispatch-broker.sock: connect: permission denied   # it's a directory

# pointing at the only real socket reaches the WRONG broker:
$ WARD_DISPATCH_BROKER_SOCK=/run/ward/broker.sock warded engineer ... 
ward: dispatch broker: broker: unsupported protocol version 0 (want 1)   # that's the credential broker, not the dispatch broker

Host: container kernel 6.12.76-linuxkit => this is Docker Desktop's LinuxKit VM. ward v0.204.0.

Root cause (high confidence)

There are two brokers and they are provisioned differently:

Credential broker - cmd/ward/broker.go (defaultBrokerSocket = "/run/ward/broker.sock", line ~25). Runs inside the container as an entrypoint daemon ("Entrypoint-internal root credential broker daemon"). The socket is created in-container, so it is always present. This one works.
Dispatch broker - cmd/ward/agent_dispatch_broker.go. startHostDispatchBroker (line ~36) does os.MkdirTemp("", "ward-dispatch-broker-*") on the host, net.Listen("unix", <tmpdir>/broker.sock), then dispatchBrokerMount(hostSock) (line ~293) bind-mounts that host socket to containerDispatchBrokerSock = "/run/ward/dispatch-broker.sock" (cmd/ward/container_compute.go:58). attachHostDispatchBroker (cmd/ward/agent_director_surface.go:149) wires it; container_compute.go:~429 exports the WARD_DISPATCH_BROKER_SOCK env.

The defect: bind-mounting a host unix socket into a container does not work under Docker Desktop / linuxkit. The host filesystem is shared into the VM via virtiofs/gRPC-FUSE, and a live unix socket node does not traverse that boundary. Docker, finding no bindable socket inode at mount time, creates an empty directory at the target - precisely the observed /run/ward/dispatch-broker.sock/ empty dir. The credential broker avoids this only because it is created inside the container, never bind-mounted.

So the host-unix-socket-bind-mount transport is fundamentally unavailable on this host class. It likely worked on a native-Linux Docker host and silently broke on Docker Desktop.

Fix direction (pick one; option A preferred)

A. Move the dispatch broker transport off a host-bind-mounted unix socket. Serve it over TCP on the host loopback/host-network (the surface container already has a host-net path, --host-net ward#330, and HostNet in container_compute) and export WARD_DISPATCH_BROKER_SOCK (or a new WARD_DISPATCH_BROKER_ADDR) as a host.docker.internal:PORT / 127.0.0.1:PORT address. TCP traverses the VM boundary cleanly; unix-socket bind-mounts do not.
B. Run a dispatch relay inside the container (like the credential broker) that forwards to the host over an already-working channel, instead of bind-mounting the host socket directly.
C. If a host-Docker (native Linux) deployment must keep the bind-mount, detect Docker Desktop/linuxkit (e.g. kernel contains linuxkit, or /proc/1/cgroup / Docker context probe) and switch transport to A automatically.

Must-fix secondary papercuts (do these regardless of transport)

Fail loud, not cryptic. When WARD_DISPATCH_BROKER_SOCK resolves to a non-socket (a dir) or is missing, the client (cmd/ward/broker_client.go) must error with a clear message like dispatch broker socket not present at <path> (host mount did not deliver a socket - see ward#<this>), never the bare connect: permission denied.
Guard against hitting the wrong broker. Dialing the credential broker yields unsupported protocol version 0 (want 1). The dispatch client should surface that as wrong broker: <path> is the credential broker, not the dispatch broker so the next person is not misled.

Acceptance criteria

From a fresh read-only director surface container on Docker Desktop / linuxkit, warded engineer <ref> --driver goose successfully reaches the host dispatch broker and launches a sibling carry (no empty-dir, no permission-denied, no protocol-version error).
A regression test or a documented manual repro proving the transport works under the VM-backed Docker host class.
The two papercut error messages above are implemented and unit-tested.
docs/broker.md (and docs/agent-director.md if it describes dispatch) updated to state the transport and the Docker Desktop constraint.

Files to start from

cmd/ward/agent_dispatch_broker.go (startHostDispatchBroker, dispatchBrokerMount, envDispatchBrokerSocket)
cmd/ward/agent_director_surface.go (attachHostDispatchBroker, line 149)
cmd/ward/container_compute.go (containerDispatchBrokerSock line 58; DispatchBrokerSock env export ~429; HostNet)
cmd/ward/broker_client.go (dial + error messages)
docs/broker.md

When done

Run the gate, commit, push to canonical main, and update docs/FEATURES.md if dispatch transport is inventoried there. Comment on ward#381 that the dispatch path is unblocked, then the agent-proxy chain can be kicked off with warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose.

## Summary (for the engineer: this is a real, reproducible infra bug - read it all before touching code) In a read-only **director surface** container, sibling dispatch (`warded engineer ...`) is 100% broken because the **host dispatch broker's unix socket never arrives in the container**. The mount target `/run/ward/dispatch-broker.sock` shows up as an **empty directory**, not a socket. This blocks the entire `agent-proxy` build chain (`coilyco-flight-deck/agent-proxy#5`..`#17`); tracked as the blocker in **ward#381** - this issue is the fix. ## Exact symptoms (reproduced) ``` $ echo $WARD_DISPATCH_BROKER_SOCK /run/ward/dispatch-broker.sock $ ls -la /run/ward/ srw-rw---- 1 root ubuntu 0 /run/ward/broker.sock # the CREDENTIAL broker (works) drwxr-xr-x 2 root root 40 /run/ward/dispatch-broker.sock/ # the DISPATCH broker: an EMPTY DIR, not a socket $ warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose ward: dispatch broker unavailable: dial /run/ward/dispatch-broker.sock: connect: permission denied # it's a directory # pointing at the only real socket reaches the WRONG broker: $ WARD_DISPATCH_BROKER_SOCK=/run/ward/broker.sock warded engineer ... ward: dispatch broker: broker: unsupported protocol version 0 (want 1) # that's the credential broker, not the dispatch broker ``` Host: container kernel `6.12.76-linuxkit` => this is **Docker Desktop's LinuxKit VM**. ward `v0.204.0`. ## Root cause (high confidence) There are **two** brokers and they are provisioned differently: - **Credential broker** - `cmd/ward/broker.go` (`defaultBrokerSocket = "/run/ward/broker.sock"`, line ~25). Runs **inside** the container as an entrypoint daemon ("Entrypoint-internal root credential broker daemon"). The socket is created in-container, so it is always present. This one works. - **Dispatch broker** - `cmd/ward/agent_dispatch_broker.go`. `startHostDispatchBroker` (line ~36) does `os.MkdirTemp("", "ward-dispatch-broker-*")` on the **host**, `net.Listen("unix", <tmpdir>/broker.sock)`, then `dispatchBrokerMount(hostSock)` (line ~293) **bind-mounts that host socket** to `containerDispatchBrokerSock = "/run/ward/dispatch-broker.sock"` (`cmd/ward/container_compute.go:58`). `attachHostDispatchBroker` (`cmd/ward/agent_director_surface.go:149`) wires it; `container_compute.go:~429` exports the `WARD_DISPATCH_BROKER_SOCK` env. The defect: **bind-mounting a host unix socket into a container does not work under Docker Desktop / linuxkit.** The host filesystem is shared into the VM via virtiofs/gRPC-FUSE, and a live unix socket node does not traverse that boundary. Docker, finding no bindable socket inode at mount time, **creates an empty directory at the target** - precisely the observed `/run/ward/dispatch-broker.sock/` empty dir. The credential broker avoids this only because it is created *inside* the container, never bind-mounted. So the host-unix-socket-bind-mount transport is fundamentally unavailable on this host class. It likely worked on a native-Linux Docker host and silently broke on Docker Desktop. ## Fix direction (pick one; option A preferred) - **A. Move the dispatch broker transport off a host-bind-mounted unix socket.** Serve it over **TCP on the host loopback/host-network** (the surface container already has a host-net path, `--host-net` ward#330, and `HostNet` in container_compute) and export `WARD_DISPATCH_BROKER_SOCK` (or a new `WARD_DISPATCH_BROKER_ADDR`) as a `host.docker.internal:PORT` / `127.0.0.1:PORT` address. TCP traverses the VM boundary cleanly; unix-socket bind-mounts do not. - **B. Run a dispatch *relay* inside the container** (like the credential broker) that forwards to the host over an already-working channel, instead of bind-mounting the host socket directly. - **C. If a host-Docker (native Linux) deployment must keep the bind-mount, detect Docker Desktop/linuxkit** (e.g. kernel contains `linuxkit`, or `/proc/1/cgroup` / Docker context probe) and switch transport to A automatically. ## Must-fix secondary papercuts (do these regardless of transport) 1. **Fail loud, not cryptic.** When `WARD_DISPATCH_BROKER_SOCK` resolves to a non-socket (a dir) or is missing, the client (`cmd/ward/broker_client.go`) must error with a clear message like `dispatch broker socket not present at <path> (host mount did not deliver a socket - see ward#<this>)`, never the bare `connect: permission denied`. 2. **Guard against hitting the wrong broker.** Dialing the credential broker yields `unsupported protocol version 0 (want 1)`. The dispatch client should surface that as `wrong broker: <path> is the credential broker, not the dispatch broker` so the next person is not misled. ## Acceptance criteria - From a fresh read-only director surface container on Docker Desktop / linuxkit, `warded engineer <ref> --driver goose` successfully reaches the host dispatch broker and launches a sibling carry (no empty-dir, no permission-denied, no protocol-version error). - A regression test or a documented manual repro proving the transport works under the VM-backed Docker host class. - The two papercut error messages above are implemented and unit-tested. - `docs/broker.md` (and `docs/agent-director.md` if it describes dispatch) updated to state the transport and the Docker Desktop constraint. ## Files to start from - `cmd/ward/agent_dispatch_broker.go` (`startHostDispatchBroker`, `dispatchBrokerMount`, `envDispatchBrokerSocket`) - `cmd/ward/agent_director_surface.go` (`attachHostDispatchBroker`, line 149) - `cmd/ward/container_compute.go` (`containerDispatchBrokerSock` line 58; `DispatchBrokerSock` env export ~429; `HostNet`) - `cmd/ward/broker_client.go` (dial + error messages) - `docs/broker.md` ## When done Run the gate, commit, push to canonical `main`, and update `docs/FEATURES.md` if dispatch transport is inventoried there. Comment on **ward#381** that the dispatch path is unblocked, then the `agent-proxy` chain can be kicked off with `warded engineer coilyco-flight-deck/agent-proxy#5 --driver goose`.

coilyco-ops commented

2026-06-26 10:00:48 +00:00

Member

🔒 Reserved by ward agent --driver goose — container engineer-goose-ward-382 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-26T10:00:46Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Goose, via ward agent

🔒 Reserved by `ward agent --driver goose` — container `engineer-goose-ward-382` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-26T10:00:46Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Goose, via `ward agent`

coilysiren commented

2026-06-26 10:08:43 +00:00

Author

Owner

^ it appears this agent crashed

coilysiren commented

2026-07-01 03:06:16 +00:00

Author

Owner

Decision from Kai (2026-07-01): go with option A, TCP only - no unix-socket path retained.

Re-confirmed the root cause independently from a live read-only director surface on Docker Desktop (6.12.76-linuxkit, ward v0.205.0): /run/ward/dispatch-broker.sock is an empty directory, connect() -> EACCES; WARD_BROKER_SOCK=/run/ward/broker.sock is the credential broker (in-container, works) and answers with unsupported protocol version 0 (want 1) when the dispatch client dials it. Matches this issue exactly.

Scope the fix as TCP-only for simplicity - do not keep the unix socket as a Linux fast path:

startHostDispatchBroker -> net.Listen("tcp", ...); return host+port(+nonce) instead of a socket path. Delete the os.MkdirTemp + net.Listen("unix", ...) machinery.
Container dials host.docker.internal:<port> in sendDispatchBrokerRequest (confirmed: host.docker.internal and gateway.docker.internal both resolve from inside a live surface container).
Delete dispatchBrokerMount and the -v socket mount (agent_director_surface.go:159, container_compute.go), and the containerDispatchBrokerSock path constant.
TCP drops the 0660-socket access control, so add an explicit Token nonce field on dispatchBrokerRequest, pass it via env, verify it in handleHostDispatchBrokerConn.
Keep the two fail-loud papercuts from the issue body (clear error on a non-socket/missing addr; "wrong broker" hint on a proto-version mismatch).

Note: the prior ward agent --driver goose carry on this issue crashed (see comment above), so it needs a fresh dispatch. Filed ward#391 as a duplicate of this before finding it - closing #391 and pointing here.

**Decision from Kai (2026-07-01): go with option A, TCP only - no unix-socket path retained.** Re-confirmed the root cause independently from a live read-only director surface on Docker Desktop (`6.12.76-linuxkit`, ward v0.205.0): `/run/ward/dispatch-broker.sock` is an empty directory, `connect()` -> EACCES; `WARD_BROKER_SOCK=/run/ward/broker.sock` is the credential broker (in-container, works) and answers with `unsupported protocol version 0 (want 1)` when the dispatch client dials it. Matches this issue exactly. Scope the fix as TCP-only for simplicity - do not keep the unix socket as a Linux fast path: - `startHostDispatchBroker` -> `net.Listen("tcp", ...)`; return host+port(+nonce) instead of a socket path. Delete the `os.MkdirTemp` + `net.Listen("unix", ...)` machinery. - Container dials `host.docker.internal:<port>` in `sendDispatchBrokerRequest` (confirmed: `host.docker.internal` and `gateway.docker.internal` both resolve from inside a live surface container). - Delete `dispatchBrokerMount` and the `-v` socket mount (`agent_director_surface.go:159`, `container_compute.go`), and the `containerDispatchBrokerSock` path constant. - TCP drops the `0660`-socket access control, so add an explicit `Token` nonce field on `dispatchBrokerRequest`, pass it via env, verify it in `handleHostDispatchBrokerConn`. - Keep the two fail-loud papercuts from the issue body (clear error on a non-socket/missing addr; "wrong broker" hint on a proto-version mismatch). Note: the prior `ward agent --driver goose` carry on this issue crashed (see comment above), so it needs a fresh dispatch. Filed ward#391 as a duplicate of this before finding it - closing #391 and pointing here.

coilysiren referenced this issue

2026-07-01 03:06:22 +00:00

Dispatch broker breaks on Docker Desktop: host unix socket bind-mount lands as an empty dir #391

coilyco-ops commented

2026-07-01 03:09:06 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-382 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-01T03:09:03Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-382` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-01T03:09:03Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-01 03:21:18 +00:00

fix(broker): fail-loud papercuts for the TCP dispatch broker (closes #382)

coilysiren closed this issue

2026-07-01 03:21:18 +00:00

coilysiren referenced this issue

2026-07-01 03:22:06 +00:00

agent-proxy director: dispatch broker unusable in container (proto v0 vs warded v1; $WARD_DISPATCH_BROKER_SOCK is an empty dir) #381

coilysiren commented

2026-07-01 03:22:27 +00:00

Author

Owner

WARD-OUTCOME: done - dispatch broker's TCP transport (from ward#391) plus the two #382 fail-loud papercuts + docs landed on main, closing this.

Honest retro: most of the substance was already done before I arrived. The transport root cause - a host unix socket bind-mounted into a Docker Desktop / linuxkit container materializing as an empty dir - had been fixed by the TCP-over-gateway move under ward#391 (which Kai flagged as a duplicate of this), and it was already on main with solid tests. So the real work here was the tail of #382's acceptance criteria that #391's commit didn't cover: the two fail-loud papercuts (unreachable-broker and wrong-broker hints) with unit tests, and the docs/broker.md transport note.

What fought back: the docs size cap. docs/broker.md was sitting at exactly its 80-line / 4000-char limit, so adding a section meant trimming existing prose almost char-for-char - I went several rounds shaving parentheticals to land at 3999. Slightly tedious, but the cap is doing its job. Also worth noting the papercuts are somewhat belt-and-suspenders under TCP-only: you can't accidentally point the dispatch client at the credential socket via a TCP addr anymore, but Kai explicitly asked to keep both, and the wrong-broker proto-version hint is still genuinely useful if someone misconfigures the addr.

Confidence: high on the papercuts and docs (unit-tested, full suite + vet green, all pre-commit hooks pass). One rough edge I did not touch: the repo ships 33 pre-existing golangci-lint findings on clean main (errcheck, gosec, revive, etc.) - none in files I changed, and the commit-time gate is the catalog suite not golangci-lint, so they don't block. Might be worth a follow-up issue to clean those up, but that's out of scope here. No end-to-end run from a live Docker Desktop surface on my side - that verification lives with the agent-proxy chain kickoff noted on ward#381.

WARD-OUTCOME: done - dispatch broker's TCP transport (from ward#391) plus the two #382 fail-loud papercuts + docs landed on main, closing this. Honest retro: most of the substance was already done before I arrived. The transport root cause - a host unix socket bind-mounted into a Docker Desktop / linuxkit container materializing as an empty dir - had been fixed by the TCP-over-gateway move under ward#391 (which Kai flagged as a duplicate of this), and it was already on main with solid tests. So the real work here was the tail of #382's acceptance criteria that #391's commit didn't cover: the two fail-loud papercuts (unreachable-broker and wrong-broker hints) with unit tests, and the `docs/broker.md` transport note. What fought back: the docs size cap. `docs/broker.md` was sitting at exactly its 80-line / 4000-char limit, so adding a section meant trimming existing prose almost char-for-char - I went several rounds shaving parentheticals to land at 3999. Slightly tedious, but the cap is doing its job. Also worth noting the papercuts are somewhat belt-and-suspenders under TCP-only: you can't accidentally point the dispatch client at the credential *socket* via a TCP addr anymore, but Kai explicitly asked to keep both, and the wrong-broker proto-version hint is still genuinely useful if someone misconfigures the addr. Confidence: high on the papercuts and docs (unit-tested, full suite + vet green, all pre-commit hooks pass). One rough edge I did **not** touch: the repo ships 33 pre-existing `golangci-lint` findings on clean main (errcheck, gosec, revive, etc.) - none in files I changed, and the commit-time gate is the catalog suite not golangci-lint, so they don't block. Might be worth a follow-up issue to clean those up, but that's out of scope here. No end-to-end run from a live Docker Desktop surface on my side - that verification lives with the agent-proxy chain kickoff noted on ward#381.