coilyco-flight-deck/ward

Fork 0

Dispatch broker breaks on Docker Desktop: host unix socket bind-mount lands as an empty dir #391

New issue

Closed

opened 2026-07-01 02:57:13 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-07-01 02:57:13 +00:00

Owner

Dispatch broker socket is a bind-mounted host unix socket - breaks on Docker Desktop (macOS)

Symptom

From inside a warded director read-only surface session, any sibling dispatch fails:

warded engineer coilyco-flight-deck/ward#N
# ward: dispatch broker unavailable: dial /run/ward/dispatch-broker.sock: connect: permission denied

/run/ward/dispatch-broker.sock is an empty directory, not a socket:

$ python3 -c 'import os,stat;p="/run/ward/dispatch-broker.sock";print("isdir",os.path.isdir(p),"issock",stat.S_ISSOCK(os.stat(p).st_mode))'
isdir True issock False
# connect() -> [Errno 13] Permission denied

The container's env is correct (WARD_DISPATCH_BROKER_SOCK=/run/ward/dispatch-broker.sock, WARD_READONLY=1), so the in-container forward path (maybeForwardAgentDispatchToHostBroker) fires and dials - it just dials a directory.

Root cause

Two brokers live under /run/ward/, provisioned in opposite directions:

WARD_BROKER_SOCK=/run/ward/broker.sock works - the container entrypoint's start_broker (cmd/ward/containerassets/entrypoint.sh:142) runs ward container broker --socket ... inside the container. Native in-container unix socket, never crosses the VM boundary.
WARD_DISPATCH_BROKER_SOCK=/run/ward/dispatch-broker.sock breaks - startHostDispatchBroker (cmd/ward/agent_dispatch_broker.go:36) does net.Listen("unix", filepath.Join(os.MkdirTemp(""), "broker.sock")) on the host, and dispatchBrokerMount (:293) bind-mounts that host socket into the container at /run/ward/dispatch-broker.sock (wired in agent_director_surface.go:159).

The host is Docker Desktop (Linux 6.12.76-linuxkit). A unix domain socket created on the macOS host and bind-mounted into a container does not survive Docker Desktop's file-sharing layer - virtiofs/gRPC-FUSE cannot proxy a socket kernel-object, and -v "auto-create missing source" semantics materialize the mount target as an empty directory. Hence the directory, and hence every dispatch dial gets EACCES.

The feature (commit 686fe1d feat(agent): broker director surface dispatch, ward#378) was evidently built/tested on a Linux host, where a host->container unix-socket bind-mount works. It breaks the moment warded director runs on a Docker Desktop / macOS host - which is the primary dev surface.

Proposed fix: TCP only

Replace the host->container unix-socket bind-mount entirely with a TCP transport that crosses the Docker Desktop VM boundary. No unix-socket path is retained - one transport on every host, for simplicity. host.docker.internal and gateway.docker.internal both resolve from inside the container (confirmed), so:

startHostDispatchBroker listens on TCP (net.Listen("tcp", ...)), bound to the docker gateway / an address the container reaches, and returns host+port (+nonce) instead of a socket path.
Container dials host.docker.internal:<port> in sendDispatchBrokerRequest (TCP dial, not unix).
A TCP port lacks the 0660-socket access control the current design leans on, so mint a per-launch shared-secret nonce, pass it via env, and verify it in handleHostDispatchBrokerConn. Add an explicit Token field on dispatchBrokerRequest rather than overloading Requester.

Rip out the unix-socket machinery as part of this - do not leave it as a Linux fallback:

dispatchBrokerMount and the -v socket mount (agent_director_surface.go:159, container_compute.go) - deleted.
containerDispatchBrokerSock path constant - replaced by host/port/nonce env.
net.Listen("unix", ...) + os.MkdirTemp in startHostDispatchBroker - replaced by the TCP listener.

The wire protocol is already trivial JSON request/response over a net.Conn (sendDispatchBrokerRequest / serveHostDispatchBroker), so only the listen/dial + auth change:

cmd/ward/agent_dispatch_broker.go - listen TCP, dial TCP, Token nonce check
cmd/ward/agent_director_surface.go - attachHostDispatchBroker returns host+port+nonce, no mount
cmd/ward/container_compute.go - env wiring (host/port/nonce), drop dispatchBrokerMount + the socket path constant
cmd/ward/containerassets/entrypoint.sh + docs/agent-surface.md - update the "how this is wired" note

This matches the existing mac-proxy / --ts-sidecar precedent (network transport over the docker network, not host file-sharing).

Blast radius

Blocks the entire in-container director-surface dispatch path on any Docker Desktop / macOS host. Concretely, it currently blocks dispatching ward#388 (make warded advisor interactive by default) from a surfaced session. Operators can still dispatch directly from a host terminal (warded engineer <ref> runs host docker directly, not through this broker), so this is specifically the surface->host forward hop.

## Dispatch broker socket is a bind-mounted host unix socket - breaks on Docker Desktop (macOS) ### Symptom From inside a `warded director` read-only surface session, any sibling dispatch fails: ``` warded engineer coilyco-flight-deck/ward#N # ward: dispatch broker unavailable: dial /run/ward/dispatch-broker.sock: connect: permission denied ``` `/run/ward/dispatch-broker.sock` is an empty **directory**, not a socket: ``` $ python3 -c 'import os,stat;p="/run/ward/dispatch-broker.sock";print("isdir",os.path.isdir(p),"issock",stat.S_ISSOCK(os.stat(p).st_mode))' isdir True issock False # connect() -> [Errno 13] Permission denied ``` The container's env is correct (`WARD_DISPATCH_BROKER_SOCK=/run/ward/dispatch-broker.sock`, `WARD_READONLY=1`), so the in-container forward path (`maybeForwardAgentDispatchToHostBroker`) fires and dials - it just dials a directory. ### Root cause Two brokers live under `/run/ward/`, provisioned in opposite directions: - `WARD_BROKER_SOCK=/run/ward/broker.sock` **works** - the container entrypoint's `start_broker` (`cmd/ward/containerassets/entrypoint.sh:142`) runs `ward container broker --socket ...` *inside* the container. Native in-container unix socket, never crosses the VM boundary. - `WARD_DISPATCH_BROKER_SOCK=/run/ward/dispatch-broker.sock` **breaks** - `startHostDispatchBroker` (`cmd/ward/agent_dispatch_broker.go:36`) does `net.Listen("unix", filepath.Join(os.MkdirTemp(""), "broker.sock"))` on the **host**, and `dispatchBrokerMount` (`:293`) bind-mounts that host socket into the container at `/run/ward/dispatch-broker.sock` (wired in `agent_director_surface.go:159`). The host is Docker Desktop (`Linux 6.12.76-linuxkit`). A unix domain socket created on the macOS host and bind-mounted into a container does **not** survive Docker Desktop's file-sharing layer - virtiofs/gRPC-FUSE cannot proxy a socket kernel-object, and `-v` "auto-create missing source" semantics materialize the mount target as an empty directory. Hence the directory, and hence every dispatch dial gets EACCES. The feature (commit `686fe1d feat(agent): broker director surface dispatch`, ward#378) was evidently built/tested on a Linux host, where a host->container unix-socket bind-mount works. It breaks the moment `warded director` runs on a Docker Desktop / macOS host - which is the primary dev surface. ### Proposed fix: TCP only Replace the host->container unix-socket bind-mount **entirely** with a TCP transport that crosses the Docker Desktop VM boundary. No unix-socket path is retained - one transport on every host, for simplicity. `host.docker.internal` and `gateway.docker.internal` both resolve from inside the container (confirmed), so: 1. `startHostDispatchBroker` listens on **TCP** (`net.Listen("tcp", ...)`), bound to the docker gateway / an address the container reaches, and returns host+port (+nonce) instead of a socket path. 2. Container dials `host.docker.internal:<port>` in `sendDispatchBrokerRequest` (TCP dial, not `unix`). 3. A TCP port lacks the `0660`-socket access control the current design leans on, so mint a **per-launch shared-secret nonce**, pass it via env, and verify it in `handleHostDispatchBrokerConn`. Add an explicit `Token` field on `dispatchBrokerRequest` rather than overloading `Requester`. Rip out the unix-socket machinery as part of this - do not leave it as a Linux fallback: - `dispatchBrokerMount` and the `-v` socket mount (`agent_director_surface.go:159`, `container_compute.go`) - deleted. - `containerDispatchBrokerSock` path constant - replaced by host/port/nonce env. - `net.Listen("unix", ...)` + `os.MkdirTemp` in `startHostDispatchBroker` - replaced by the TCP listener. The wire protocol is already trivial JSON request/response over a `net.Conn` (`sendDispatchBrokerRequest` / `serveHostDispatchBroker`), so only the listen/dial + auth change: - `cmd/ward/agent_dispatch_broker.go` - listen TCP, dial TCP, `Token` nonce check - `cmd/ward/agent_director_surface.go` - `attachHostDispatchBroker` returns host+port+nonce, no mount - `cmd/ward/container_compute.go` - env wiring (host/port/nonce), drop `dispatchBrokerMount` + the socket path constant - `cmd/ward/containerassets/entrypoint.sh` + `docs/agent-surface.md` - update the "how this is wired" note This matches the existing `mac-proxy` / `--ts-sidecar` precedent (network transport over the docker network, not host file-sharing). ### Blast radius Blocks the entire in-container director-surface dispatch path on any Docker Desktop / macOS host. Concretely, it currently blocks dispatching ward#388 (make `warded advisor` interactive by default) from a surfaced session. Operators can still dispatch directly from a host terminal (`warded engineer <ref>` runs host docker directly, not through this broker), so this is specifically the surface->host forward hop.

coilyco-ops commented

2026-07-01 02:58:54 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-391 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-01T02:58:50Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-391` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-01T02:58:50Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue

2026-07-01 03:06:16 +00:00

Fix: host dispatch-broker unix socket does not bind-mount into containers under Docker Desktop / linuxkit (lands as empty dir) #382

coilysiren commented

2026-07-01 03:06:22 +00:00

Author

Owner

Duplicate of ward#382 (and ward#381), which already has the full root-cause and the TCP-only decision. Closing in favor of #382.

coilysiren closed this issue

2026-07-01 03:06:22 +00:00

coilysiren referenced this issue from a commit

2026-07-01 03:08:45 +00:00

fix(agent): TCP dispatch broker survives Docker Desktop (ward#391)

coilysiren commented

2026-07-01 03:09:32 +00:00

Author

Owner

WARD-OUTCOME: done - dispatch broker now speaks TCP over the docker gateway, so surface-session dispatch survives Docker Desktop.

This one was a pleasure to carry because you'd already done the hard diagnostic work in the issue - root cause pinned to the file:line, the fix shape spelled out (TCP + nonce, rip out the unix-socket path). I basically executed the plan. The transport swap itself was small: the wire protocol was already JSON over a net.Conn, so only listen/dial + auth changed.

Two things fought back, both cosmetic rather than logical. First, the code-comments pre-commit hook caps contiguous comment blocks at 2 lines, and I'd written generous 4-7 line explanations - had to compress every one and drop a blank-line separator where mine abutted an existing block. Second, docs/agent-surface.md has a 4000-char cap and my "why TCP" paragraph blew past it; took a few passes to trim under. Good discipline forcing, annoying in the moment.

Confidence is high on the Go logic and the test coverage (token gate, add-host wiring, canonical request all pinned), but I want to flag the one thing I could NOT exercise in-container: the actual host.docker.internal round-trip on a real Docker Desktop host. The unit tests use loopback TCP + net.Pipe, which proves the protocol and auth but not that the gateway name resolves and routes end to end. Worth a real warded director smoke test on the Mac to close that gap - that's the only follow-up I'd file. I bound the host listener to 0.0.0.0 (nonce-guarded, gosec-suppressed with justification) rather than hunting a tighter gateway-only bind; if that feels too broad for a dev laptop it'd be a reasonable hardening later.

WARD-OUTCOME: done - dispatch broker now speaks TCP over the docker gateway, so surface-session dispatch survives Docker Desktop. This one was a pleasure to carry because you'd already done the hard diagnostic work in the issue - root cause pinned to the file:line, the fix shape spelled out (TCP + nonce, rip out the unix-socket path). I basically executed the plan. The transport swap itself was small: the wire protocol was already JSON over a net.Conn, so only listen/dial + auth changed. Two things fought back, both cosmetic rather than logical. First, the `code-comments` pre-commit hook caps contiguous comment blocks at 2 lines, and I'd written generous 4-7 line explanations - had to compress every one and drop a blank-line separator where mine abutted an existing block. Second, docs/agent-surface.md has a 4000-char cap and my "why TCP" paragraph blew past it; took a few passes to trim under. Good discipline forcing, annoying in the moment. Confidence is high on the Go logic and the test coverage (token gate, add-host wiring, canonical request all pinned), but I want to flag the one thing I could NOT exercise in-container: the actual host.docker.internal round-trip on a real Docker Desktop host. The unit tests use loopback TCP + net.Pipe, which proves the protocol and auth but not that the gateway name resolves and routes end to end. Worth a real `warded director` smoke test on the Mac to close that gap - that's the only follow-up I'd file. I bound the host listener to 0.0.0.0 (nonce-guarded, gosec-suppressed with justification) rather than hunting a tighter gateway-only bind; if that feels too broad for a dev laptop it'd be a reasonable hardening later.

coilysiren referenced this issue

2026-07-01 03:22:27 +00:00

Fix: host dispatch-broker unix socket does not bind-mount into containers under Docker Desktop / linuxkit (lands as empty dir) #382