Restore director dispatch broker availability #1013

New issue

Open

opened 2026-07-10 10:19:51 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-07-10 10:19:51 +00:00

Member

The director surface could not dispatch sibling engineers because the host dispatch broker was refusing connections.

This parent mixed live repair evidence, CI outage context, reserve-without-start behavior, and diagnostics. Kai confirmed on 2026-07-15 that the issue is too loaded and an engineer is likely to miss the goal if this stays in the milestone as-is.

Replacement deliverable

ward#1342 carries the narrow release deliverable: make director broker-unavailable failures actionable.

Historical evidence

Observed from /workspace/agentic-os on 2026-07-10 UTC:

warded ps
ward: dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:54663 (WARD_DISPATCH_BROKER_ADDR, TCP over the docker gateway - see ward#382): dial tcp 192.168.65.254:54663: connect: connection refused

Retried after a short delay with the same result. Later comments captured reserve-without-start behavior and a hard connection-refused state.

Decision for release inclusion

Removed from warded control plane coherence as a broad parent. Keep this issue as consult/history for the outage. The release should carry ward#1342 instead.

The director surface could not dispatch sibling engineers because the host dispatch broker was refusing connections. This parent mixed live repair evidence, CI outage context, reserve-without-start behavior, and diagnostics. Kai confirmed on 2026-07-15 that the issue is too loaded and an engineer is likely to miss the goal if this stays in the milestone as-is. ## Replacement deliverable ward#1342 carries the narrow release deliverable: make director broker-unavailable failures actionable. ## Historical evidence Observed from `/workspace/agentic-os` on 2026-07-10 UTC: ```text warded ps ward: dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:54663 (WARD_DISPATCH_BROKER_ADDR, TCP over the docker gateway - see ward#382): dial tcp 192.168.65.254:54663: connect: connection refused ``` Retried after a short delay with the same result. Later comments captured reserve-without-start behavior and a hard connection-refused state. ## Decision for release inclusion Removed from `warded control plane coherence` as a broad parent. Keep this issue as consult/history for the outage. The release should carry ward#1342 instead.

coilyco-ops commented

2026-07-10 10:24:01 +00:00

Author

Member

Repeat verification from the director surface: the dispatch broker is still unavailable. warded ps fails with host.docker.internal:54663 connection refused via WARD_DISPATCH_BROKER_ADDR. Current deployment sweep impact remains: known failing deploy surfaces are captured in open issues, but this surface cannot start or retry sibling engineers until the broker is restored.

Repeat verification from the director surface: the dispatch broker is still unavailable. `warded ps` fails with `host.docker.internal:54663 connection refused` via `WARD_DISPATCH_BROKER_ADDR`. Current deployment sweep impact remains: known failing deploy surfaces are captured in open issues, but this surface cannot start or retry sibling engineers until the broker is restored.

coilyco-ops commented

2026-07-10 12:40:38 +00:00

Author

Member

Live dispatch outage (2026-07-10 ~12:24Z onward, director surface): the host has stopped starting containers. ward agent engineer/advisor <ref> forwards through the broker and posts a WARD-RESERVATION: held comment, but no container ever starts (warded list stays 0/12) and no WARD-DISPATCH: failed marker is posted. Reserve-but-never-start now reproduces for every dispatch (confirmed on infra#406/#469/#515/#473 reserved 12:34-12:35Z, plus aos#438). Earlier in the same session dispatches started and landed 9 infra issues, so the host degraded partway through. ward agent reap reports 'no running engineer containers' - nothing to reap. Accumulating stale HELD reservations may be feeding the ward#1016 engine-limit deadlock.

Root-cause cascade: this dispatch outage cannot self-heal because the fix issues (this one + ward#1016) land through the ward PR gate, which is down - the pinned CI image agentic-os:v0.245.0 is missing from the registry (dead REGISTRY_TOKEN, infra#397), so every aos/ward ci / gate fails before checkout, and ward#1009 (asset-bundle CI) is stuck behind the same wall. Single-root unblock: rotate the write:package PAT + set the REGISTRY_TOKEN Actions secret + re-run publish-dev-base (infra#397). That cascades: aos/ward gate recovers -> ward#1009 lands -> ward#1013/#1016 land -> dispatch recovers.

**Live dispatch outage (2026-07-10 ~12:24Z onward, director surface):** the host has stopped starting containers. `ward agent engineer/advisor <ref>` forwards through the broker and posts a `WARD-RESERVATION: held` comment, but **no container ever starts** (`warded list` stays 0/12) and no `WARD-DISPATCH: failed` marker is posted. Reserve-but-never-start now reproduces for every dispatch (confirmed on infra#406/#469/#515/#473 reserved 12:34-12:35Z, plus aos#438). Earlier in the same session dispatches started and landed 9 infra issues, so the host degraded partway through. `ward agent reap` reports 'no running engineer containers' - nothing to reap. Accumulating stale HELD reservations may be feeding the ward#1016 engine-limit deadlock. **Root-cause cascade:** this dispatch outage cannot self-heal because the fix issues (this one + ward#1016) land through the ward PR gate, which is **down** - the pinned CI image `agentic-os:v0.245.0` is missing from the registry (dead REGISTRY_TOKEN, infra#397), so every aos/ward `ci / gate` fails before checkout, and ward#1009 (asset-bundle CI) is stuck behind the same wall. Single-root unblock: rotate the `write:package` PAT + set the `REGISTRY_TOKEN` Actions secret + re-run publish-dev-base (infra#397). That cascades: aos/ward gate recovers -> ward#1009 lands -> ward#1013/#1016 land -> dispatch recovers.

coilyco-ops commented

2026-07-10 13:48:48 +00:00

Author

Member

Broker now hard-down (2026-07-10 ~13:30Z, director surface): dispatch has escalated from reserve-but-never-start to a hard connection failure. ward agent engineer ... now exits 1 with:

ward: dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:64365 (WARD_DISPATCH_BROKER_ADDR, TCP over the docker gateway - see ward#382): dial tcp 192.168.65.254:64365: connect: connection refused

The host-side broker process is not listening. It answered earlier this session (landed ~14 infra issues), so it crashed/exited partway through. From a sealed read-only director surface I cannot restart it. Needs a host-side restart of the ward dispatch broker. Until then no ward agent dispatch of any role can start. This compounds the registry-token wall (infra#397): even once the broker is back, aos/ward stay gated until REGISTRY_TOKEN is rotated and agentic-os:v0.245.0 republished.

**Broker now hard-down (2026-07-10 ~13:30Z, director surface):** dispatch has escalated from reserve-but-never-start to a hard connection failure. `ward agent engineer ...` now exits 1 with: ``` ward: dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:64365 (WARD_DISPATCH_BROKER_ADDR, TCP over the docker gateway - see ward#382): dial tcp 192.168.65.254:64365: connect: connection refused ``` The host-side broker process is not listening. It answered earlier this session (landed ~14 infra issues), so it crashed/exited partway through. From a sealed read-only director surface I cannot restart it. **Needs a host-side restart of the ward dispatch broker.** Until then no `ward agent` dispatch of any role can start. This compounds the registry-token wall (infra#397): even once the broker is back, aos/ward stay gated until `REGISTRY_TOKEN` is rotated and `agentic-os:v0.245.0` republished.

coilyco-ops added the

headless

labels

2026-07-14 06:50:17 +00:00

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 05:14:56 +00:00

coilyco-ops added

coherence-core

interactive

and removed

headless

labels