Director surface dispatch broker address refuses connections, blocking sibling warded dispatch

coilyco-ops commented

2026-07-14 07:41:53 +00:00

Owner

Problem

From the read-only director surface on 2026-07-14, sibling dispatch is blocked even though the dispatch broker env is present:

WARD_DISPATCH_BROKER_ADDR=host.docker.internal:54293
WARD_DISPATCH_BROKER_TOKEN=<present>

Both ward agent pr status coilyco-flight-deck/ward#1236 and warded engineer coilyco-flight-deck/agentic-os#466 --harness codex --override-reservation ... failed with:

ward: dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:54293 (WARD_DISPATCH_BROKER_ADDR, TCP over the docker gateway - see ward#382): dial tcp 192.168.65.254:54293: connect: connection refused

This is distinct from the closed ward#382 unix-socket transport bug: the surface now has the TCP address and token, but the advertised broker port is not listening from inside the container. ward agent dispatch-health and warded director queue still work, so the failure is specific to commands that need the host dispatch broker.

Impact

The director cannot dispatch sibling engineers from this surface, which blocks autonomous burn-down of queued headless issues across ward, agentic-os, and cli-guard.

Do

Reproduce from a fresh read-only director surface with WARD_DISPATCH_BROKER_ADDR set.
Fix the host broker lifecycle or surface env handoff so the advertised TCP address accepts dispatch requests for the lifetime of the surface.
If the broker is intentionally absent, fail before surfacing a stale/unreachable address and make warded explain the missing broker state.
Add coverage for an advertised-but-unreachable broker address if there is a suitable unit seam.

Acceptance

From a fresh director surface, warded engineer <headless issue> --harness codex reaches the host broker and either accepts or policy-rejects the dispatch, but does not fail with TCP connection refused.
ward agent pr status <repo#pr> works from the same surface or degrades through the intended non-broker path.
The error message distinguishes broker-not-running from policy rejection and stale reservation.

## Problem From the read-only director surface on 2026-07-14, sibling dispatch is blocked even though the dispatch broker env is present: ```text WARD_DISPATCH_BROKER_ADDR=host.docker.internal:54293 WARD_DISPATCH_BROKER_TOKEN=<present> ``` Both `ward agent pr status coilyco-flight-deck/ward#1236` and `warded engineer coilyco-flight-deck/agentic-os#466 --harness codex --override-reservation ...` failed with: ```text ward: dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:54293 (WARD_DISPATCH_BROKER_ADDR, TCP over the docker gateway - see ward#382): dial tcp 192.168.65.254:54293: connect: connection refused ``` This is distinct from the closed ward#382 unix-socket transport bug: the surface now has the TCP address and token, but the advertised broker port is not listening from inside the container. `ward agent dispatch-health` and `warded director queue` still work, so the failure is specific to commands that need the host dispatch broker. ## Impact The director cannot dispatch sibling engineers from this surface, which blocks autonomous burn-down of queued headless issues across ward, agentic-os, and cli-guard. ## Do * Reproduce from a fresh read-only director surface with `WARD_DISPATCH_BROKER_ADDR` set. * Fix the host broker lifecycle or surface env handoff so the advertised TCP address accepts dispatch requests for the lifetime of the surface. * If the broker is intentionally absent, fail before surfacing a stale/unreachable address and make `warded` explain the missing broker state. * Add coverage for an advertised-but-unreachable broker address if there is a suitable unit seam. ## Acceptance * From a fresh director surface, `warded engineer <headless issue> --harness codex` reaches the host broker and either accepts or policy-rejects the dispatch, but does not fail with TCP connection refused. * `ward agent pr status <repo#pr>` works from the same surface or degrades through the intended non-broker path. * The error message distinguishes broker-not-running from policy rejection and stale reservation.

coilyco-ops added the

labels

2026-07-14 07:42:04 +00:00

coilyco-ops commented

2026-07-14 07:47:32 +00:00

Author

Owner

Additional same-surface evidence: ward agent pr merge coilyco-flight-deck/ward#1241 also fails before eligibility checks with the same broker refusal (host.docker.internal:54293, connect: connection refused). That leaves a green, mergeable PR unable to land from the native director merge lane.

Additional same-surface evidence: `ward agent pr merge coilyco-flight-deck/ward#1241` also fails before eligibility checks with the same broker refusal (`host.docker.internal:54293`, connect: connection refused). That leaves a green, mergeable PR unable to land from the native director merge lane.

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-14 09:06:17 +00:00

Director surface dispatch broker address refuses connections, blocking sibling warded dispatch #1257

coilyco-ops closed this issue

2026-07-14 09:34:56 +00:00

coilyco-ops commented

2026-07-14 09:35:20 +00:00

Author

Owner

WARDED_WORKFLOW: #1257

details

workflow: pull-request-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default merged: yes felt: narrow, then mechanical after the probe-aware listener fix confidence: high surprises: the new TCP probe consumed single-accept test listeners until they were taught to loop follow-ups: none

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1257 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default merged: yes felt: narrow, then mechanical after the probe-aware listener fix confidence: high surprises: the new TCP probe consumed single-accept test listeners until they were taught to loop follow-ups: none </details>

coilyco-ops reopened this issue

2026-07-14 09:55:17 +00:00

coilyco-ops commented

2026-07-14 09:55:23 +00:00

Author

Owner

Reopened from the same 2026-07-14 director surface after PR #1257 reported merged. Fresh sequence: ward agent dispatch-health reported ok for AOS with queued=17 inflight=0 held=2 submitted=0 merge-ready=0 running=0, then warded coilyco-flight-deck/agentic-os#432 --harness codex first failed with read response from host.docker.internal:61320: EOF; a retry immediately failed with dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:61320 ... connect: connection refused. This means the advertised TCP broker can still disappear after a successful health check.

Reopened from the same 2026-07-14 director surface after PR #1257 reported merged. Fresh sequence: `ward agent dispatch-health` reported ok for AOS with `queued=17 inflight=0 held=2 submitted=0 merge-ready=0 running=0`, then `warded coilyco-flight-deck/agentic-os#432 --harness codex` first failed with `read response from host.docker.internal:61320: EOF`; a retry immediately failed with `dispatch broker unavailable: the host dispatch broker did not answer at host.docker.internal:61320 ... connect: connection refused`. This means the advertised TCP broker can still disappear after a successful health check.

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-14 10:37:31 +00:00

Fix brokered dispatch fallback when the host broker is unavailable #1272

coilyco-ops commented

2026-07-14 10:41:48 +00:00

Author

Owner

WARDED_WORKFLOW: #1272

details

workflow: pull-request-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default
felt: narrow, then mechanical
confidence: high
surprises: none
follow-ups: none

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1272 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default felt: narrow, then mechanical confidence: high surprises: none follow-ups: none </details>

coilyco-ops commented

2026-07-14 11:31:15 +00:00

Author

Owner

Director burn-down update from 2026-07-14: PR #1272 is green, but Forgejo reports mergeable=false. ward ops forgejo pr update coilyco-flight-deck ward 1272 returned 409 Conflict with merge failed because of conflict, so this branch needs a narrow conflict repair against current main, then CI rerun and director merge.

Director burn-down update from 2026-07-14: PR #1272 is green, but Forgejo reports mergeable=false. `ward ops forgejo pr update coilyco-flight-deck ward 1272` returned 409 Conflict with `merge failed because of conflict`, so this branch needs a narrow conflict repair against current main, then CI rerun and director merge.

coilyco-ops commented

2026-07-14 11:37:53 +00:00

Author

Owner

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1240 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-14T11:37:52Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1240 · branch issue-1240 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1240 · ward v0.679.0 · dispatched 2026-07-14T11:37:47Z
Comment thread: 5 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-14T07:47:32Z), @coilyco-ops (2026-07-14T09:35:20Z), @coilyco-ops (2026-07-14T09:55:23Z), @coilyco-ops (2026-07-14T10:41:48Z), @coilyco-ops (2026-07-14T11:31:15Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.679.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1240` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-14T11:37:52Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1240` · branch `issue-1240` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1240` · ward `v0.679.0` · dispatched `2026-07-14T11:37:47Z` - **Comment thread:** 5 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-14T07:47:32Z), @coilyco-ops (2026-07-14T09:35:20Z), @coilyco-ops (2026-07-14T09:55:23Z), @coilyco-ops (2026-07-14T10:41:48Z), @coilyco-ops (2026-07-14T11:31:15Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.679.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops closed this issue

2026-07-14 11:48:52 +00:00

coilyco-ops commented

2026-07-14 11:49:22 +00:00

Author

Owner

WARDED_WORKFLOW: #1272

details

workflow: pull-request-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default
merged: yes
felt: narrow, then mechanical after the merge repair
confidence: high
surprises: the merge conflict was confined to the broker surface, but the repo-level hooks still needed a clean staged merge
follow-ups: none

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1272 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default merged: yes felt: narrow, then mechanical after the merge repair confidence: high surprises: the merge conflict was confined to the broker surface, but the repo-level hooks still needed a clean staged merge follow-ups: none </details>

Rows
Columns

Director surface dispatch broker address refuses connections, blocking sibling warded dispatch #1240

Problem

Impact

Do

Acceptance