Keep the host dispatch broker alive after brokered dispatch failures

coilyco-ops commented

2026-07-14 07:59:34 +00:00

Member

Problem

The host dispatch broker should not stop serving requests because one brokered dispatch fails.

A per-request launch failure should be returned to the requester, recorded in logs, and contained to that request. It should not close the broker listener or leave a live director surface with stale broker environment variables.

Incident Evidence

During the 2026-07-14 director incident:

The director container stayed alive after the terminal detached.
A brokered engineer launch failed with Docker exit status 125 because the target engineer container name was already in use.
At the same timestamp, local system logs showed the broker listener close.
After that point, commands inside the director saw broker connection failures because the director still had broker configuration but no reachable broker was listening.
No Go panic log, macOS crash report, Docker daemon failure, or director container death was found for that timestamp.

The working hypothesis is that the Docker launch failure escaped the per-request path and unwound or canceled the host broker owner.

Proposed Fix

Harden the broker request boundary so launch failures cannot terminate the broker service.

Specific areas to inspect:

Ensure errors from the brokered dispatch path are serialized into the response and do not cancel the listener context.
Add panic recovery at the per-request handler boundary, with structured logging and an error response.
Separate the lifetime of the host broker from the lifetime of any individual Docker child process.
If the broker truly has to exit, make the director surface detect and report the stale broker state explicitly instead of continuing with a dead broker address.

Tests

Add a regression test or integration fixture where:

A broker is started for a director-like surface.
A brokered launch returns an exit-125-style Docker name-conflict error.
The request receives a classified failure response.
The broker remains reachable.
A second brokered request succeeds or returns its own independent response instead of connection refused.

Acceptance Criteria

A dispatch failure cannot close the broker listener.
A synthetic Docker 125/name-conflict failure is covered by a regression test.
The broker logs the failed request and continues accepting later requests.
A stale broker address inside a live surface is detected and reported with a clear remediation path.

ward#570
ward#1240

## Problem The host dispatch broker should not stop serving requests because one brokered dispatch fails. A per-request launch failure should be returned to the requester, recorded in logs, and contained to that request. It should not close the broker listener or leave a live director surface with stale broker environment variables. ## Incident Evidence During the 2026-07-14 director incident: * The director container stayed alive after the terminal detached. * A brokered engineer launch failed with Docker `exit status 125` because the target engineer container name was already in use. * At the same timestamp, local system logs showed the broker listener close. * After that point, commands inside the director saw broker connection failures because the director still had broker configuration but no reachable broker was listening. * No Go panic log, macOS crash report, Docker daemon failure, or director container death was found for that timestamp. The working hypothesis is that the Docker launch failure escaped the per-request path and unwound or canceled the host broker owner. ## Proposed Fix Harden the broker request boundary so launch failures cannot terminate the broker service. Specific areas to inspect: * Ensure errors from the brokered dispatch path are serialized into the response and do not cancel the listener context. * Add panic recovery at the per-request handler boundary, with structured logging and an error response. * Separate the lifetime of the host broker from the lifetime of any individual Docker child process. * If the broker truly has to exit, make the director surface detect and report the stale broker state explicitly instead of continuing with a dead broker address. ## Tests Add a regression test or integration fixture where: 1. A broker is started for a director-like surface. 2. A brokered launch returns an exit-125-style Docker name-conflict error. 3. The request receives a classified failure response. 4. The broker remains reachable. 5. A second brokered request succeeds or returns its own independent response instead of connection refused. ## Acceptance Criteria * A dispatch failure cannot close the broker listener. * A synthetic Docker 125/name-conflict failure is covered by a regression test. * The broker logs the failed request and continues accepting later requests. * A stale broker address inside a live surface is detected and reported with a clear remediation path. ## Related * ward#570 * ward#1240

coilysiren referenced this issue

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

coilyco-ops added the

headless

P1

labels

2026-07-15 04:51:18 +00:00

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 05:14:51 +00:00

coilyco-ops commented

2026-07-15 05:18:40 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1243 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T05:18:39Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1243 · branch issue-1243 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1243 · ward v0.697.0 · dispatched 2026-07-15T05:18:26Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.697.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1243` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T05:18:39Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1243` · branch `issue-1243` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1243` · ward `v0.697.0` · dispatched `2026-07-15T05:18:26Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.697.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 05:25:16 +00:00

Keep the host dispatch broker alive after brokered dispatch failures #1322

coilyco-ops added the

coherence-core

label

2026-07-15 05:28:20 +00:00

coilyco-ops closed this issue

2026-07-15 05:41:13 +00:00

coilyco-ops commented

2026-07-15 05:41:30 +00:00

Author

Member

WARDED_WORKFLOW: #1322

details

workflow: pull-request-and-merge; review summary: skipped intentionally because the temporary ward default pending brokered QA. Implementation felt narrow after the broker boundary fix, but the repository-wide smart-defaults bundle mismatch had to be synced to get CI green. Confidence: high. Surprise: the baked defaults asset and its canonical source had drifted apart. Follow-up: none.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1322 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped intentionally because the temporary ward default pending brokered QA. Implementation felt narrow after the broker boundary fix, but the repository-wide smart-defaults bundle mismatch had to be synced to get CI green. Confidence: high. Surprise: the baked defaults asset and its canonical source had drifted apart. Follow-up: none. </details>

coilyco-ops commented

2026-07-15 07:16:36 +00:00

Author

Member

WARD-TRIAGE: warded control plane coherence milestone

This issue is part of the warded control plane coherence sprint. The release thesis is to make warded feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run.

For this sprint, headless means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

WARD-TRIAGE: warded control plane coherence milestone This issue is part of the `warded control plane coherence` sprint. The release thesis is to make `warded` feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run. For this sprint, `headless` means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

coilyco-ops commented

2026-07-15 07:16:48 +00:00

Author

Member

WARD-TRIAGE: coherence-core label

This issue is part of the coherence-core review set for the warded control plane coherence sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy.

Core does not mean broader scope. Keep the implementation narrow, verify current main before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

WARD-TRIAGE: coherence-core label This issue is part of the `coherence-core` review set for the `warded control plane coherence` sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy. Core does not mean broader scope. Keep the implementation narrow, verify current `main` before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

coilyco-ops referenced this issue

2026-07-15 10:08:23 +00:00

Clear ghost failed-before-start rows after successful brokered runs #1380

coilyco-ops commented

2026-07-15 10:09:32 +00:00

Author

Member

QA follow-up: PASS for the original broker-failure containment criteria, with a separate operational follow-up filed as #1380.

Evidence reviewed: PR #1322 merged at 4466ee28b2, PR combined status is success, and current main has since passed later promote/release gates. The merged broker code keeps serveHostDispatchBroker accepting after per-connection errors, recovers panics at both the request handler and launch worker boundaries, serializes launch errors back into dispatchBrokerResponse, classifies Docker name-conflict / exit-125-shaped failures as request failures, and reports stale/unreachable broker addresses with broker-unreachable or broker-timeout diagnostics plus the remediation to restart the director surface.

Focused verification on current main passed:
WARD_CONFIG_REF=file://$PWD/.ward/ward-kdl go test ./cmd/ward -run "TestServeHostDispatchBrokerSurvivesFailedLaunch|TestDispatchBrokerPanicErrorClassifiesDockerNameConflict|TestForwardAgentEngineerReportsBrokerUnreachable|TestForwardAgentEngineerReportsBrokerTimeout|TestSendDispatchBrokerRequestReportsStructuredBrokerError" -count=1

While checking the live ops surface, I found ward agent list still renders the successful historical #1243 run as a ghost failed before container start row even though ward agent logs coilyco-flight-deck/ward#1243 shows outcome pushed-to-main. That is not part of the original listener-survival acceptance path, so I filed and dispatched #1380 to fix the stale list/health artifact.

QA follow-up: PASS for the original broker-failure containment criteria, with a separate operational follow-up filed as #1380. Evidence reviewed: PR #1322 merged at 4466ee28b27eb096a50a98257483678b211a7ecf, PR combined status is success, and current main has since passed later promote/release gates. The merged broker code keeps `serveHostDispatchBroker` accepting after per-connection errors, recovers panics at both the request handler and launch worker boundaries, serializes launch errors back into `dispatchBrokerResponse`, classifies Docker name-conflict / exit-125-shaped failures as request failures, and reports stale/unreachable broker addresses with `broker-unreachable` or `broker-timeout` diagnostics plus the remediation to restart the director surface. Focused verification on current main passed: `WARD_CONFIG_REF=file://$PWD/.ward/ward-kdl go test ./cmd/ward -run "TestServeHostDispatchBrokerSurvivesFailedLaunch|TestDispatchBrokerPanicErrorClassifiesDockerNameConflict|TestForwardAgentEngineerReportsBrokerUnreachable|TestForwardAgentEngineerReportsBrokerTimeout|TestSendDispatchBrokerRequestReportsStructuredBrokerError" -count=1` While checking the live ops surface, I found `ward agent list` still renders the successful historical #1243 run as a ghost `failed before container start` row even though `ward agent logs coilyco-flight-deck/ward#1243` shows outcome `pushed-to-main`. That is not part of the original listener-survival acceptance path, so I filed and dispatched #1380 to fix the stale list/health artifact.

Rows
Columns

Keep the host dispatch broker alive after brokered dispatch failures #1243

Problem

Incident Evidence

Proposed Fix

Tests

Acceptance Criteria

Related