coilyco-flight-deck/ward

Fork 0

Reproduce the director broker exit after a Docker name-conflict 125 #1244

New issue

Closed

opened 2026-07-14 07:59:35 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-14 07:59:35 +00:00

Member

Problem

The 2026-07-14 director incident needs a focused repro and forensic fix so the underlying failure path is proven, not just inferred.

Evidence Gathered

The director surface stayed alive after the terminal detached.
Docker events for the director around the incident window did not show a container death.
Later brokered commands from inside the director failed because the advertised broker address no longer had a live listener.
The audit log recorded a brokered launch-path failure at 2026-07-14 00:36:44 PDT with exit status 125 and no useful stderr tail.
The host wrapper log for the same brokered engineer launch recorded Docker refusing the launch because the target engineer container name was already in use.
Local system logs showed the broker listener close at the same timestamp as the exit-125 brokered launch failure.
No Go panic log, OS crash report, Docker daemon crash, or director container death was found for that timestamp.

Current Hypothesis

A brokered Docker name-conflict failure propagated beyond the individual dispatch request and caused the host broker listener to close. The director container then kept running with stale broker configuration, so later commands saw broker connection failures rather than the original Docker launch failure.

There is also a forensic gap: the audit row for the brokered launch path appears to preserve the parent/director argv rather than the brokered request shape, which makes the failed action harder to identify from audit alone.

Repro Plan

Start a director surface with a host dispatch broker.
Create or preserve a conflicting engineer container name that a brokered launch will try to reuse.
Trigger a brokered engineer launch through the director.
Confirm the Docker 125/name-conflict error is returned and logged.
Confirm whether the broker listener remains reachable after the failed request.
Confirm whether audit metadata records the brokered target role/ref separately from the parent director argv.

Fix Direction

This likely splits into two implementation fixes:

Keep the broker alive after dispatch failures. See ward#1243.
Persist broker-dispatch logs in the normal agent-log tree so the first failure reason is visible. See ward#1242.

Also consider improving audit fields for brokered launches so the audit stream records parent session, broker request, target role, target ref, and classified launch error separately.

Acceptance Criteria

There is a repeatable local repro for the Docker 125/name-conflict broker path.
The repro demonstrates the pre-fix broker-lifetime behavior or proves a different root cause.
The final fix includes either a regression test or an explicit documented reason an integration test is not practical.
Audit/log output after the fix is enough to explain the failure without direct manual inspection of unrelated host logs.

ward#1240
ward#1242
ward#1243

## Problem The 2026-07-14 director incident needs a focused repro and forensic fix so the underlying failure path is proven, not just inferred. ## Evidence Gathered * The director surface stayed alive after the terminal detached. * Docker events for the director around the incident window did not show a container death. * Later brokered commands from inside the director failed because the advertised broker address no longer had a live listener. * The audit log recorded a brokered launch-path failure at `2026-07-14 00:36:44 PDT` with `exit status 125` and no useful stderr tail. * The host wrapper log for the same brokered engineer launch recorded Docker refusing the launch because the target engineer container name was already in use. * Local system logs showed the broker listener close at the same timestamp as the exit-125 brokered launch failure. * No Go panic log, OS crash report, Docker daemon crash, or director container death was found for that timestamp. ## Current Hypothesis A brokered Docker name-conflict failure propagated beyond the individual dispatch request and caused the host broker listener to close. The director container then kept running with stale broker configuration, so later commands saw broker connection failures rather than the original Docker launch failure. There is also a forensic gap: the audit row for the brokered launch path appears to preserve the parent/director argv rather than the brokered request shape, which makes the failed action harder to identify from audit alone. ## Repro Plan * Start a director surface with a host dispatch broker. * Create or preserve a conflicting engineer container name that a brokered launch will try to reuse. * Trigger a brokered engineer launch through the director. * Confirm the Docker 125/name-conflict error is returned and logged. * Confirm whether the broker listener remains reachable after the failed request. * Confirm whether audit metadata records the brokered target role/ref separately from the parent director argv. ## Fix Direction This likely splits into two implementation fixes: * Keep the broker alive after dispatch failures. See ward#1243. * Persist broker-dispatch logs in the normal agent-log tree so the first failure reason is visible. See ward#1242. Also consider improving audit fields for brokered launches so the audit stream records parent session, broker request, target role, target ref, and classified launch error separately. ## Acceptance Criteria * There is a repeatable local repro for the Docker 125/name-conflict broker path. * The repro demonstrates the pre-fix broker-lifetime behavior or proves a different root cause. * The final fix includes either a regression test or an explicit documented reason an integration test is not practical. * Audit/log output after the fix is enough to explain the failure without direct manual inspection of unrelated host logs. ## Related * ward#1240 * ward#1242 * ward#1243

coilysiren referenced this issue

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

coilyco-ops added the

headless

labels

2026-07-15 04:51:17 +00:00

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 05:14:51 +00:00

coilyco-ops commented

2026-07-15 06:55:00 +00:00

Author

Member

Release triage note from Kai on 2026-07-15: headless carry is valid if the reproduction is synthetic. The engineer should simulate Docker exit 125/name-conflict behavior in a fixture or test and prove the broker/request lifetime behavior without requiring a live host repro.

coilyco-ops commented

2026-07-15 07:16:36 +00:00

Author

Member

WARD-TRIAGE: warded control plane coherence milestone

This issue is part of the warded control plane coherence sprint. The release thesis is to make warded feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run.

For this sprint, headless means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

WARD-TRIAGE: warded control plane coherence milestone This issue is part of the `warded control plane coherence` sprint. The release thesis is to make `warded` feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run. For this sprint, `headless` means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

coilyco-ops commented

2026-07-15 10:27:06 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1244 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T10:27:05Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1244 · branch issue-1244 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1244 · ward v0.710.0 · dispatched 2026-07-15T10:27:00Z
Comment thread: 2 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-15T06:55:00Z), @coilyco-ops (2026-07-15T07:16:36Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1244` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T10:27:05Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1244` · branch `issue-1244` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1244` · ward `v0.710.0` · dispatched `2026-07-15T10:27:00Z` - **Comment thread:** 2 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-15T06:55:00Z), @coilyco-ops (2026-07-15T07:16:36Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 10:32:00 +00:00

Reproduce the director broker exit after a Docker name-conflict 125 #1385

coilyco-ops closed this issue

2026-07-15 10:34:22 +00:00

coilyco-ops commented

2026-07-15 10:39:23 +00:00

Author

Member

QA follow-up: PASS. PR #1385 merged at 959ee02122. PR test run 8464 succeeded. Its promote run 8466 was cancelled by the next main push from #1386, and the later #1386 promote/release path included #1385 on main and succeeded (promote 8467, release 8469).

Evidence checked:

The synthetic repro now uses the actual Docker duplicate-name shape: exit status 125: Conflict. The container name ... is already in use.
TestDispatchBrokerPanicErrorClassifiesExit125NameConflictAsRequestFailure verifies the exit-125/name-conflict error is classified as a request failure and preserves the useful Docker details.
TestServeHostDispatchBrokerSurvivesExit125NameConflict proves the broker listener remains usable after the failed launch by sending a second request through the same listener.
Focused local verification passed: go test ./cmd/ward -run 'TestDispatchBrokerPanicErrorClassifiesExit125NameConflictAsRequestFailure|TestServeHostDispatchBrokerSurvivesExit125NameConflict' -count=1.\n\nThe issue criteria are met; no follow-up issue needed.

QA follow-up: PASS. PR #1385 merged at 959ee02122aa7502c69867204c69fb925dece982. PR test run 8464 succeeded. Its promote run 8466 was cancelled by the next main push from #1386, and the later #1386 promote/release path included #1385 on main and succeeded (promote 8467, release 8469). Evidence checked: - The synthetic repro now uses the actual Docker duplicate-name shape: `exit status 125: Conflict. The container name ... is already in use`. - `TestDispatchBrokerPanicErrorClassifiesExit125NameConflictAsRequestFailure` verifies the exit-125/name-conflict error is classified as a request failure and preserves the useful Docker details. - `TestServeHostDispatchBrokerSurvivesExit125NameConflict` proves the broker listener remains usable after the failed launch by sending a second request through the same listener. - Focused local verification passed: `go test ./cmd/ward -run 'TestDispatchBrokerPanicErrorClassifiesExit125NameConflictAsRequestFailure|TestServeHostDispatchBrokerSurvivesExit125NameConflict' -count=1`.\n\nThe issue criteria are met; no follow-up issue needed.