coilyco-flight-deck/ward

Fork 0

Brokered redispatch masks live worker with Docker name conflict and EOF #788

New issue

Closed

opened 2026-07-09 08:47:47 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-07-09 08:47:47 +00:00

Member

Problem

A director surface retried ward#786 while the original worker container engineer-codex-ward-786 was still running. The fresh host-side launch failed at Docker with a deterministic name conflict, but the director surface saw only a broker EOF or a later broker connection refusal. That reads like a broker crash instead of the actionable launch conflict.

Observed sequence

The first broker dispatch launched engineer-codex-ward-786, and that container stayed running.
That launch did not get durable reservation visibility. The dispatch log says the remote reservation was not posted and the re-check could not read comments because the host-side command path was skewed during the launch.
Later redispatches saw no live local holder, acquired the same local reservation path, and attempted the same deterministic container name.
Docker rejected the retries with: Conflict. The container name "/engineer-codex-ward-786" is already in use.
The failed redispatch rollback released the reservation path for ward#786. Because the local sentinel is per issue, that appears able to erase the only local signal for an already-running older container on the same issue.
The dispatch broker client saw read response ... EOF instead of a structured broker error carrying the Docker conflict and host log path.

Expected behavior

If the deterministic container name already exists and is running, ward agent should refuse before reservation or launch with a clear existing-worker message, even when the local sentinel is absent or stale.
Launch-failure rollback should be ownership-aware. It should release only the sentinel or remote hold acquired by the current launch, never a live older worker for the same issue.
Brokered launch failures should return a structured broker response. The director surface should see the Docker conflict and dispatch log path, not EOF.
Reservation post/read command skew should fail with an actionable host-binary/version message instead of leaving the issue unreserved and continuing as if only cross-host dedupe was degraded.

Evidence

Local dispatch logs under ~/.ward/agent-logs/dispatch/*ward-786.log around 2026-07-09T08:24Z through 2026-07-09T08:35Z show the launch, the missing reservation post, the Docker name conflict, and the broker-side rollback.
Docker showed engineer-codex-ward-786 still Up during diagnosis.
The director transcripts at fail1.txt and fail2.txt show the surface symptoms: broker EOF, then broker connection refused.

Acceptance

Add tests for redispatching the same issue when the deterministic container name is already running but the local sentinel is absent.
Add tests proving launch-failure rollback does not remove a sentinel unless it belongs to the current launch.
Add tests proving brokered Docker launch failures reach the surface as structured errors with the dispatch log path.
The director surface gets a clear conflict message instead of EOF for this class.
ward exec test passes.

Dispatch note

Do not stop or mutate the live ward#786 worker while investigating this issue. It is doing the workflow-semantics repair. This issue is about the dispatch/reservation/broker failure exposed by the retry loop.

## Problem A director surface retried `ward#786` while the original worker container `engineer-codex-ward-786` was still running. The fresh host-side launch failed at Docker with a deterministic name conflict, but the director surface saw only a broker EOF or a later broker connection refusal. That reads like a broker crash instead of the actionable launch conflict. ## Observed sequence * The first broker dispatch launched `engineer-codex-ward-786`, and that container stayed running. * That launch did not get durable reservation visibility. The dispatch log says the remote reservation was not posted and the re-check could not read comments because the host-side command path was skewed during the launch. * Later redispatches saw no live local holder, acquired the same local reservation path, and attempted the same deterministic container name. * Docker rejected the retries with: `Conflict. The container name "/engineer-codex-ward-786" is already in use`. * The failed redispatch rollback released the reservation path for `ward#786`. Because the local sentinel is per issue, that appears able to erase the only local signal for an already-running older container on the same issue. * The dispatch broker client saw `read response ... EOF` instead of a structured broker error carrying the Docker conflict and host log path. ## Expected behavior * If the deterministic container name already exists and is running, `ward agent` should refuse before reservation or launch with a clear existing-worker message, even when the local sentinel is absent or stale. * Launch-failure rollback should be ownership-aware. It should release only the sentinel or remote hold acquired by the current launch, never a live older worker for the same issue. * Brokered launch failures should return a structured broker response. The director surface should see the Docker conflict and dispatch log path, not EOF. * Reservation post/read command skew should fail with an actionable host-binary/version message instead of leaving the issue unreserved and continuing as if only cross-host dedupe was degraded. ## Evidence * Local dispatch logs under `~/.ward/agent-logs/dispatch/*ward-786.log` around `2026-07-09T08:24Z` through `2026-07-09T08:35Z` show the launch, the missing reservation post, the Docker name conflict, and the broker-side rollback. * Docker showed `engineer-codex-ward-786` still `Up` during diagnosis. * The director transcripts at `fail1.txt` and `fail2.txt` show the surface symptoms: broker EOF, then broker connection refused. ## Acceptance * Add tests for redispatching the same issue when the deterministic container name is already running but the local sentinel is absent. * Add tests proving launch-failure rollback does not remove a sentinel unless it belongs to the current launch. * Add tests proving brokered Docker launch failures reach the surface as structured errors with the dispatch log path. * The director surface gets a clear conflict message instead of EOF for this class. * `ward exec test` passes. ## Dispatch note Do not stop or mutate the live `ward#786` worker while investigating this issue. It is doing the workflow-semantics repair. This issue is about the dispatch/reservation/broker failure exposed by the retry loop.

coilyco-ops commented

2026-07-09 08:47:58 +00:00

Author

Member

🔒 Reserved by ward agent --harness codex — container engineer-codex-ward-788 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-09T08:47:56Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#788 · branch issue-788 · harness codex · workflow direct-main
Run: engineer-codex-ward-788 · ward v0.485.0 · dispatched 2026-07-09T08:47:56Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.485.0).

— Codex, via ward agent

🔒 Reserved by `ward agent --harness codex` — container `engineer-codex-ward-788` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-09T08:47:56Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#788` · branch `issue-788` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-ward-788` · ward `v0.485.0` · dispatched `2026-07-09T08:47:56Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.485.0). </details>  — Codex, via `ward agent`

coilyco-ops added the

headless

labels

2026-07-09 18:09:21 +00:00

coilyco-ops commented

2026-07-09 21:53:54 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-788 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T21:53:52Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#788 · branch issue-788 · harness codex · workflow direct-to-main
Run: engineer-codex-ward-788 · ward v0.522.0 · dispatched 2026-07-09T21:53:52Z
Comment thread: 0 included in the pre-flight read, 1 stripped (ward's own automated comments).
- stripped: @coilyco-ops (2026-07-09T08:47:58Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-788` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T21:53:52Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#788` · branch `issue-788` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-ward-788` · ward `v0.522.0` · dispatched `2026-07-09T21:53:52Z` - **Comment thread:** 0 included in the pre-flight read, 1 stripped (ward's own automated comments). - stripped: @coilyco-ops (2026-07-09T08:47:58Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-09 22:23:21 +00:00

fix brokered redispatch conflict handling

coilyco-ops closed this issue

2026-07-09 22:23:21 +00:00

coilyco-ops commented

2026-07-09 22:24:26 +00:00

Author

Member

WARD-OUTCOME: done

details

workflow: direct-to-main; review summary: review gate skipped by ~/.ward/config.yaml default
felt: practical. brokered redispatch now surfaces Docker name conflicts and log paths.
confidence: medium
surprises: repo-local scratch caused hook noise, so I moved caches out before the final merge.
follow-ups: none.
The in-container review gate was intentionally skipped because review gate skipped by ~/.ward/config.yaml default.

WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-to-main; review summary: review gate skipped by ~/.ward/config.yaml default felt: practical. brokered redispatch now surfaces Docker name conflicts and log paths. confidence: medium surprises: repo-local scratch caused hook noise, so I moved caches out before the final merge. follow-ups: none. The in-container review gate was intentionally skipped because review gate skipped by ~/.ward/config.yaml default. </details>