coilyco-flight-deck/ward

Fork 0

Prevent duplicate workers for the same reserved issue across harnesses #1034

New issue

Closed

opened 2026-07-10 17:02:31 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-10 17:02:31 +00:00

Member

Problem

The director burndown surfaced a duplicate-worker state for a single reserved issue. ward#996 was dispatched by this director with Codex, while another director attempted Claude around the same time. The issue thread says the Claude attempt failed because the issue was already reserved, but warded list still shows both containers running for the same issue.

Live evidence from 2026-07-10 UTC:

warded list
ward agent: running engineer containers (12/12, at capacity)

  engineer-claude-ward-996
    ref: coilyco-flight-deck/ward#996
    harness: claude
    issue: 996
    status: running

  engineer-codex-ward-996
    ref: coilyco-flight-deck/ward#996
    harness: codex
    issue: 996
    status: running

But the issue thread also reports:

WARD-DISPATCH: failed
Attempted harness: claude
Failure: issue coilyco-flight-deck/ward#996 is already reserved remotely ... wait for it to finish or pass --force to override

That means the remote reservation path and running-container inventory can disagree, and the global pool can spend two slots on one issue while believing one attempt failed. This is especially bad during director burndown because it reduces available capacity and lets two engineers race on the same branch/issue.

Do

Trace how a worker container can remain running or become visible in warded list after dispatch reports remote-reservation failure.
Ensure the reservation check is authoritative before container creation across harnesses and concurrent directors.
If a container is created and then the reservation fails, stop it and make the failed state visible without counting it as a running engineer.
Make warded list and issue comments agree on whether a run is active, failed, or rejected.
Add a regression test or broker fixture for two near-simultaneous dispatches of the same issue with different harnesses.

Acceptance

Two concurrent dispatches for the same issue cannot both leave running engineer containers unless --force intentionally transfers/reclaims the reservation.
A remote-reservation failure leaves no running worker container for that failed attempt.
warded list does not show duplicate active workers for the same issue after one attempt was rejected.
The issue thread outcome and list inventory agree on the active worker identity.

Filed from the read-only director surface (she/her).

## Problem The director burndown surfaced a duplicate-worker state for a single reserved issue. `ward#996` was dispatched by this director with Codex, while another director attempted Claude around the same time. The issue thread says the Claude attempt failed because the issue was already reserved, but `warded list` still shows both containers running for the same issue. Live evidence from 2026-07-10 UTC: ```text warded list ward agent: running engineer containers (12/12, at capacity) engineer-claude-ward-996 ref: coilyco-flight-deck/ward#996 harness: claude issue: 996 status: running engineer-codex-ward-996 ref: coilyco-flight-deck/ward#996 harness: codex issue: 996 status: running ``` But the issue thread also reports: ```text WARD-DISPATCH: failed Attempted harness: claude Failure: issue coilyco-flight-deck/ward#996 is already reserved remotely ... wait for it to finish or pass --force to override ``` That means the remote reservation path and running-container inventory can disagree, and the global pool can spend two slots on one issue while believing one attempt failed. This is especially bad during director burndown because it reduces available capacity and lets two engineers race on the same branch/issue. ## Do * Trace how a worker container can remain running or become visible in `warded list` after dispatch reports remote-reservation failure. * Ensure the reservation check is authoritative before container creation across harnesses and concurrent directors. * If a container is created and then the reservation fails, stop it and make the failed state visible without counting it as a running engineer. * Make `warded list` and issue comments agree on whether a run is active, failed, or rejected. * Add a regression test or broker fixture for two near-simultaneous dispatches of the same issue with different harnesses. ## Acceptance * Two concurrent dispatches for the same issue cannot both leave running engineer containers unless `--force` intentionally transfers/reclaims the reservation. * A remote-reservation failure leaves no running worker container for that failed attempt. * `warded list` does not show duplicate active workers for the same issue after one attempt was rejected. * The issue thread outcome and list inventory agree on the active worker identity. Filed from the read-only director surface (she/her).

coilyco-ops added the

headless

labels

2026-07-10 17:02:31 +00:00

coilyco-ops commented

2026-07-10 17:02:46 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1034 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T17:02:45Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1034 · branch issue-1034 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1034 · ward v0.584.0 · dispatched 2026-07-10T17:02:45Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1034` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T17:02:45Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1034` · branch `issue-1034` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1034` · ward `v0.584.0` · dispatched `2026-07-10T17:02:45Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 17:16:10 +00:00

Prevent duplicate workers for the same reserved issue #1044

coilyco-ops commented

2026-07-10 17:17:15 +00:00

Author

Member

WARD-OUTCOME: merge-ready

details

workflow: pull-requests-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default
felt: the race was in launch serialization, not the issue text.
confidence: high
surprises: the repo-wide hook surfaced a couple of comment-hygiene files, so those got trimmed too.
follow-ups: PR #1044 is green and mergeable.

WARD-OUTCOME: merge-ready <details><summary>details</summary> workflow: pull-requests-and-merge; review summary: skipped because review gate skipped by ~/.ward/config.yaml default felt: the race was in launch serialization, not the issue text. confidence: high surprises: the repo-wide hook surfaced a couple of comment-hygiene files, so those got trimmed too. follow-ups: PR #1044 is green and mergeable. </details>

coilyco-ops commented

2026-07-10 18:15:29 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1034 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T18:15:28Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1034 · branch issue-1034 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1034 · ward v0.584.0 · dispatched 2026-07-10T18:15:28Z
Comment thread: 1 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T17:17:15Z)
- stripped: @coilyco-ops (2026-07-10T17:02:46Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1034` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T18:15:28Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1034` · branch `issue-1034` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1034` · ward `v0.584.0` · dispatched `2026-07-10T18:15:28Z` - **Comment thread:** 1 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T17:17:15Z) - stripped: @coilyco-ops (2026-07-10T17:02:46Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0). </details> </details>  — Codex, via `ward agent`