Reservation collision silently strands review-driven follow-up dispatches (ward#987 rotted 3 days) #1149

New issue

Closed

opened 2026-07-13 03:27:08 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-13 03:27:08 +00:00

Member

What stranded

coilyco-flight-deck/ward#987 ("feat: add broker logs surface", issue #927) sat open from 2026-07-10 until a stranded-PR sweep found it on 2026-07-13. It is now 180 commits behind main and conflicts in 6 files. The thread on #927 shows exactly how it stranded, and the mechanism is general, not specific to this PR.

The mechanism

The engineer run built the surface, opened PR #987, and reported WARD-OUTCOME: merge-ready.
A blocking review landed: the new log surface did not make an empty or silent or exited log source legible (the fresh repro on the thread, where warded logs ward#943 printed a bare container id and no log lines).
A forwarded redispatch was fired to address that blocking review, roughly 30 seconds after the run reported out.
That redispatch failed on a reservation collision:

WARD-DISPATCH: failed
Failure: ward agent engineer --harness codex: issue coilyco-flight-deck/ward#927
  is already reserved remotely (by @coilyco-ops at 2026-07-10T09:01:57Z);
  wait for it to finish or pass --force to override

Nothing retried. The thread still carries an unhandled  marker three days later, and the PR quietly rotted.

The reservation carries a 1h TTL, and the redispatch arrived while the original run's reservation was still held. So the follow-up that was supposed to fix the blocking review was rejected by the lock the finishing run had not yet released. The failure was reported to the thread and then dropped on the floor.

Why it matters

This is a silent-stranding path, not a one-off. Any review-driven follow-up dispatched promptly after a run reports out can collide with that run's own not-yet-released reservation, fail, and leave the PR with no owner. Nothing sweeps , so the work waits for a human to notice. Here it took a manual sweep three days later, by which point main had moved 180 commits and the branch no longer merges.

Ask

Close the hole, whichever way fits ward's design:

Release the reservation before the run posts its terminal outcome, so an immediate follow-up does not race a lock the run is done with, or
Have a forwarded redispatch that fails on a stale-or-finishing reservation retry or wait, rather than reporting failure once and stopping, and
Give  an owner. A marker nothing ever reads is not a retry mechanism. Either the director heartbeat sweeps it, or the marker should not be written.

Note on #987 and #927 themselves

Do not rebase #987. main independently superseded it: aca1b7e ("feat: add ward agent logs") shipped the broker log surface, and main now also carries streamEmptyDockerAgentLogsSource plus a live transcript-tree fallback, which is precisely the empty-source legibility the blocking review demanded. The shipped ward agent logs help text already advertises the broker path.

So the correct disposition is to close #987 as superseded and re-check #927 against main (it looks satisfied, but confirm the exact repro on the thread before closing it). Closing the PR is currently blocked on agentic-os#488, which adds the missing pr close verb, since ward exposes no way to close a PR today.

## What stranded `coilyco-flight-deck/ward#987` ("feat: add broker logs surface", issue #927) sat open from 2026-07-10 until a stranded-PR sweep found it on 2026-07-13. It is now 180 commits behind `main` and conflicts in 6 files. The thread on #927 shows exactly how it stranded, and the mechanism is general, not specific to this PR. ## The mechanism 1. The engineer run built the surface, opened PR #987, and reported `WARD-OUTCOME: merge-ready`. 2. A blocking review landed: the new log surface did not make an empty or silent or exited log source legible (the fresh repro on the thread, where `warded logs ward#943` printed a bare container id and no log lines). 3. A forwarded redispatch was fired to address that blocking review, roughly 30 seconds after the run reported out. 4. **That redispatch failed on a reservation collision:** ```text WARD-DISPATCH: failed Failure: ward agent engineer --harness codex: issue coilyco-flight-deck/ward#927 is already reserved remotely (by @coilyco-ops at 2026-07-10T09:01:57Z); wait for it to finish or pass --force to override ``` 5. Nothing retried. The thread still carries an unhandled `` marker three days later, and the PR quietly rotted. The reservation carries a 1h TTL, and the redispatch arrived while the original run's reservation was still held. So the follow-up that was supposed to fix the blocking review was rejected by the lock the finishing run had not yet released. The failure was reported to the thread and then dropped on the floor. ## Why it matters This is a silent-stranding path, not a one-off. Any review-driven follow-up dispatched promptly after a run reports out can collide with that run's own not-yet-released reservation, fail, and leave the PR with no owner. Nothing sweeps ``, so the work waits for a human to notice. Here it took a manual sweep three days later, by which point `main` had moved 180 commits and the branch no longer merges. ## Ask Close the hole, whichever way fits ward's design: * Release the reservation before the run posts its terminal outcome, so an immediate follow-up does not race a lock the run is done with, **or** * Have a forwarded redispatch that fails on a *stale-or-finishing* reservation retry or wait, rather than reporting failure once and stopping, **and** * Give `` an owner. A marker nothing ever reads is not a retry mechanism. Either the director heartbeat sweeps it, or the marker should not be written. ## Note on #987 and #927 themselves Do not rebase #987. `main` independently superseded it: `aca1b7e` ("feat: add ward agent logs") shipped the broker log surface, and `main` now also carries `streamEmptyDockerAgentLogsSource` plus a live transcript-tree fallback, which is precisely the empty-source legibility the blocking review demanded. The shipped `ward agent logs` help text already advertises the broker path. So the correct disposition is to **close #987 as superseded** and re-check #927 against `main` (it looks satisfied, but confirm the exact repro on the thread before closing it). Closing the PR is currently blocked on agentic-os#488, which adds the missing `pr close` verb, since ward exposes no way to close a PR today.

coilyco-ops commented

2026-07-13 03:27:24 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-claude-ward-1149 on host KAI-DESKTOP-TOWER.

Reserved by ward agent --harness claude (reserved 2026-07-13T03:27:22Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1149 · branch issue-1149 · harness claude · workflow pull-request-and-merge
Run: engineer-claude-ward-1149 · ward v0.633.0 · dispatched 2026-07-13T03:27:22Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.633.0).

— Claude (she/her), via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-claude-ward-1149` on host `KAI-DESKTOP-TOWER`. Reserved by `ward agent --harness claude` (reserved 2026-07-13T03:27:22Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1149` · branch `issue-1149` · harness `claude` · workflow `pull-request-and-merge` - **Run:** `engineer-claude-ward-1149` · ward `v0.633.0` · dispatched `2026-07-13T03:27:22Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.633.0). </details> </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-13 03:58:10 +00:00

fix(agent): stop reservation collisions stranding review-driven redispatches

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-13 03:58:39 +00:00

fix(agent): stop reservation collisions silently stranding review-driven redispatches #1163

coilyco-ops closed this issue

2026-07-13 03:59:41 +00:00

coilyco-ops commented

2026-07-13 04:01:25 +00:00

Author

Member

WARD-OUTCOME: merge-ready

details

workflow: pull-request-and-merge; review summary: skipped - the in-container review gate was intentionally skipped per the temporary ward default pending brokered QA.
PR: #1163 (branch issue-1149, head d1444cce). CI green (test / test success). The director lane already merged it (merge commit 69febb8d), so the landed state is verified on main.
What landed, three prongs matching the ask: (1) a terminal WARD-OUTCOME now supersedes the reservation at check time in freshReservationComment/reservationClaims (and submitted joined terminalReservationOutcome), so a follow-up dispatched right after a run reports out no longer collides with the finishing run's hold; (2) a forwarded dispatch that still collides with a live hold defers - needs-redispatch marker WITHOUT a release marker, no docker-stop of the issue-scoped container (same-harness that IS the live run), no unlock - and the reaper keeps the terminal release comment off a thread a newer reservation re-holds; (3)  has an owner: the director heartbeat refresh sweeps parked entries (submitted/merge-ready/blocked/failed) whose newest thread signal is an unhandled marker and re-queues them, bounded by redispatchAttemptCap. Docs in docs/agent-reservation.md and docs/agent-director-dispatch.md.
Retro: the codebase had already grown the reaper-side terminal release (which is why the incident felt surprising until traced) - the real holes were the check side not honoring outcomes, the ledger treating merge-ready as untracked-terminal so nothing ever re-queued, and the collision failure comment actively retracting the live run's hold and (same-harness) stopping its container. That last one was an unexpected find. Confidence high on the pure-function changes (all pinned by unit tests, full suite + vet + lint green); moderate on fleet-level emergent behavior of the sweep, since it only runs where a director heartbeat runs. Known accepted residual: a release comment still retracts reservations newer than itself (pre-existing ward#600 semantics); the reap newer-reservation guard narrows that window.
Follow-ups for a human/later run: close PR #987 as superseded once agentic-os#488 lands the pr close verb, and re-check #927 against main (looks satisfied by ward agent logs + streamEmptyDockerAgentLogsSource, but confirm the exact repro on its thread before closing).

WARD-OUTCOME: merge-ready <details><summary>details</summary> - workflow: pull-request-and-merge; review summary: skipped - the in-container review gate was intentionally skipped per the temporary ward default pending brokered QA. - PR: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1163 (branch `issue-1149`, head `d1444cce`). CI green (`test / test` success). The director lane already merged it (merge commit `69febb8d`), so the landed state is verified on `main`. - What landed, three prongs matching the ask: (1) a terminal `WARD-OUTCOME` now supersedes the reservation at check time in `freshReservationComment`/`reservationClaims` (and `submitted` joined `terminalReservationOutcome`), so a follow-up dispatched right after a run reports out no longer collides with the finishing run's hold; (2) a forwarded dispatch that still collides with a live hold defers - needs-redispatch marker WITHOUT a release marker, no docker-stop of the issue-scoped container (same-harness that IS the live run), no unlock - and the reaper keeps the terminal release comment off a thread a newer reservation re-holds; (3) `` has an owner: the director heartbeat refresh sweeps parked entries (submitted/merge-ready/blocked/failed) whose newest thread signal is an unhandled marker and re-queues them, bounded by `redispatchAttemptCap`. Docs in `docs/agent-reservation.md` and `docs/agent-director-dispatch.md`. - Retro: the codebase had already grown the reaper-side terminal release (which is why the incident felt surprising until traced) - the real holes were the check side not honoring outcomes, the ledger treating merge-ready as untracked-terminal so nothing ever re-queued, and the collision failure comment actively retracting the live run's hold and (same-harness) stopping its container. That last one was an unexpected find. Confidence high on the pure-function changes (all pinned by unit tests, full suite + vet + lint green); moderate on fleet-level emergent behavior of the sweep, since it only runs where a director heartbeat runs. Known accepted residual: a release comment still retracts reservations newer than itself (pre-existing ward#600 semantics); the reap newer-reservation guard narrows that window. - Follow-ups for a human/later run: close PR #987 as superseded once agentic-os#488 lands the `pr close` verb, and re-check #927 against `main` (looks satisfied by `ward agent logs` + `streamEmptyDockerAgentLogsSource`, but confirm the exact repro on its thread before closing). </details>