ward agent: stagger/serialize dispatch so concurrent runs don't race onto one issue (jittered reservation re-check + broker per-ref lock + closed-issue skip) #600

New issue

Closed

opened 2026-07-04 08:37:51 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-04 08:37:51 +00:00

Owner

Problem

Multiple ward agent runs can be dispatched near-simultaneously for the same issue and all spin containers, because the reservation check and reservation post are not atomic. Each dispatcher reads "is this reserved?", sees nothing, posts its own reservation, and proceeds - so N near-simultaneous dispatches yield N containers on one issue: duplicate work, merge races, wasted spins.

Evidence

coilyco-flight-deck/ward#507 collected four reservation comments inside 17 seconds (2026-07-04T08:28:20, :26, :31, :37), and the carrying engineer's retro said it plainly: "three back-to-back reservation comments today suggest the container restarted a few times against an issue that was already closed ... a re-dispatch guard against an already-closed/already-landed issue could save a container spin."

Ask (Kai)

Engineers should wait a few seconds (10? 30? 60?) before claiming, to make sure multiple are not getting dispatched at once.

Proposed design - double-checked reservation with jitter

A flat sleep is not enough: if several dispatches fire on the same tick, a fixed 10/30/60s wait just makes them all collide 10/30/60s later. Break the symmetry and re-check:

On dispatch, read existing reservation comments for the ref. If a live (non-stale) reservation exists, yield (do not start) unless --force.
If none, post this run's reservation, then wait a jittered short interval (randomized per run, e.g. 5-15s - jitter matters more than the absolute value), then re-read reservations.
If another reservation now exists that wins a deterministic tiebreak (earliest timestamp, else lexically-min container id), this run yields and tears down before doing any work. Only the single winner proceeds.

Expose the wait/jitter window as a flag/env with a sane default.

Cleaner primary fix - a broker-side per-ref lock

Dispatch now flows through the host dispatch broker. If the broker holds a short per-issue-ref lock/mutex, two warded #N for the same N serialize at the broker before any container starts - killing the race at the source with zero wasted spin, instead of tearing down an already-spun container. Recommend the broker lock as the primary fix where feasible, with the jittered double-check (above) as the in-container backstop for any dispatch path that bypasses the broker.

Companion guard - skip an already-closed / already-landed issue

Flagged by the #507 engineer in the same retro. Before starting work, check issue state: if it is closed (or its closes #N is already an ancestor of main), no-op and tear down instead of spinning a container to rediscover "already done." #507 burned ~3 spins on exactly this. Shares the dispatch pre-flight code path with the race guard, so implement together.

Acceptance

Two near-simultaneous warded #N for the same N result in exactly one container doing work; the others yield cleanly (ideally before container start via the broker lock).
A dispatch at an already-closed / already-landed issue no-ops without a full engineer spin.
The single-dispatch happy path is unchanged (no added latency beyond the small jitter window, and only when a collision is possible).

Refs

ward#507 comment (the four back-to-back reservations + the re-dispatch-guard observation).
ward#494 (conversation-lock / reservation work - related surface).

Provenance: director surface session, 2026-07-04.

## Problem Multiple `ward agent` runs can be dispatched near-simultaneously for the **same issue** and all spin containers, because the reservation **check** and reservation **post** are not atomic. Each dispatcher reads "is this reserved?", sees nothing, posts its own reservation, and proceeds - so N near-simultaneous dispatches yield N containers on one issue: duplicate work, merge races, wasted spins. ## Evidence `coilyco-flight-deck/ward#507` collected **four** reservation comments inside 17 seconds (2026-07-04T08:28:20, :26, :31, :37), and the carrying engineer's retro said it plainly: *"three back-to-back reservation comments today suggest the container restarted a few times against an issue that was already closed ... a re-dispatch guard against an already-closed/already-landed issue could save a container spin."* ## Ask (Kai) Engineers should wait a few seconds (10? 30? 60?) before claiming, to make sure multiple are not getting dispatched at once. ## Proposed design - double-checked reservation with jitter A flat sleep is not enough: if several dispatches fire on the same tick, a fixed 10/30/60s wait just makes them all collide 10/30/60s later. Break the symmetry and re-check: 1. On dispatch, read existing reservation comments for the ref. If a **live** (non-stale) reservation exists, yield (do not start) unless `--force`. 2. If none, post this run's reservation, then wait a **jittered** short interval (randomized per run, e.g. 5-15s - jitter matters more than the absolute value), then **re-read** reservations. 3. If another reservation now exists that wins a deterministic tiebreak (earliest timestamp, else lexically-min container id), **this run yields and tears down** before doing any work. Only the single winner proceeds. Expose the wait/jitter window as a flag/env with a sane default. ## Cleaner primary fix - a broker-side per-ref lock Dispatch now flows through the **host dispatch broker**. If the broker holds a short per-issue-ref lock/mutex, two `warded #N` for the same `N` serialize **at the broker before any container starts** - killing the race at the source with zero wasted spin, instead of tearing down an already-spun container. Recommend the broker lock as the primary fix where feasible, with the jittered double-check (above) as the in-container backstop for any dispatch path that bypasses the broker. ## Companion guard - skip an already-closed / already-landed issue Flagged by the #507 engineer in the same retro. Before starting work, check issue state: if it is **closed** (or its `closes #N` is already an ancestor of `main`), no-op and tear down instead of spinning a container to rediscover "already done." #507 burned ~3 spins on exactly this. Shares the dispatch pre-flight code path with the race guard, so implement together. ## Acceptance - Two near-simultaneous `warded #N` for the same `N` result in exactly **one** container doing work; the others yield cleanly (ideally before container start via the broker lock). - A dispatch at an already-closed / already-landed issue no-ops without a full engineer spin. - The single-dispatch happy path is unchanged (no added latency beyond the small jitter window, and only when a collision is possible). ## Refs - ward#507 comment (the four back-to-back reservations + the re-dispatch-guard observation). - ward#494 (conversation-lock / reservation work - related surface). Provenance: director surface session, 2026-07-04.

coilyco-ops commented

2026-07-04 08:38:04 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-600 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T08:37:58Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-600` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T08:37:58Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-04 09:00:05 +00:00

feat(agent): stagger concurrent dispatch so runs don't race onto one issue

coilysiren closed this issue

2026-07-04 09:00:06 +00:00

coilysiren commented

2026-07-04 09:00:48 +00:00

Author

Owner

WARD-OUTCOME: done - broker per-ref lock + jittered reservation re-check + closed-issue skip landed on main

Honest retro: this one was mostly about reading before touching. The three
asks map onto three different seams (the broker, the reservation acquire path,
and the resolve-work pre-flight), so the front-loaded read of agent.md /
agent-preflight.md / agent-reservation.md paid off - I could place each guard
where it belonged instead of bolting on a fourth mechanism.

Two things fought back a little. First, the container name is deterministic per
issue, so "lexically-min container id" alone doesn't break a cross-host tie - I
widened the tiebreak to container@host to keep it a total order. Second, and
more interesting: the obvious "loser posts a release marker to retract itself"
is a trap, because the release semantics aren't per-run - a release stamped at/
after the latest reservation frees the winner too. So the loser just yields
and lets its comment lapse at the TTL. Correct, but it leaves one extra comment
on the rare non-broker race path.

Confidence: high on the pure logic (tiebreak, claim parsing, env parsing, closed
skip - all unit-tested) and the broker lock. Medium on the end-to-end race
timing, since I couldn't exercise two real concurrent dispatches with Docker +
Forgejo here; the design leans on the jitter minimum exceeding comment
propagation, which unit tests can't prove.

Follow-ups worth filing: (1) main landed an issue-comment delete leaf
(d2d6489) while I was working - wiring deleteIssueComment into the issueForge
interface would let the loser cleanly retract its own comment instead of leaving
it to TTL, killing the last bit of thread noise. (2) The "closes #N already an
ancestor of main" half of the closed-issue guard is unimplemented - it needs a
clone/fetch not available at host pre-flight, so I shipped only the closed-state
check (which is what burned the #507 spins). (3) The jitter adds 5-15s to every
detached dispatch, director loop included; WARD_AGENT_RESERVE_RECHECK=0 disables
it, but a smarter "only wait when contention was seen at post time" could reclaim
that latency without losing correctness.

WARD-OUTCOME: done - broker per-ref lock + jittered reservation re-check + closed-issue skip landed on main Honest retro: this one was mostly about reading before touching. The three asks map onto three different seams (the broker, the reservation acquire path, and the resolve-work pre-flight), so the front-loaded read of agent.md / agent-preflight.md / agent-reservation.md paid off - I could place each guard where it belonged instead of bolting on a fourth mechanism. Two things fought back a little. First, the container name is deterministic per issue, so "lexically-min container id" alone doesn't break a cross-host tie - I widened the tiebreak to `container@host` to keep it a total order. Second, and more interesting: the obvious "loser posts a release marker to retract itself" is a trap, because the release semantics aren't per-run - a release stamped at/ after the latest reservation frees the *winner* too. So the loser just yields and lets its comment lapse at the TTL. Correct, but it leaves one extra comment on the rare non-broker race path. Confidence: high on the pure logic (tiebreak, claim parsing, env parsing, closed skip - all unit-tested) and the broker lock. Medium on the end-to-end race timing, since I couldn't exercise two real concurrent dispatches with Docker + Forgejo here; the design leans on the jitter minimum exceeding comment propagation, which unit tests can't prove. Follow-ups worth filing: (1) main landed an issue-comment *delete* leaf (d2d6489) while I was working - wiring `deleteIssueComment` into the issueForge interface would let the loser cleanly retract its own comment instead of leaving it to TTL, killing the last bit of thread noise. (2) The "closes #N already an ancestor of main" half of the closed-issue guard is unimplemented - it needs a clone/fetch not available at host pre-flight, so I shipped only the closed-state check (which is what burned the #507 spins). (3) The jitter adds 5-15s to every detached dispatch, director loop included; WARD_AGENT_RESERVE_RECHECK=0 disables it, but a smarter "only wait when contention was seen at post time" could reclaim that latency without losing correctness.