Fail fast on engineer-pool capacity contention at the dispatch broker, before touching the issue tracker #1041

New issue

Closed

opened 2026-07-10 17:11:21 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-10 17:11:21 +00:00

Member

Problem

When the engineer pool is at capacity (e.g. warded list => 12/12, at capacity), a fresh dispatch discovers the contention late - far enough down the dispatch flow that it has already touched the issue tracker. The condition surfaces as a WARD-DISPATCH: failed / reservation / "queue full" comment on the target issue, when the actual state ("no local engineer slot right now") is a transient, local, host-side concern.

Pool/queue fullness is not a fact about the issue - it is a fact about this host's dispatcher at this instant. It should never leave a mark on the shared issue thread, which is a durable, human-and-agent-read surface. Posting there is noise: it re-notifies, it interleaves with real run history (the reservation and outcome comments), and it makes a purely-local backpressure event read as though something happened to the work.

Observed

warded list reports 12/12, at capacity during director burndown.
Dispatch attempts that lose a capacity race still emit WARD-DISPATCH: failed comments onto the target issue (seen on agentic-os#438, ward#996).
The failure text is about local worker containers ("already has a running worker container ... wait for it to finish"), yet it lands on the remote tracker.

Do

Move the capacity check to the front of the dispatch broker, ahead of any remote side effect: before reserving the issue, before posting any comment, before container creation. If no local engineer slot is free, fail fast right there.
On a capacity-full result, surface it locally only - a non-zero exit / stderr line to the operator or director heartbeat (engineer pool full, N/N, not dispatched) - and let the caller (heartbeat / lane) retry when a slot frees. Do not reserve, do not comment on the issue.
Preserve the distinction: a genuine remote reservation conflict (another run is actually working the issue) is a real, issue-relevant state and may still be recorded per existing behavior. Capacity backpressure is not, and stays invisible to the tracker.

Acceptance

Dispatching against a full engineer pool leaves zero new comments or edits on the target issue.
The full-pool condition is reported to the operator / heartbeat locally with the current N/N count, and the issue stays dispatchable (no reservation held) once a slot frees.
The remote-reservation-conflict path is unchanged.

ward#1034 - duplicate workers / reservation-vs-inventory disagreement. Adjacent but distinct: that issue is about correctness of the reservation check; this one is about where in the flow capacity backpressure is detected and that it stays local.
"Stop director dispatch from exceeding the global engineer limit" - the pool-limit enforcement this ordering change rides on.

## Problem When the engineer pool is at capacity (e.g. `warded list` => `12/12, at capacity`), a fresh dispatch discovers the contention **late** - far enough down the dispatch flow that it has already touched the **issue tracker**. The condition surfaces as a `WARD-DISPATCH: failed` / reservation / "queue full" comment on the target issue, when the actual state ("no local engineer slot right now") is a **transient, local, host-side** concern. Pool/queue fullness is not a fact about the issue - it is a fact about this host's dispatcher at this instant. It should never leave a mark on the shared issue thread, which is a durable, human-and-agent-read surface. Posting there is noise: it re-notifies, it interleaves with real run history (the reservation and outcome comments), and it makes a purely-local backpressure event read as though something happened to the work. ## Observed * `warded list` reports `12/12, at capacity` during director burndown. * Dispatch attempts that lose a capacity race still emit `WARD-DISPATCH: failed` comments onto the target issue (seen on `agentic-os#438`, `ward#996`). * The failure text is about local worker containers ("already has a running worker container ... wait for it to finish"), yet it lands on the remote tracker. ## Do * Move the **capacity check to the front of the dispatch broker**, ahead of any remote side effect: before reserving the issue, before posting any comment, before container creation. If no local engineer slot is free, fail fast right there. * On a capacity-full result, surface it **locally only** - a non-zero exit / stderr line to the operator or director heartbeat (`engineer pool full, N/N, not dispatched`) - and let the caller (heartbeat / lane) retry when a slot frees. Do **not** reserve, do **not** comment on the issue. * Preserve the distinction: a genuine **remote reservation conflict** (another run is actually working the issue) is a real, issue-relevant state and may still be recorded per existing behavior. Capacity backpressure is not, and stays invisible to the tracker. ## Acceptance * Dispatching against a full engineer pool leaves **zero** new comments or edits on the target issue. * The full-pool condition is reported to the operator / heartbeat locally with the current N/N count, and the issue stays dispatchable (no reservation held) once a slot frees. * The remote-reservation-conflict path is unchanged. ## Related * `ward#1034` - duplicate workers / reservation-vs-inventory disagreement. Adjacent but distinct: that issue is about correctness of the reservation check; this one is about **where in the flow** capacity backpressure is detected and that it stays local. * "Stop director dispatch from exceeding the global engineer limit" - the pool-limit enforcement this ordering change rides on.

coilyco-ops referenced this issue

2026-07-10 17:17:13 +00:00

Add --override-capacity opt-in past the OOM pool ceiling; rename --force to --override-reservation (an --override-* flag family) #1045

coilyco-ops added the

headless

labels

2026-07-14 06:50:15 +00:00

coilyco-ops commented

2026-07-14 07:00:12 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1041 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-14T07:00:11Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1041 · branch issue-1041 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1041 · ward v0.667.0 · dispatched 2026-07-14T07:00:04Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.667.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1041` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-14T07:00:11Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1041` · branch `issue-1041` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1041` · ward `v0.667.0` · dispatched `2026-07-14T07:00:04Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.667.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-14 07:05:59 +00:00

Fail fast on engineer-pool capacity contention at the dispatch broker, before touching the issue tracker #1233

coilyco-ops closed this issue

2026-07-14 07:07:51 +00:00

coilyco-ops commented

2026-07-14 07:08:26 +00:00

Author

Member

WARDED_WORKFLOW: #1233

details

workflow: pull-request-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default

Implementation felt straightforward once the broker capacity branch stopped flowing into issue comments. Confidence is high. Surprise was limited to the PR checks already being green before merge.

follow-ups: none.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1233 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default Implementation felt straightforward once the broker capacity branch stopped flowing into issue comments. Confidence is high. Surprise was limited to the PR checks already being green before merge. follow-ups: none. </details>