coilyco-flight-deck/ward

Fork 0

Treat global engineer concurrency cap as deferred instead of dispatch failure #902

New issue

Closed

opened 2026-07-10 00:47:52 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-07-10 00:47:52 +00:00

Member

Problem

A director-surface dispatch for #900 hit the global engineer cap and posted a failure comment:

WARD-DISPATCH: failed
Failure: `ward agent engineer --harness codex: global engineer limit is reached: 10 running (limit 10); wait for a run to finish or run `ward agent reap` for stale engineers`

Being at the configured concurrency cap is not an implementation failure. It is normal backpressure. The director should idle/retry until a slot opens, bounded by its normal timeout/backoff policy, rather than marking the issue as failed.

Expected

Global engineer cap reached should classify as deferred/capacity-waiting, not failed.
The issue should not receive a WARD-DISPATCH: failed comment solely because the cap is full.
Director heartbeat should keep the item queued or waiting and retry on later cycles after runs finish.
The user-facing message should name the cap and the retry behavior, not tell the reader the dispatch failed.

Do

Find where the global engineer cap error is produced and where director/broker dispatch classifies it.
Add a typed/sentinel error or structured classification for capacity cap reached if one does not exist.
Teach director-surface broker forwarding and normal director dispatch to treat that classification as deferred.
Preserve genuine failures for auth, image, docker, malformed ref, untrusted owner, or other non-capacity launch errors.

Acceptance

A test covers global cap reached during director dispatch and asserts the ledger/comment state is queued/deferred, not failed.
A test covers a non-capacity dispatch failure still producing failure state.
ward exec test passes, or the issue comment names the exact failing check.

Closes this issue when merged.

## Problem A director-surface dispatch for #900 hit the global engineer cap and posted a failure comment: ```text WARD-DISPATCH: failed Failure: `ward agent engineer --harness codex: global engineer limit is reached: 10 running (limit 10); wait for a run to finish or run `ward agent reap` for stale engineers` ``` Being at the configured concurrency cap is not an implementation failure. It is normal backpressure. The director should idle/retry until a slot opens, bounded by its normal timeout/backoff policy, rather than marking the issue as failed. ## Expected - Global engineer cap reached should classify as deferred/capacity-waiting, not failed. - The issue should not receive a `WARD-DISPATCH: failed` comment solely because the cap is full. - Director heartbeat should keep the item queued or waiting and retry on later cycles after runs finish. - The user-facing message should name the cap and the retry behavior, not tell the reader the dispatch failed. ## Do - Find where the global engineer cap error is produced and where director/broker dispatch classifies it. - Add a typed/sentinel error or structured classification for capacity cap reached if one does not exist. - Teach director-surface broker forwarding and normal director dispatch to treat that classification as deferred. - Preserve genuine failures for auth, image, docker, malformed ref, untrusted owner, or other non-capacity launch errors. ## Acceptance - A test covers global cap reached during director dispatch and asserts the ledger/comment state is queued/deferred, not failed. - A test covers a non-capacity dispatch failure still producing failure state. - `ward exec test` passes, or the issue comment names the exact failing check. Closes this issue when merged.

coilyco-ops added the

headless

labels

2026-07-10 00:47:56 +00:00

coilyco-ops commented

2026-07-10 00:48:00 +00:00

Author

Member

WARD-DISPATCH: failed ❌

failure details

This forwarded dispatch failed after the issue was already reserved.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/ward#902 --harness codex
Container: engineer-codex-ward-902
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T004756Z-director-codex-zv98-coilyco-flight-deck-ward-902.log
Failure: ward agent engineer --harness codex: global engineer limit is reached: 10 running (limit 10); wait for a run to finish or run ward agent reap for stale engineers

Retry: choose another harness if the first one is down, or rerun with --force if the reservation is stale.

— Codex, via ward agent

WARD-DISPATCH: failed ❌ <details><summary>failure details</summary> This forwarded dispatch failed after the issue was already reserved. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/ward#902 --harness codex` Container: `engineer-codex-ward-902` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T004756Z-director-codex-zv98-coilyco-flight-deck-ward-902.log` Failure: `ward agent engineer --harness codex: global engineer limit is reached: 10 running (limit 10); wait for a run to finish or run `ward agent reap` for stale engineers` Retry: choose another harness if the first one is down, or rerun with `--force` if the reservation is stale. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 04:46:25 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-902 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T04:46:24Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#902 · branch issue-902 · harness codex · workflow pull-requests
Run: engineer-codex-ward-902 · ward v0.555.0 · dispatched 2026-07-10T04:46:24Z
Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T00:48:00Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-902` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T04:46:24Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#902` · branch `issue-902` · harness `codex` · workflow `pull-requests` - **Run:** `engineer-codex-ward-902` · ward `v0.555.0` · dispatched `2026-07-10T04:46:24Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T00:48:00Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-10 04:46:32 +00:00

Always return a log locator for broker-forwarded dispatches #941

coilyco-ops commented

2026-07-10 04:54:44 +00:00

Author

Member

WARD-RESERVATION: released 🛑

release details

Run never started. ward container reap released container ward (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --force.

— Codex, via ward agent

WARD-RESERVATION: released 🛑 <details><summary>release details</summary> Run never started. `ward container reap` released container `ward` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--force`. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 04:57:02 +00:00

Author

Member

WARD-RESERVATION: released 🛑

release details

— Codex, via ward agent

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 04:58:39 +00:00

Treat global engineer concurrency cap as deferred instead of dispatch failure #945

coilyco-ops commented

2026-07-10 04:59:54 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-requests; review summary: review gate skipped by ~/.ward/config.yaml default

Implemented a typed engineer-cap backpressure error, requeued it in director dispatch, and changed broker-forwarded launches to post a deferred comment instead of failed. PR #945 is open and green.

confidence: high
surprise: the repo's test and hook runs needed temp/cache overrides because the overlay was nearly full.
follow-ups: none.

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-requests; review summary: review gate skipped by ~/.ward/config.yaml default Implemented a typed engineer-cap backpressure error, requeued it in director dispatch, and changed broker-forwarded launches to post a deferred comment instead of failed. PR #945 is open and green. confidence: high surprise: the repo's test and hook runs needed temp/cache overrides because the overlay was nearly full. follow-ups: none. </details>

coilyco-ops closed this issue

2026-07-10 05:08:30 +00:00