coilyco-flight-deck/ward

Fork 0

Broker-forwarded advisor dispatch returns success with no durable result #936

New issue

Closed

opened 2026-07-10 04:08:20 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-10 04:08:20 +00:00

Member

Observed from a read-only Codex director surface on 2026-07-10 while retrying ward advisor dispatches.

Failure shape:

warded advisor coilyco-flight-deck/ward#934 --thoroughness deep returned success with only ward dispatch broker: forwarded ....
No new reservation, advisor result, dispatch-failure comment, or log source appeared on ward#934 after the retry.
Retrying with --harness claude behaved the same way.
Retrying with --force behaved the same way.
warded advisor coilyco-flight-deck/ward#932 --thoroughness standard also returned success with only ward dispatch broker: forwarded ..., but produced no new advisor comment.
warded logs coilyco-flight-deck/ward#934 reports no engineer log source, and warded ps shows only the unrelated engineer-codex-agentic-os-402 container.

The earlier visible failure on ward#934 was a bootstrap 404 for v0.555.0/ward-linux-arm64, but the release API now shows that asset exists. The active regression now appears to be that broker-forwarded advisor dispatch can return success while producing no durable run, no failure annotation, and no observable log source.

Impact:

A director surface can believe a retry was accepted while the target issue remains untouched. That breaks the capture-and-dispatch loop and makes redispatch failures silent.

Expected behavior:

If a broker-forwarded advisor dispatch cannot launch or complete, ward should post a WARD-DISPATCH: failed comment with the host log path, or return non-zero to the caller. Success should mean a durable run/result exists or is observable.

Related context:

ward#934 was the retried deep advisor dispatch.
ward#932 was the smaller advisor retry used to confirm this was not issue-specific.
ward#931 recently added engineer dispatch smoke coverage, but this failure is advisor/forwarded-dispatch observability.

Observed from a read-only Codex director surface on 2026-07-10 while retrying ward advisor dispatches. Failure shape: * `warded advisor coilyco-flight-deck/ward#934 --thoroughness deep` returned success with only `ward dispatch broker: forwarded ...`. * No new reservation, advisor result, dispatch-failure comment, or log source appeared on ward#934 after the retry. * Retrying with `--harness claude` behaved the same way. * Retrying with `--force` behaved the same way. * `warded advisor coilyco-flight-deck/ward#932 --thoroughness standard` also returned success with only `ward dispatch broker: forwarded ...`, but produced no new advisor comment. * `warded logs coilyco-flight-deck/ward#934` reports no engineer log source, and `warded ps` shows only the unrelated `engineer-codex-agentic-os-402` container. The earlier visible failure on ward#934 was a bootstrap 404 for `v0.555.0/ward-linux-arm64`, but the release API now shows that asset exists. The active regression now appears to be that broker-forwarded advisor dispatch can return success while producing no durable run, no failure annotation, and no observable log source. Impact: A director surface can believe a retry was accepted while the target issue remains untouched. That breaks the capture-and-dispatch loop and makes redispatch failures silent. Expected behavior: If a broker-forwarded advisor dispatch cannot launch or complete, ward should post a `WARD-DISPATCH: failed` comment with the host log path, or return non-zero to the caller. Success should mean a durable run/result exists or is observable. Related context: * ward#934 was the retried deep advisor dispatch. * ward#932 was the smaller advisor retry used to confirm this was not issue-specific. * ward#931 recently added engineer dispatch smoke coverage, but this failure is advisor/forwarded-dispatch observability.

coilyco-ops commented

2026-07-10 04:09:36 +00:00

Author

Member

Additional retry evidence:

warded engineer coilyco-flight-deck/ward#936 returned ward dispatch broker: forwarded ... with exit 0, but after 20 seconds ward#936 had no comments/reservation and warded ps showed no running engineer containers.
After all other engineer containers had exited, warded advisor coilyco-flight-deck/ward#934 --harness codex --thoroughness deep was retried again. It also returned only ward dispatch broker: forwarded ...; after 20 seconds ward#934 still had no new comment and warded ps showed no running engineer containers.

This rules out the earlier active engineer as the cause. Broker-forwarded dispatch is returning apparent success while creating no durable run and no failure annotation.

Additional retry evidence: * `warded engineer coilyco-flight-deck/ward#936` returned `ward dispatch broker: forwarded ...` with exit 0, but after 20 seconds ward#936 had no comments/reservation and `warded ps` showed no running engineer containers. * After all other engineer containers had exited, `warded advisor coilyco-flight-deck/ward#934 --harness codex --thoroughness deep` was retried again. It also returned only `ward dispatch broker: forwarded ...`; after 20 seconds ward#934 still had no new comment and `warded ps` showed no running engineer containers. This rules out the earlier active engineer as the cause. Broker-forwarded dispatch is returning apparent success while creating no durable run and no failure annotation.

coilyco-ops commented

2026-07-10 04:19:57 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-936 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T04:19:56Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#936 · branch issue-936 · harness codex · workflow pull-requests
Run: engineer-codex-ward-936 · ward v0.554.0 · dispatched 2026-07-10T04:19:56Z
Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T04:09:36Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.554.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-936` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T04:19:56Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#936` · branch `issue-936` · harness `codex` · workflow `pull-requests` - **Run:** `engineer-codex-ward-936` · ward `v0.554.0` · dispatched `2026-07-10T04:19:56Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T04:09:36Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.554.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 04:24:06 +00:00

Author

Member

WARD-RESERVATION: released 🛑

release details

Run never started. ward container reap released container ward (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --force.

— Codex, via ward agent

WARD-RESERVATION: released 🛑 <details><summary>release details</summary> Run never started. `ward container reap` released container `ward` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--force`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 04:25:55 +00:00

fix: wait for broker launch acknowledgement #939

coilyco-ops commented

2026-07-10 04:27:17 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-requests; review summary: review gate skipped by ~/.ward/config.yaml default
retrospective: the fix was small once the early-success branch was isolated and pinned with a regression test.
confidence: high
surprises: the Forgejo Actions run was already green when I checked the PR.
follow-ups: none.

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-requests; review summary: review gate skipped by ~/.ward/config.yaml default retrospective: the fix was small once the early-success branch was isolated and pinned with a regression test. confidence: high surprises: the Forgejo Actions run was already green when I checked the PR. follow-ups: none. </details>