Director surface heartbeat marks broker-forwarded runs failed when local Docker is unavailable

coilyco-ops commented

2026-07-10 00:45:41 +00:00

Member

Problem

From a read-only director surface on 2026-07-10, ward agent director --repo coilyco-flight-deck/ward --harness codex --engineer-harness codex --no-triage --max-parallel 10 --max-cycles 2 correctly forwarded engineer dispatches through the host broker, but the next reconcile tried to inspect Docker from the surface container:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  coilyco-flight-deck/ward#875 -> failed: container exited without a WARD-OUTCOME comment; read its log

The same happened for #895, #894, #879, #893, #885, #892, #896, and #876. #890 was re-queued as container died pre-launch even though the dispatch had been forwarded through the host broker.

Expected

A director running inside a read-only surface should reconcile broker-forwarded engineers through broker-visible state, issue reservations/comments, or another host-side read path. It should not classify runs as failed merely because the surface container itself has no Docker daemon.

Do

Reproduce with a director surface command that forwards engineers through the dispatch broker.
Trace the reconcile path that calls local Docker after broker forwarding.
Route that read through the broker or make the surface path rely on issue-thread state until host-side state is available.
Ensure a missing local Docker socket in a director surface does not produce false failed: container exited without a WARD-OUTCOME comment ledger entries.

Acceptance

A test covers director-surface reconcile with no local Docker socket after broker-forwarded dispatches.
Broker-forwarded runs are not marked failed only because the surface cannot reach Docker.
ward exec test passes, or the issue comment names the exact failing check.

Closes this issue when merged.

## Problem From a read-only director surface on 2026-07-10, `ward agent director --repo coilyco-flight-deck/ward --harness codex --engineer-harness codex --no-triage --max-parallel 10 --max-cycles 2` correctly forwarded engineer dispatches through the host broker, but the next reconcile tried to inspect Docker from the surface container: ```text Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? coilyco-flight-deck/ward#875 -> failed: container exited without a WARD-OUTCOME comment; read its log ``` The same happened for #895, #894, #879, #893, #885, #892, #896, and #876. #890 was re-queued as `container died pre-launch` even though the dispatch had been forwarded through the host broker. ## Expected A director running inside a read-only surface should reconcile broker-forwarded engineers through broker-visible state, issue reservations/comments, or another host-side read path. It should not classify runs as failed merely because the surface container itself has no Docker daemon. ## Do - Reproduce with a director surface command that forwards engineers through the dispatch broker. - Trace the reconcile path that calls local Docker after broker forwarding. - Route that read through the broker or make the surface path rely on issue-thread state until host-side state is available. - Ensure a missing local Docker socket in a director surface does not produce false `failed: container exited without a WARD-OUTCOME comment` ledger entries. ## Acceptance - A test covers director-surface reconcile with no local Docker socket after broker-forwarded dispatches. - Broker-forwarded runs are not marked failed only because the surface cannot reach Docker. - `ward exec test` passes, or the issue comment names the exact failing check. Closes this issue when merged.

coilyco-ops added the

headless

P1

labels

2026-07-10 00:45:52 +00:00

coilyco-ops commented

2026-07-10 00:45:56 +00:00

Author

Member

WARD-DISPATCH: failed ❌

failure details

This forwarded dispatch failed after the issue was already reserved.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/ward#900 --harness codex
Container: engineer-codex-ward-900
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T004551Z-director-codex-zv98-coilyco-flight-deck-ward-900.log
Failure: ward agent engineer --harness codex: global engineer limit is reached: 10 running (limit 10); wait for a run to finish or run ward agent reap for stale engineers

Retry: choose another harness if the first one is down, or rerun with --force if the reservation is stale.

— Codex, via ward agent

WARD-DISPATCH: failed ❌ <details><summary>failure details</summary> This forwarded dispatch failed after the issue was already reserved. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/ward#900 --harness codex` Container: `engineer-codex-ward-900` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T004551Z-director-codex-zv98-coilyco-flight-deck-ward-900.log` Failure: `ward agent engineer --harness codex: global engineer limit is reached: 10 running (limit 10); wait for a run to finish or run `ward agent reap` for stale engineers` Retry: choose another harness if the first one is down, or rerun with `--force` if the reservation is stale. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-10 00:47:52 +00:00

Treat global engineer concurrency cap as deferred instead of dispatch failure #902

coilyco-ops commented

2026-07-10 02:06:14 +00:00

Author

Member

Fresh reproduction from a director read-only surface on 2026-07-10 while trying to dispatch coilyco-bridge/deploy#127.

Observed sequence:

warded coilyco-bridge/deploy#127 --repo coilyco-flight-deck/infrastructure --repo coilyco-bridge/ward-mcp --harness codex ... returned success with ward dispatch broker: forwarded ... to host ward.
First reported host log path: /Users/kai/.ward/agent-logs/dispatch/20260710T015942Z-director-codex-hx94-coilyco-bridge-deploy-127.log.
deploy#127 had no reservation comment and no outcome comment afterward.
warded logs coilyco-bridge/deploy#127 --tail 240 returned dispatch broker: no engineer log source matches "coilyco-bridge/deploy#127".
Re-dispatch at 20260710T020525Z returned the same forwarded-success shape and also produced no reservation comment within a few seconds.
This director container cannot access local Docker: docker ps returns Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Impact: the director surface can report a work item as dispatched even though the issue never becomes reserved and no engineer log source exists. In this case deploy#127 remains open and uncarried.

This looks related to the existing Docker-unavailable broker-forwarded surface issue, but it also needs the launch path to fail loud or create a visible deferred/failed reservation rather than returning a successful forwarded message.

Fresh reproduction from a director read-only surface on 2026-07-10 while trying to dispatch `coilyco-bridge/deploy#127`. Observed sequence: * `warded coilyco-bridge/deploy#127 --repo coilyco-flight-deck/infrastructure --repo coilyco-bridge/ward-mcp --harness codex ...` returned success with `ward dispatch broker: forwarded ... to host ward`. * First reported host log path: `/Users/kai/.ward/agent-logs/dispatch/20260710T015942Z-director-codex-hx94-coilyco-bridge-deploy-127.log`. * `deploy#127` had no reservation comment and no outcome comment afterward. * `warded logs coilyco-bridge/deploy#127 --tail 240` returned `dispatch broker: no engineer log source matches "coilyco-bridge/deploy#127"`. * Re-dispatch at `20260710T020525Z` returned the same forwarded-success shape and also produced no reservation comment within a few seconds. * This director container cannot access local Docker: `docker ps` returns `Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?` Impact: the director surface can report a work item as dispatched even though the issue never becomes reserved and no engineer log source exists. In this case `deploy#127` remains open and uncarried. This looks related to the existing Docker-unavailable broker-forwarded surface issue, but it also needs the launch path to fail loud or create a visible deferred/failed reservation rather than returning a successful forwarded message.

coilyco-ops commented

2026-07-10 02:14:14 +00:00

Author

Member

Additional retry evidence from the same director surface on 2026-07-10.

After Kai asked to retry deploy#127:

Codex retry at 20260710T021313Z returned broker-forwarded with host log path /Users/kai/.ward/agent-logs/dispatch/20260710T021313Z-director-codex-hx94-coilyco-bridge-deploy-127.log. Five seconds later, deploy#127 still had no reservation comments and warded logs coilyco-bridge/deploy#127 --tail 80 still returned no engineer log source matches.
Claude alternate-harness retry at 20260710T021347Z returned broker-forwarded with host log path /Users/kai/.ward/agent-logs/dispatch/20260710T021347Z-director-codex-hx94-coilyco-bridge-deploy-127.log. Five seconds later, deploy#127 still had no reservation comments and the same no-log-source result.

This suggests the problem is not Codex-specific. The broker launch path is acknowledging forwarding while no visible reservation or log source is created for the target issue.

Additional retry evidence from the same director surface on 2026-07-10. After Kai asked to retry `deploy#127`: * Codex retry at `20260710T021313Z` returned broker-forwarded with host log path `/Users/kai/.ward/agent-logs/dispatch/20260710T021313Z-director-codex-hx94-coilyco-bridge-deploy-127.log`. Five seconds later, `deploy#127` still had no reservation comments and `warded logs coilyco-bridge/deploy#127 --tail 80` still returned `no engineer log source matches`. * Claude alternate-harness retry at `20260710T021347Z` returned broker-forwarded with host log path `/Users/kai/.ward/agent-logs/dispatch/20260710T021347Z-director-codex-hx94-coilyco-bridge-deploy-127.log`. Five seconds later, `deploy#127` still had no reservation comments and the same no-log-source result. This suggests the problem is not Codex-specific. The broker launch path is acknowledging forwarding while no visible reservation or log source is created for the target issue.

coilyco-ops referenced this issue

2026-07-10 04:13:53 +00:00

Mine warded-agent history for regression tests and review doctrine #934

coilyco-ops commented

2026-07-10 04:46:20 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-900 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T04:46:19Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#900 · branch issue-900 · harness codex · workflow pull-requests
Run: engineer-codex-ward-900 · ward v0.555.0 · dispatched 2026-07-10T04:46:19Z
Comment thread: 3 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T00:45:56Z), @coilyco-ops (2026-07-10T02:06:14Z), @coilyco-ops (2026-07-10T02:14:14Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-900` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T04:46:19Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#900` · branch `issue-900` · harness `codex` · workflow `pull-requests` - **Run:** `engineer-codex-ward-900` · ward `v0.555.0` · dispatched `2026-07-10T04:46:19Z` - **Comment thread:** 3 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T00:45:56Z), @coilyco-ops (2026-07-10T02:06:14Z), @coilyco-ops (2026-07-10T02:14:14Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-10 04:46:32 +00:00

Always return a log locator for broker-forwarded dispatches #941

coilyco-ops commented

2026-07-10 04:55:01 +00:00

Author

Member

WARD-RESERVATION: released 🛑

release details

Run never started. ward container reap released container ward (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --force.

— Codex, via ward agent

WARD-RESERVATION: released 🛑 <details><summary>release details</summary> Run never started. `ward container reap` released container `ward` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--force`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 05:07:11 +00:00

fix director brokered reconcile #948

coilyco-ops commented

2026-07-10 05:08:39 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-requests; review summary: review gate skipped by ~/.ward/config.yaml default
felt: narrow and operational, mostly spent clearing hook-cache pressure before landing the fix
confidence: high
surprises: the local pre-commit cache filled the overlay and tmpfs before the code path did
follow-ups: none

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-requests; review summary: review gate skipped by ~/.ward/config.yaml default felt: narrow and operational, mostly spent clearing hook-cache pressure before landing the fix confidence: high surprises: the local pre-commit cache filled the overlay and tmpfs before the code path did follow-ups: none </details>

coilyco-ops closed this issue

2026-07-10 05:09:08 +00:00

Rows
Columns

Director surface heartbeat marks broker-forwarded runs failed when local Docker is unavailable #900

Problem

Expected

Do

Acceptance