Investigate ward binary disappearing during director sessions #413

New issue

Closed

opened 2026-07-10 04:56:40 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-10 04:56:40 +00:00

Member

Observed twice in read-only director surfaces: the ward executable target disappears during an active session, leaving broken symlinks.

Latest observation, 2026-07-10:

Earlier in the same director session, warded --help and warded list worked.
Later, dispatch attempts failed with /bin/bash: line 1: /usr/local/bin/warded: No such file or directory.
Inspection showed the symlinks still existed:
- /usr/local/bin/ward -> /opt/ward/ward
- /usr/local/bin/warded -> ward
But the target was missing:
- ls: cannot access /opt/ward/ward: No such file or directory
The agent did not delete or modify /opt/ward/ward. The workaround was to download ward-linux-arm64 v0.554.0 into /scratch/ward and run that directly.

Impact:

A read-only director surface can lose its dispatcher mid-session. That breaks capture-and-dispatch exactly when the surface needs to file and launch sibling work. The error is misleading because warded appears to be installed, but the symlink target is gone.

Expected behavior:

/opt/ward/ward remains present for the lifetime of a director container, or
the symlink target is placed somewhere immutable/stable for the container lifetime, or
the entrypoint/doctor detects the missing target and repairs/fails loud before the agent starts.

Acceptance criteria:

Root cause identified for why /opt/ward/ward can disappear in a live surface.
A regression check covers the director-surface binary lifetime, not just image build existence.
If agentic-os#408 adds a dev-base build-time doctor, this ticket should add the runtime/lifetime check that catches the mid-session disappearance.

Notes:

This has now happened twice according to Kai. The second occurrence was captured during work on dispatch/release-gating tickets, and the current container still shows the broken state.

Observed twice in read-only director surfaces: the `ward` executable target disappears during an active session, leaving broken symlinks. Latest observation, 2026-07-10: * Earlier in the same director session, `warded --help` and `warded list` worked. * Later, dispatch attempts failed with `/bin/bash: line 1: /usr/local/bin/warded: No such file or directory`. * Inspection showed the symlinks still existed: * `/usr/local/bin/ward -> /opt/ward/ward` * `/usr/local/bin/warded -> ward` * But the target was missing: * `ls: cannot access /opt/ward/ward: No such file or directory` * The agent did not delete or modify `/opt/ward/ward`. The workaround was to download `ward-linux-arm64` v0.554.0 into `/scratch/ward` and run that directly. Impact: A read-only director surface can lose its dispatcher mid-session. That breaks capture-and-dispatch exactly when the surface needs to file and launch sibling work. The error is misleading because `warded` appears to be installed, but the symlink target is gone. Expected behavior: * `/opt/ward/ward` remains present for the lifetime of a director container, or * the symlink target is placed somewhere immutable/stable for the container lifetime, or * the entrypoint/doctor detects the missing target and repairs/fails loud before the agent starts. Acceptance criteria: * Root cause identified for why `/opt/ward/ward` can disappear in a live surface. * A regression check covers the director-surface binary lifetime, not just image build existence. * If `agentic-os#408` adds a dev-base build-time doctor, this ticket should add the runtime/lifetime check that catches the mid-session disappearance. Notes: This has now happened twice according to Kai. The second occurrence was captured during work on dispatch/release-gating tickets, and the current container still shows the broken state.

coilyco-ops commented

2026-07-10 04:56:48 +00:00

Author

Member

WARD-DISPATCH: failed ❌

failure details

This forwarded dispatch failed after the issue was already reserved.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/agentic-os#413 --harness codex
Container: engineer-codex-agentic-os-413
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T045646Z-director-codex-vd75-coilyco-flight-deck-agentic-os-413.log
Failure: ward agent engineer --harness codex: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run ward agent reap for stale engineers

Retry: choose another harness if the first one is down, or rerun with --force if the reservation is stale.

— Codex, via ward agent

WARD-DISPATCH: failed ❌ <details><summary>failure details</summary> This forwarded dispatch failed after the issue was already reserved. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/agentic-os#413 --harness codex` Container: `engineer-codex-agentic-os-413` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T045646Z-director-codex-vd75-coilyco-flight-deck-agentic-os-413.log` Failure: `ward agent engineer --harness codex: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run `ward agent reap` for stale engineers` Retry: choose another harness if the first one is down, or rerun with `--force` if the reservation is stale. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from coilyco-flight-deck/ward

2026-07-10 05:37:53 +00:00

Fix broken warded launcher in read-only director surfaces #951

coilyco-ops commented

2026-07-10 05:37:53 +00:00

Author

Member

This is the same failure mode as coilyco-flight-deck/ward#951: the read-only director surface resolves warded through /opt/ward/ward, then the mounted target is missing or unstable during the session.

Closing this duplicate in favor of ward#951 because the fix belongs in ward's director-surface launcher/mount plumbing. The observations from this issue were copied there.

This is the same failure mode as coilyco-flight-deck/ward#951: the read-only director surface resolves `warded` through `/opt/ward/ward`, then the mounted target is missing or unstable during the session. Closing this duplicate in favor of ward#951 because the fix belongs in ward's director-surface launcher/mount plumbing. The observations from this issue were copied there.

coilyco-ops closed this issue

2026-07-10 05:37:53 +00:00