Fix broken warded launcher in read-only director surfaces

coilyco-ops commented

2026-07-10 05:31:41 +00:00

Member

Problem

A read-only director surface was required to dispatch sibling engineers for ward#950 and agentic-os#415, but both warded engineer ... attempts failed immediately:

/bin/bash: line 1: /usr/local/bin/warded: cannot execute: required file not found

Observed details from the surface:

/usr/local/bin/warded is a symlink to ward, and /usr/local/bin/ward is a symlink to /opt/ward/ward.
/opt/ward is a read-only host mount.
/opt/ward/ward appears and disappears across probes from inside the container. ls /opt/ward showed an empty directory at one point, while Python os.path.exists('/opt/ward/ward') returned true in a separate probe, then later head and Python read attempts returned No such file or directory.
ldd /usr/local/bin/warded reported not a dynamic executable, but execution still failed before warded could print help.

Impact

The surface cannot satisfy the read-only director capture-and-dispatch obligation for newly filed work. The issue can still be captured through the Forgejo API, but the sibling run never starts.

Requested change

Make read-only director surfaces mount a stable executable warded/ward binary or provide a separate dispatch-only launcher path that works inside the container architecture. Add a smoke check at surface startup that runs warded --help or equivalent and fails fast if dispatch is unavailable.

## Problem A read-only director surface was required to dispatch sibling engineers for `ward#950` and `agentic-os#415`, but both `warded engineer ...` attempts failed immediately: ```text /bin/bash: line 1: /usr/local/bin/warded: cannot execute: required file not found ``` Observed details from the surface: * `/usr/local/bin/warded` is a symlink to `ward`, and `/usr/local/bin/ward` is a symlink to `/opt/ward/ward`. * `/opt/ward` is a read-only host mount. * `/opt/ward/ward` appears and disappears across probes from inside the container. `ls /opt/ward` showed an empty directory at one point, while Python `os.path.exists('/opt/ward/ward')` returned true in a separate probe, then later `head` and Python read attempts returned `No such file or directory`. * `ldd /usr/local/bin/warded` reported `not a dynamic executable`, but execution still failed before `warded` could print help. ## Impact The surface cannot satisfy the read-only director capture-and-dispatch obligation for newly filed work. The issue can still be captured through the Forgejo API, but the sibling run never starts. ## Requested change Make read-only director surfaces mount a stable executable `warded`/`ward` binary or provide a separate dispatch-only launcher path that works inside the container architecture. Add a smoke check at surface startup that runs `warded --help` or equivalent and fails fast if dispatch is unavailable.

coilyco-ops referenced this issue

2026-07-10 05:31:41 +00:00

Add ward-kdl fetch overlays and first input guard sugar #950

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-10 05:31:42 +00:00

Replace Forgejo Actions log bash bridge with ward fetch overlay #415

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-10 05:35:06 +00:00

Inline remaining Telegram alert steps so CI can prepare workflows #416

coilyco-ops commented

2026-07-10 05:37:53 +00:00

Author

Member

This is the canonical issue for the same failure captured earlier as coilyco-flight-deck/agentic-os#413.

The AOS issue has one useful extra observation: an earlier surface saw warded --help and warded list work, then later the same active session failed with /usr/local/bin/warded: No such file or directory; the workaround was downloading ward-linux-arm64 v0.554.0 into /scratch/ward. That points at director-surface runtime/mount lifetime, not just image build presence.

Closing the AOS duplicate keeps this in ward, where the director surface launcher and /opt/ward mount are owned.

This is the canonical issue for the same failure captured earlier as coilyco-flight-deck/agentic-os#413. The AOS issue has one useful extra observation: an earlier surface saw `warded --help` and `warded list` work, then later the same active session failed with `/usr/local/bin/warded: No such file or directory`; the workaround was downloading `ward-linux-arm64` v0.554.0 into `/scratch/ward`. That points at director-surface runtime/mount lifetime, not just image build presence. Closing the AOS duplicate keeps this in ward, where the director surface launcher and `/opt/ward` mount are owned.

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-10 05:37:53 +00:00

Investigate ward binary disappearing during director sessions #413

coilyco-ops commented

2026-07-10 05:41:52 +00:00

Author

Member

Scope expansion from Kai: do not fix this as only "director asset deleted by dispatched engineer". Treat it as a general many-to-many asset-lifetime collision.

Any long-lived container role can share /opt/ward or other mounted ward asset directories with any other role, and any reaper path for one container can invalidate the backing path for another still-running container. The fix should make asset ownership explicit across all producer/consumer combinations, not special-case director -> engineer.

Acceptance criteria update:

Asset directories are per-container, or shared directories are reference-counted / lease-tracked by every live container that mounts them.
Reaping one container cannot delete or rotate assets mounted into any other live container, regardless of role.
The runtime check covers the generic invariant: every live container with /opt/ward mounted can still open and execute the mounted ward binary after unrelated containers are reaped.
Tests or a harness scenario should include at least two independent live consumers plus reaping of a third container that used the same asset source, so the fix is not only director-dispatch specific.

Scope expansion from Kai: do not fix this as only "director asset deleted by dispatched engineer". Treat it as a general many-to-many asset-lifetime collision. Any long-lived container role can share `/opt/ward` or other mounted ward asset directories with any other role, and any reaper path for one container can invalidate the backing path for another still-running container. The fix should make asset ownership explicit across all producer/consumer combinations, not special-case director -> engineer. Acceptance criteria update: * Asset directories are per-container, or shared directories are reference-counted / lease-tracked by every live container that mounts them. * Reaping one container cannot delete or rotate assets mounted into any other live container, regardless of role. * The runtime check covers the generic invariant: every live container with `/opt/ward` mounted can still open and execute the mounted ward binary after unrelated containers are reaped. * Tests or a harness scenario should include at least two independent live consumers plus reaping of a third container that used the same asset source, so the fix is not only director-dispatch specific.

coilyco-ops commented

2026-07-10 05:43:53 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-951 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T05:43:52Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#951 · branch issue-951 · harness codex · workflow pull-requests
Run: engineer-codex-ward-951 · ward v0.555.0 · dispatched 2026-07-10T05:43:52Z
Comment thread: 2 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T05:37:53Z), @coilyco-ops (2026-07-10T05:41:52Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-951` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T05:43:52Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#951` · branch `issue-951` · harness `codex` · workflow `pull-requests` - **Run:** `engineer-codex-ward-951` · ward `v0.555.0` · dispatched `2026-07-10T05:43:52Z` - **Comment thread:** 2 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T05:37:53Z), @coilyco-ops (2026-07-10T05:41:52Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 05:53:05 +00:00

Author

Member

WARD-RESERVATION: released 🛑

release details

Run never started. ward container reap released container ward (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --force.

— Codex, via ward agent

WARD-RESERVATION: released 🛑 <details><summary>release details</summary> Run never started. `ward container reap` released container `ward` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--force`. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 06:44:05 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-951 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T06:44:04Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#951 · branch issue-951 · harness codex · workflow pull-requests
Run: engineer-codex-ward-951 · ward v0.555.0 · dispatched 2026-07-10T06:44:04Z
Comment thread: 3 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T05:37:53Z), @coilyco-ops (2026-07-10T05:41:52Z), @coilyco-ops (2026-07-10T05:53:05Z)
- stripped: @coilyco-ops (2026-07-10T05:43:53Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-951` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T06:44:04Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#951` · branch `issue-951` · harness `codex` · workflow `pull-requests` - **Run:** `engineer-codex-ward-951` · ward `v0.555.0` · dispatched `2026-07-10T06:44:04Z` - **Comment thread:** 3 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T05:37:53Z), @coilyco-ops (2026-07-10T05:41:52Z), @coilyco-ops (2026-07-10T05:53:05Z) - stripped: @coilyco-ops (2026-07-10T05:43:53Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 06:48:02 +00:00

Author

Member

WARD-RESERVATION: released 🛑

release details

Run never started. ward container reap released container ward (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --force.

— Codex, via ward agent

WARD-RESERVATION: released 🛑 <details><summary>release details</summary> Run never started. `ward container reap` released container `ward` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the hold it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--force`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 06:49:52 +00:00

Fix broken warded launcher in read-only director surfaces #961

coilyco-ops commented

2026-07-10 06:54:44 +00:00

Author

Member

WARD-OUTCOME: blocked 🛑

details

workflow: pull-requests; review summary: skip because --skip-preflight / --no-preflight also skips review

The launcher fix is in place and validated locally with go test ./cmd/ward -run 'TestEntrypoint|TestLaunchAgentReportsOOMKilled' and pre-commit run --all-files.

Forgejo PR #961 failed twice in CI setup before checkout. Run 1564 failed in 3s with no job logs, and the retrigger commit 373656d5 produced the same failure as run 1565. canRerun is false and the job page exposes no step logs, so there is nothing actionable left to patch in-repo.

confidence: medium
surprise: the workflow dies in setup, not in the test steps, so this looks like a runner/setup outage or platform regression.
follow-up: rerun the PR once the Forgejo runner lane is healthy.

WARD-OUTCOME: blocked 🛑 <details><summary>details</summary> workflow: pull-requests; review summary: skip because --skip-preflight / --no-preflight also skips review The launcher fix is in place and validated locally with `go test ./cmd/ward -run 'TestEntrypoint|TestLaunchAgentReportsOOMKilled'` and `pre-commit run --all-files`. Forgejo PR #961 failed twice in CI setup before checkout. Run 1564 failed in 3s with no job logs, and the retrigger commit `373656d5` produced the same failure as run 1565. `canRerun` is false and the job page exposes no step logs, so there is nothing actionable left to patch in-repo. confidence: medium surprise: the workflow dies in setup, not in the test steps, so this looks like a runner/setup outage or platform regression. follow-up: rerun the PR once the Forgejo runner lane is healthy. </details>

coilyco-ops closed this issue

2026-07-10 06:59:43 +00:00

Rows
Columns

Fix broken warded launcher in read-only director surfaces #951

Problem

Impact

Requested change