Keep director surfaces from running out of writable test space

coilyco-ops commented

2026-07-10 10:10:13 +00:00

Member

Problem

During the 2026-07-10 director review pass, local verification of ward#1002 failed before reaching the Go tests because the director surface filesystem was full.

Observed from the read-only director container:

$ go test ./cmd/ward -run 'Test(...)' -count=1
crypto/internal/fips140/nistec: mkdir /scratch/go-build4002013359/b147/: no space left on device
...
FAIL github.com/coilyco-flight-deck/ward/cmd/ward [build failed]

$ df -h /scratch /tmp /workspace
Filesystem      Size  Used Avail Use% Mounted on
overlay          59G   56G   32M 100% /

A shallow PR checkout in /scratch was only about 5 MB, and /tmp was small, so this was not the review clone itself. The result is that a director surface can read and file issues but cannot run focused verification commands for Go PRs.

This directly degraded the review for ward#1002: the PR had to be blocked on the remote red status without a local test reproduction because the surface could not compile.

Acceptance criteria

A fresh read-only director surface has enough writable temp/cache space for focused repo verification, including a small Go package test after downloading modules.
If the surface intentionally has a small writable budget, warded/director startup prints the limit and the recommended cache/temp location.
Go build cache/temp paths in director surfaces point at a writable location with enough space, or the surface preflight fails with an actionable diagnostic.
A disk-full condition is visible in warded list, warded logs, or the reservation/outcome path instead of silently degrading review quality.
The fix does not require agents to clean host Docker state manually during a review pass.

Related: coilyco-flight-deck/ward#1002 and the ongoing director usability sweep from coilysiren/inbox#188.

## Problem During the 2026-07-10 director review pass, local verification of `ward#1002` failed before reaching the Go tests because the director surface filesystem was full. Observed from the read-only director container: ```text $ go test ./cmd/ward -run 'Test(...)' -count=1 crypto/internal/fips140/nistec: mkdir /scratch/go-build4002013359/b147/: no space left on device ... FAIL github.com/coilyco-flight-deck/ward/cmd/ward [build failed] $ df -h /scratch /tmp /workspace Filesystem Size Used Avail Use% Mounted on overlay 59G 56G 32M 100% / ``` A shallow PR checkout in `/scratch` was only about 5 MB, and `/tmp` was small, so this was not the review clone itself. The result is that a director surface can read and file issues but cannot run focused verification commands for Go PRs. This directly degraded the review for `ward#1002`: the PR had to be blocked on the remote red status without a local test reproduction because the surface could not compile. ## Acceptance criteria * A fresh read-only director surface has enough writable temp/cache space for focused repo verification, including a small Go package test after downloading modules. * If the surface intentionally has a small writable budget, `warded`/director startup prints the limit and the recommended cache/temp location. * Go build cache/temp paths in director surfaces point at a writable location with enough space, or the surface preflight fails with an actionable diagnostic. * A disk-full condition is visible in `warded list`, `warded logs`, or the reservation/outcome path instead of silently degrading review quality. * The fix does not require agents to clean host Docker state manually during a review pass. Related: coilyco-flight-deck/ward#1002 and the ongoing director usability sweep from coilysiren/inbox#188.

coilyco-ops added the

headless

P2

labels

2026-07-10 10:10:13 +00:00

coilyco-ops commented

2026-07-10 10:10:25 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1008 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:10:24Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1008 · branch issue-1008 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1008 · ward v0.573.0 · dispatched 2026-07-10T10:10:24Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1008` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:10:24Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1008` · branch `issue-1008` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1008` · ward `v0.573.0` · dispatched `2026-07-10T10:10:24Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 10:43:49 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1008 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:43:49Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1008 · branch issue-1008 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1008 · ward v0.580.0 · dispatched 2026-07-10T10:43:49Z
Comment thread: 0 included in the pre-flight read, 1 stripped (ward's own automated comments).
- stripped: @coilyco-ops (2026-07-10T10:10:25Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1008` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:43:49Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1008` · branch `issue-1008` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1008` · ward `v0.580.0` · dispatched `2026-07-10T10:43:49Z` - **Comment thread:** 0 included in the pre-flight read, 1 stripped (ward's own automated comments). - stripped: @coilyco-ops (2026-07-10T10:10:25Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 10:53:34 +00:00

Keep director surfaces from running out of writable test space #1024

coilyco-ops commented

2026-07-10 10:56:51 +00:00

Author

Member

WARD-OUTCOME: blocked 🛑

details

workflow: pull-requests-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default

Blocked on coilyco-flight-deck/ward#1009. The PR CI go test step failed in run 1621 after 7s, and the local ward exec test reproduction fails on the current source-bundle/specgen cleanup with:

TestCapabilityGuardfilesExist
TestDefaultsAssetsMirrorWardKDL
TestRunDoctorWithValidBundle
TestExecAssetsMirrorWardKDL
TestFleetAssetsMirrorWardKDL
TestOverrideForgejoCreateIssueAddsQuietFlag
TestForgejoGraftInventory
TestOverrideForgejoViewIssueSwapsLeaf
TestOpsAssetsMatchWardKDL
TestOpsForgejoMounts
TestOpsForgejoIssueListAllMounts
TestOpsForgejoIssueCommentDeleteMounts
TestOpsForgejoNamespaceRerooted
TestRunSetupWithUnsetRef
TestRunSetupWithFixtureRef
TestSmartDefaultsBaked
TestTopologyAssetsMirrorWardKDL
TestMountWardKdlExecMountsNewSurfaces
TestMountWardKdlExecSkipsCollisions

The scratch/cache fix itself is in place. The branch is blocked by the repo-wide source-bundle/specgen cleanup, not by this change.

WARD-OUTCOME: blocked 🛑 <details><summary>details</summary> workflow: pull-requests-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default Blocked on coilyco-flight-deck/ward#1009. The PR CI `go test` step failed in run 1621 after 7s, and the local `ward exec test` reproduction fails on the current source-bundle/specgen cleanup with: - TestCapabilityGuardfilesExist - TestDefaultsAssetsMirrorWardKDL - TestRunDoctorWithValidBundle - TestExecAssetsMirrorWardKDL - TestFleetAssetsMirrorWardKDL - TestOverrideForgejoCreateIssueAddsQuietFlag - TestForgejoGraftInventory - TestOverrideForgejoViewIssueSwapsLeaf - TestOpsAssetsMatchWardKDL - TestOpsForgejoMounts - TestOpsForgejoIssueListAllMounts - TestOpsForgejoIssueCommentDeleteMounts - TestOpsForgejoNamespaceRerooted - TestRunSetupWithUnsetRef - TestRunSetupWithFixtureRef - TestSmartDefaultsBaked - TestTopologyAssetsMirrorWardKDL - TestMountWardKdlExecMountsNewSurfaces - TestMountWardKdlExecSkipsCollisions The scratch/cache fix itself is in place. The branch is blocked by the repo-wide source-bundle/specgen cleanup, not by this change. </details>

coilyco-ops commented

2026-07-10 17:11:28 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1008 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T17:11:27Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1008 · branch issue-1008 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1008 · ward v0.584.0 · dispatched 2026-07-10T17:11:27Z
Comment thread: 1 included in the pre-flight read, 2 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T10:56:51Z)
- stripped: @coilyco-ops (2026-07-10T10:10:25Z), @coilyco-ops (2026-07-10T10:43:49Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1008` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T17:11:27Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1008` · branch `issue-1008` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1008` · ward `v0.584.0` · dispatched `2026-07-10T17:11:27Z` - **Comment thread:** 1 included in the pre-flight read, 2 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T10:56:51Z) - stripped: @coilyco-ops (2026-07-10T10:10:25Z), @coilyco-ops (2026-07-10T10:43:49Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 17:34:57 +00:00

Author

Member

WARD-OUTCOME: blocked 🛑

details

workflow: pull-requests-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default

Blocked on coilyco-flight-deck/ward#1008. The PR CI go test step failed in run 1656 after 3m3s, and the local ward exec test reproduction still trips the repo-wide source-bundle/specgen cleanup with:

TestCapabilityGuardfilesExist
TestDefaultsAssetsMirrorWardKDL
TestRunDoctorWithValidBundle
TestExecAssetsMirrorWardKDL
TestFleetAssetsMirrorWardKDL
TestOverrideForgejoCreateIssueAddsQuietFlag
TestForgejoGraftInventory
TestOverrideForgejoViewIssueSwapsLeaf
TestOpsAssetsMatchWardKDL
TestOpsForgejoMounts
TestOpsForgejoIssueListAllMounts
TestOpsForgejoIssueCommentDeleteMounts
TestOpsForgejoNamespaceRerooted
TestRunSetupWithUnsetRef
TestRunSetupWithFixtureRef
TestSmartDefaultsBaked
TestTopologyAssetsMirrorWardKDL
TestMountWardKdlExecMountsNewSurfaces
TestMountWardKdlExecSkipsCollisions

The scratch/cache fix itself is in place. The branch is blocked by the repo-wide source-bundle/specgen cleanup, not by this change.

WARD-OUTCOME: blocked 🛑 <details><summary>details</summary> workflow: pull-requests-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default Blocked on coilyco-flight-deck/ward#1008. The PR CI `go test` step failed in run 1656 after 3m3s, and the local `ward exec test` reproduction still trips the repo-wide source-bundle/specgen cleanup with: - TestCapabilityGuardfilesExist - TestDefaultsAssetsMirrorWardKDL - TestRunDoctorWithValidBundle - TestExecAssetsMirrorWardKDL - TestFleetAssetsMirrorWardKDL - TestOverrideForgejoCreateIssueAddsQuietFlag - TestForgejoGraftInventory - TestOverrideForgejoViewIssueSwapsLeaf - TestOpsAssetsMatchWardKDL - TestOpsForgejoMounts - TestOpsForgejoIssueListAllMounts - TestOpsForgejoIssueCommentDeleteMounts - TestOpsForgejoNamespaceRerooted - TestRunSetupWithUnsetRef - TestRunSetupWithFixtureRef - TestSmartDefaultsBaked - TestTopologyAssetsMirrorWardKDL - TestMountWardKdlExecMountsNewSurfaces - TestMountWardKdlExecSkipsCollisions The scratch/cache fix itself is in place. The branch is blocked by the repo-wide source-bundle/specgen cleanup, not by this change. </details>

coilyco-ops referenced this issue

2026-07-10 17:34:58 +00:00

Keep director surfaces from running out of writable test space #1024

coilyco-ops commented

2026-07-11 01:32:14 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1008 on host KAI-DESKTOP-TOWER.

Reserved by ward agent --harness codex (reserved 2026-07-11T01:32:15Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1008 · branch issue-1008 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1008 · ward v0.605.0 · dispatched 2026-07-11T01:32:15Z
Comment thread: 2 included in the pre-flight read, 3 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T10:56:51Z), @coilyco-ops (2026-07-10T17:34:57Z)
- stripped: @coilyco-ops (2026-07-10T10:10:25Z), @coilyco-ops (2026-07-10T10:43:49Z), @coilyco-ops (2026-07-10T17:11:28Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1008` on host `KAI-DESKTOP-TOWER`. Reserved by `ward agent --harness codex` (reserved 2026-07-11T01:32:15Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1008` · branch `issue-1008` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1008` · ward `v0.605.0` · dispatched `2026-07-11T01:32:15Z` - **Comment thread:** 2 included in the pre-flight read, 3 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T10:56:51Z), @coilyco-ops (2026-07-10T17:34:57Z) - stripped: @coilyco-ops (2026-07-10T10:10:25Z), @coilyco-ops (2026-07-10T10:43:49Z), @coilyco-ops (2026-07-10T17:11:28Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-11 01:47:01 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA

felt: narrow, mostly test-harness cleanup after the scratch fix.
confidence: high.
surprise: the full cmd/ward suite still has unrelated failures outside this path.
follow-ups: none.

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA felt: narrow, mostly test-harness cleanup after the scratch fix. confidence: high. surprise: the full `cmd/ward` suite still has unrelated failures outside this path. follow-ups: none. </details>

coilysiren closed this issue

2026-07-11 02:03:44 +00:00

coilysiren referenced this issue

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

Rows
Columns

Keep director surfaces from running out of writable test space #1008

Problem

Acceptance criteria