coilyco-flight-deck/ward

Watch

Recover full host filesystem causing pre-agent Docker resource deaths #1603

New issue

Closed

opened 2026-07-28 09:23:39 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-28 09:23:39 +00:00

Member

Problem

Fresh brokered engineers are failing before the agent starts because the host filesystem backing /scratch is full.

Observed from the read-only burndown surface on 2026-07-28 while dispatching coilyco-flight-deck/ward#1597:

ward-container: fatal: Docker resource constraint for /scratch: only 0B free of 125.7GiB; need at least 8.0MiB for focused Go verification; recommended cache/temp location is /scratch (Go cache root /scratch/go-build)

Host proof from the same surface:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       126G  124G     0 100% /gitcache

du -h --max-depth=1 /scratch
0       /scratch

So the visible /scratch directory is empty, but the shared backing filesystem is full. This caused coilyco-flight-deck/ward#1597 to exit before launching the agent.

Desired outcome

Recover enough host space for fresh Ward engineers to launch, and make the failure easier to diagnose or self-heal if repo/cache growth fills the same filesystem again.

Acceptance

Identify the disk consumer on the host filesystem backing /scratch and /gitcache.
Clean only Ward-owned stale/cache state or document the exact operator action if cleanup is unsafe from an engineer container.
Verify a fresh headless engineer can pass the pre-agent resource check after cleanup.
Redispatch or explicitly unblock coilyco-flight-deck/ward#1597 once launch capacity is restored.
If a code change is warranted, keep it focused on preflight diagnostics or cache cleanup safety, then land through the normal Ward workflow.

Context

This was surfaced during the autonomous Ward burndown. It is now a lane blocker for new work, so prioritize diagnosis and conservative recovery over unrelated refactors.

## Problem Fresh brokered engineers are failing before the agent starts because the host filesystem backing /scratch is full. Observed from the read-only burndown surface on 2026-07-28 while dispatching coilyco-flight-deck/ward#1597: ```text ward-container: fatal: Docker resource constraint for /scratch: only 0B free of 125.7GiB; need at least 8.0MiB for focused Go verification; recommended cache/temp location is /scratch (Go cache root /scratch/go-build) ``` Host proof from the same surface: ```text Filesystem Size Used Avail Use% Mounted on /dev/vda1 126G 124G 0 100% /gitcache du -h --max-depth=1 /scratch 0 /scratch ``` So the visible /scratch directory is empty, but the shared backing filesystem is full. This caused coilyco-flight-deck/ward#1597 to exit before launching the agent. ## Desired outcome Recover enough host space for fresh Ward engineers to launch, and make the failure easier to diagnose or self-heal if repo/cache growth fills the same filesystem again. ## Acceptance - Identify the disk consumer on the host filesystem backing /scratch and /gitcache. - Clean only Ward-owned stale/cache state or document the exact operator action if cleanup is unsafe from an engineer container. - Verify a fresh headless engineer can pass the pre-agent resource check after cleanup. - Redispatch or explicitly unblock coilyco-flight-deck/ward#1597 once launch capacity is restored. - If a code change is warranted, keep it focused on preflight diagnostics or cache cleanup safety, then land through the normal Ward workflow. ## Context This was surfaced during the autonomous Ward burndown. It is now a lane blocker for new work, so prioritize diagnosis and conservative recovery over unrelated refactors.

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-28 09:23:39 +00:00

coilyco-ops referenced this issue from a commit

2026-07-28 09:32:24 +00:00

Improve scratch space failure diagnostics

coilyco-ops closed this issue

2026-07-28 09:32:24 +00:00

coilyco-ops commented

2026-07-28 09:40:28 +00:00

Author

Member

WARD-WORKFLOW: done ✅

details

workflow: merge-remote-main; review summary: in-container review gate skipped explicitly because the temporary ward default is pending brokered QA.

summary: landed 6e040e74 on main; low-space scratch failures now include backing mount details, explain the empty-scratch/full-backing-filesystem case, and name conservative Ward-owned cache inspection/cleanup targets. Current disk state showed /scratch and /gitcache healthy with ~65G free; visible Ward-owned consumers were /gitcache/config-bundle (~618M), /gitcache/surface-scratch (~239M), and repo mirrors. No cleanup was safe or needed from this container. coilyco-flight-deck/ward#1597 was already redispatched and reservation-held after capacity returned.

validation: ward exec test, ward exec vet, ward exec lint, ward exec build, pre-commit. Post-push aosguard verification was blocked by missing SSM/IMDS credentials; public Forgejo API showed my promote run #2498 was superseded, and latest main still contains 6e040e74 with promote run #2500 running.

felt: small and direct. The main surprise was that the host space had already recovered and the repo was busy enough to supersede CI runs while I was polling.

confidence: high on the code path and local validation; medium on release-run terminal state because the active run was still running after supersession.

follow-ups: add an explicit broker-side cache reaper only if future evidence shows a specific stale Ward-owned path grows large enough to justify automatic deletion.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> workflow: merge-remote-main; review summary: in-container review gate skipped explicitly because the temporary ward default is pending brokered QA. summary: landed `6e040e74` on `main`; low-space scratch failures now include backing mount details, explain the empty-scratch/full-backing-filesystem case, and name conservative Ward-owned cache inspection/cleanup targets. Current disk state showed `/scratch` and `/gitcache` healthy with ~65G free; visible Ward-owned consumers were `/gitcache/config-bundle` (~618M), `/gitcache/surface-scratch` (~239M), and repo mirrors. No cleanup was safe or needed from this container. coilyco-flight-deck/ward#1597 was already redispatched and reservation-held after capacity returned. validation: `ward exec test`, `ward exec vet`, `ward exec lint`, `ward exec build`, pre-commit. Post-push `aosguard` verification was blocked by missing SSM/IMDS credentials; public Forgejo API showed my promote run #2498 was superseded, and latest main still contains `6e040e74` with promote run #2500 running. felt: small and direct. The main surprise was that the host space had already recovered and the repo was busy enough to supersede CI runs while I was polling. confidence: high on the code path and local validation; medium on release-run terminal state because the active run was still running after supersession. follow-ups: add an explicit broker-side cache reaper only if future evidence shows a specific stale Ward-owned path grows large enough to justify automatic deletion. </details>

coilyco-ops commented

2026-07-28 09:40:41 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-ward-1603 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-ward-1603` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#1603

No description provided.

Rows
Columns