fix(forgejo): recycle deploy and publisher runner scratch nightly

coilyco-ops commented

2026-08-09 05:46:34 +00:00

Owner

Why

kai-server root filesystem is at 89.6 percent (461.6 GB of 514.9 GB), past the 85 percent critical line. Kubelet has been failing image garbage collection every five minutes, wanting 49.7 GB and finding ~0 bytes eligible, because every image is in use. The pressure is data, not images.

The measured cause: forgejo-runner-recycle covered only the three general pools plus the Flight Deck image-build runner. The scoped deploy and publisher pools use the identical 24Gi-capped emptyDir docker-lib scratch and the identical start-time wipe, but were never in the recycle inventory, so their scratch grew for the whole pod lifetime.

Pod ephemeral storage on kai-server, from the kubelet summary API:

forgejo-runner-deploy-factory-game-v3 - 23.92 GB, against its own 24Gi sizeLimit
forgejo-runner-deploy-sirens-discord-ops - 23.33 GB
forgejo-runner-deploy-sirens-echo - 15.85 GB
forgejo-runner-deploy-eco-app - 14.99 GB
forgejo-runner-deploy-galaxy-gen - 14.57 GB
forgejo-runner-deploy-ward-mcp - 5.74 GB
forgejo-runner-deploy-scoped - 3.06 GB

That is 101 GB, 97 percent of all pod ephemeral storage on the node. factory-game-v3 is close enough to its sizeLimit to be evicted.

What

Derive recycle targets from the forgejo-runner-deploy.yml and forgejo-runner-publisher.yml bases too, so the inventory follows the scoped overlays rather than a hand-maintained list. Targets go from 4 to 17.
Assert the emptyDir scratch contract for both newly covered bases.
Update the docs/FEATURES.md coverage boundary.

The ser8 overlay replaces both resourceNames and the container list, so ser8 coverage is unchanged.

Verification

ward exec forgejo-runner-recycle-test static validation passes: forgejo runner recycle manifest: ok. The image-entrypoint stage cannot run on an arm64 macOS host without Docker and fails there identically on unmodified main, so it is not a regression from this change.
pre-commit run --all-files passes, including the Forgejo runner recycle guard and the Forgejo storage lifecycle guard.

Not included

forgejo-runner-deploy-sirens-discord-ops holds 23.33 GB but has no manifest anywhere in this repo. It carries Flux apps labels, so it was applied once and its source was later removed without a prune. It is excluded from this change because the recycle inventory is derived from tracked overlays, and because restarting a workload with no tracked manifest has no rollback artifact. It needs a separate retirement decision.

## Why kai-server root filesystem is at **89.6 percent** (461.6 GB of 514.9 GB), past the 85 percent critical line. Kubelet has been failing image garbage collection every five minutes, wanting 49.7 GB and finding ~0 bytes eligible, because every image is in use. The pressure is data, not images. The measured cause: `forgejo-runner-recycle` covered only the three general pools plus the Flight Deck image-build runner. The scoped **deploy** and **publisher** pools use the identical 24Gi-capped `emptyDir` `docker-lib` scratch and the identical start-time wipe, but were never in the recycle inventory, so their scratch grew for the whole pod lifetime. Pod ephemeral storage on kai-server, from the kubelet summary API: - `forgejo-runner-deploy-factory-game-v3` - 23.92 GB, against its own 24Gi `sizeLimit` - `forgejo-runner-deploy-sirens-discord-ops` - 23.33 GB - `forgejo-runner-deploy-sirens-echo` - 15.85 GB - `forgejo-runner-deploy-eco-app` - 14.99 GB - `forgejo-runner-deploy-galaxy-gen` - 14.57 GB - `forgejo-runner-deploy-ward-mcp` - 5.74 GB - `forgejo-runner-deploy-scoped` - 3.06 GB That is 101 GB, 97 percent of all pod ephemeral storage on the node. factory-game-v3 is close enough to its `sizeLimit` to be evicted. ## What - Derive recycle targets from the `forgejo-runner-deploy.yml` and `forgejo-runner-publisher.yml` bases too, so the inventory follows the scoped overlays rather than a hand-maintained list. Targets go from 4 to 17. - Assert the `emptyDir` scratch contract for both newly covered bases. - Update the `docs/FEATURES.md` coverage boundary. The ser8 overlay replaces both `resourceNames` and the container list, so ser8 coverage is unchanged. ## Verification - `ward exec forgejo-runner-recycle-test` static validation passes: `forgejo runner recycle manifest: ok`. The image-entrypoint stage cannot run on an arm64 macOS host without Docker and fails there identically on unmodified `main`, so it is not a regression from this change. - `pre-commit run --all-files` passes, including the Forgejo runner recycle guard and the Forgejo storage lifecycle guard. ## Not included `forgejo-runner-deploy-sirens-discord-ops` holds 23.33 GB but has **no manifest anywhere in this repo**. It carries Flux `apps` labels, so it was applied once and its source was later removed without a prune. It is excluded from this change because the recycle inventory is derived from tracked overlays, and because restarting a workload with no tracked manifest has no rollback artifact. It needs a separate retirement decision.

coilyco-ops added 1 commit

2026-08-09 05:46:34 +00:00

fix(forgejo): recycle deploy and publisher runner scratch nightly

TruffleHog / Scan for secrets (pull_request) Successful in 6s

Details

CI / lint (pull_request) Successful in 1m1s

Details

362b8b769a

The forgejo-runner-recycle CronJob covered only the three general pools and
the Flight Deck image-build runner. The scoped deploy and publisher pools use
the same 24Gi-capped emptyDir docker-lib scratch and the same start-time wipe,
but were never recycled, so their scratch grew for the life of the pod.

On kai-server that reached 101 GB of pod ephemeral storage across seven
runners and drove the root filesystem to 89.6 percent, past the 85 percent
critical line. Kubelet then failed image garbage collection every five minutes
because no image was eligible to free. forgejo-runner-deploy-factory-game-v3
had reached 23.92 GB against its own 24Gi sizeLimit and was close to eviction.

Derive the recycle targets from the deploy and publisher bases as well, so the
inventory follows the overlays instead of being maintained by hand, and assert
the emptyDir scratch contract for both bases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-09 05:46:34 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-09 06:14:09 +00:00

fix(forgejo): recycle runner scratch by label, not a name list

coilyco-ops referenced this pull request

2026-08-09 06:14:44 +00:00

fix(forgejo): recycle runner scratch by label, not a name list #770

coilyco-ops commented

2026-08-09 06:16:05 +00:00

Author

Owner

Closing in favor of #770, which resolves the same incident.

The diagnosis here is correct and #770 keeps it: the scoped deploy and publisher pools carry the same 24Gi emptyDir scratch contract as the general pools and were never in the recycle inventory. The measurements match an independent read, and the sirens-discord-ops exclusion reasoning was right.

What changed is the mechanism. This PR's body says targets become derived from the scoped overlays, but the derivation lives only in scripts/test-forgejo-runner-recycle.py. The manifest still carries 13 hand-copied 15-line container blocks and 13 resourceNames, and the test then enforces that hand-maintained list. After merge the rancher/kubectl digest would sit at 21 copies across two files, every runner would be enumerated twice in two cluster inventories, and every future coilyco-bridge/deploy service would become a required 16-line copy-paste gated by infra CI.

#770 labels each scratch-bearing base with coilyco.io/runner-scratch: recycle and selects on it. Same 17 kai-server targets and 4 ser8 targets, verified by rendering. The recycle manifest loses 97 lines, the ser8 overlay loses its whole 89-line patch set and consumes the base unpatched, and the digest appears once. The test asserts scratch and label imply each other in both directions, which catches the drift a name list cannot.

Two things carried forward unchanged: sirens-discord-ops still needs a retirement decision, and the aggregate 24Gi-times-N ceiling still exceeds the disk.

Thanks for finding this one. The scratch measurement is what made the fix obvious.

Closing in favor of #770, which resolves the same incident. The diagnosis here is correct and #770 keeps it: the scoped deploy and publisher pools carry the same 24Gi `emptyDir` scratch contract as the general pools and were never in the recycle inventory. The measurements match an independent read, and the `sirens-discord-ops` exclusion reasoning was right. What changed is the mechanism. This PR's body says targets become derived from the scoped overlays, but the derivation lives only in `scripts/test-forgejo-runner-recycle.py`. The manifest still carries 13 hand-copied 15-line container blocks and 13 `resourceNames`, and the test then enforces that hand-maintained list. After merge the `rancher/kubectl` digest would sit at 21 copies across two files, every runner would be enumerated twice in two cluster inventories, and every future `coilyco-bridge/deploy` service would become a required 16-line copy-paste gated by infra CI. #770 labels each scratch-bearing base with `coilyco.io/runner-scratch: recycle` and selects on it. Same 17 kai-server targets and 4 ser8 targets, verified by rendering. The recycle manifest loses 97 lines, the ser8 overlay loses its whole 89-line patch set and consumes the base unpatched, and the digest appears once. The test asserts scratch and label imply each other in both directions, which catches the drift a name list cannot. Two things carried forward unchanged: `sirens-discord-ops` still needs a retirement decision, and the aggregate 24Gi-times-N ceiling still exceeds the disk. Thanks for finding this one. The scratch measurement is what made the fix obvious.

coilyco-ops closed this pull request

2026-08-09 06:16:14 +00:00

coilyco-ops referenced this pull request

2026-08-09 06:26:11 +00:00

Retire the untracked forgejo-runner-deploy-sirens-discord-ops runner #771

coilyco-ops referenced this pull request

2026-08-09 06:33:04 +00:00

lint.yml fails on every pull_request run while passing on main #773