fix(forgejo): cap deploy and publisher scratch at 10Gi #772

Merged

coilysiren merged 1 commit from aos/claude/runner-scratch-10gi into main

2026-08-09 20:19:52 +00:00

coilyco-ops commented

2026-08-09 06:32:12 +00:00

Owner

Why

Fourteen deploy and publisher pools at a 24Gi emptyDir sizeLimit permitted 336 GiB of scratch on a 479.55 GiB filesystem. The disk cannot honour that, so the cap was decorative: the node reached 89.6 percent before any single pool got near its own limit.

The dind ephemeral-storage request of 12Gi was the sharper problem. A request is a scheduling reservation, not a ceiling, so the fleet reserved 168 GiB of node ephemeral storage whether or not it was used.

What

	before	after
`emptyDir` `sizeLimit`	24Gi	10Gi
`dind` ephemeral-storage limit	24Gi	10Gi
`dind` ephemeral-storage request	12Gi	4Gi
permitted across the fleet	336 GiB	140 GiB
reserved across the fleet	168 GiB	56 GiB

Volume cap and container limit are kept equal so both trip at the same point instead of one masking the other. The guard in scripts/check-forgejo-runner-config.py is updated for the deploy and publisher checks only.

Is 10Gi enough

Roughly triple what the busiest pool needs. Now that #770 recycles nightly, growth is bounded to one pod-day rather than a whole pod lifetime. The worst observed rate was factory-game-v3 at 22.28 GiB over 8 days, under 3 GiB per day.

The pools to watch are the build-from-context rollouts, factory-game-v3 and galaxy-gen, which run docker build in the DinD sidecar. If one ever needs more than 10Gi of Docker scratch inside a single day it will be evicted mid-job rather than silently ballooning. That is the intended tradeoff, and the eviction is the signal.

Scope

General pools stay at 24Gi. They share a base with ser8's active runners, a different disk with different headroom, so changing them is a separate decision.

The image-build runner uses a disposable local-path PVC rather than an emptyDir and is untouched.

Rendered deploy/_flux/apps confirms the split: 13 pools at 10Gi, 3 general pools at 24Gi. Thirteen rather than fourteen because forgejo-runner-deploy-sirens-discord-ops has no manifest, tracked in #771.

Rollout note

These are pod-template fields, so Flux applying this rolls all 13 StatefulSets within its 10-minute reconcile window. That interrupts any in-flight deploy job and wipes scratch as a side effect. Worth merging when the deploy lane is quiet.

Verification

pre-commit run --all-files passes, including the Forgejo runner config guard and the storage lifecycle guard
ward exec forgejo-runner-recycle-test static stage passes: forgejo runner recycle manifest: ok
The image-entrypoint stage fails on this arm64 macOS host with Exec format error on the amd64 kubectl, unchanged from main and untouched by this diff

Context

Node is currently at 74.05 percent, status ok, down from 89.65 percent critical, after Kai's attended rollout restart. This change keeps the ceiling from being able to put it back.

## Why Fourteen deploy and publisher pools at a 24Gi `emptyDir` `sizeLimit` permitted **336 GiB** of scratch on a **479.55 GiB** filesystem. The disk cannot honour that, so the cap was decorative: the node reached 89.6 percent before any single pool got near its own limit. The `dind` `ephemeral-storage` **request** of 12Gi was the sharper problem. A request is a scheduling reservation, not a ceiling, so the fleet reserved **168 GiB** of node ephemeral storage whether or not it was used. ## What | | before | after | |---|---|---| | `emptyDir` `sizeLimit` | 24Gi | **10Gi** | | `dind` ephemeral-storage limit | 24Gi | **10Gi** | | `dind` ephemeral-storage request | 12Gi | **4Gi** | | permitted across the fleet | 336 GiB | **140 GiB** | | reserved across the fleet | 168 GiB | **56 GiB** | Volume cap and container limit are kept equal so both trip at the same point instead of one masking the other. The guard in `scripts/check-forgejo-runner-config.py` is updated for the deploy and publisher checks only. ## Is 10Gi enough Roughly triple what the busiest pool needs. Now that #770 recycles nightly, growth is bounded to one pod-day rather than a whole pod lifetime. The worst observed rate was `factory-game-v3` at 22.28 GiB over 8 days, **under 3 GiB per day**. The pools to watch are the build-from-context rollouts, `factory-game-v3` and `galaxy-gen`, which run `docker build` in the DinD sidecar. If one ever needs more than 10Gi of Docker scratch inside a single day it will be evicted mid-job rather than silently ballooning. That is the intended tradeoff, and the eviction is the signal. ## Scope General pools stay at 24Gi. They share a base with ser8's active runners, a different disk with different headroom, so changing them is a separate decision. The image-build runner uses a disposable `local-path` PVC rather than an `emptyDir` and is untouched. Rendered `deploy/_flux/apps` confirms the split: **13 pools at 10Gi, 3 general pools at 24Gi**. Thirteen rather than fourteen because `forgejo-runner-deploy-sirens-discord-ops` has no manifest, tracked in #771. ## Rollout note These are pod-template fields, so Flux applying this rolls all 13 StatefulSets within its 10-minute reconcile window. That interrupts any in-flight deploy job and wipes scratch as a side effect. Worth merging when the deploy lane is quiet. ## Verification - `pre-commit run --all-files` passes, including the Forgejo runner config guard and the storage lifecycle guard - `ward exec forgejo-runner-recycle-test` static stage passes: `forgejo runner recycle manifest: ok` - The image-entrypoint stage fails on this arm64 macOS host with `Exec format error` on the amd64 kubectl, unchanged from `main` and untouched by this diff ## Context Node is currently at **74.05 percent, status ok**, down from 89.65 percent critical, after Kai's attended rollout restart. This change keeps the ceiling from being able to put it back.

coilyco-ops added 1 commit

2026-08-09 06:32:12 +00:00

fix(forgejo): cap deploy and publisher scratch at 10Gi

TruffleHog / Scan for secrets (pull_request) Successful in 5s

Details

CI / lint (pull_request) Failing after 8s

Details

39e56cffa3

Fourteen deploy and publisher pools at a 24Gi emptyDir sizeLimit permitted
336 GiB of scratch on a 479.55 GiB filesystem. The disk cannot honour that,
so the cap was decorative: the node hit 89.6 percent before any single pool
approached its own limit.

The dind ephemeral-storage request of 12Gi was the sharper problem. That is
a scheduling reservation, not a ceiling, so the fleet reserved 168 GiB of
node ephemeral storage whether or not it was used.

Drop the emptyDir sizeLimit and the matching dind ephemeral-storage limit to
10Gi, and the request to 4Gi. Permitted scratch goes 336 GiB to 140 GiB and
reserved goes 168 GiB to 56 GiB. Keeping the volume cap and the container
limit equal means both trip at the same point instead of one masking the
other.

10Gi is roughly triple what the busiest pool needs. Now that #770 recycles
nightly, growth is bounded to one pod-day, and the worst observed rate was
factory-game-v3 at 22.28 GiB over 8 days, under 3 GiB per day.

General pools stay at 24Gi. They are shared with ser8's active runners, a
different disk with different headroom, and are out of scope here.

Refs #771 for the untracked sirens-discord-ops pool, which has no manifest
and so is unaffected by this change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-09 06:32:12 +00:00

coilyco-ops referenced this pull request

2026-08-09 06:33:04 +00:00

lint.yml fails on every pull_request run while passing on main #773