fix(forgejo): recycle runner scratch by label, not a name list #770

Merged

coilysiren merged 1 commit from aos/claude/runner-recycle-selector into main

2026-08-09 06:19:43 +00:00

coilyco-ops commented

2026-08-09 06:14:44 +00:00

Owner

Why

Replaces #769, which diagnosed the same incident correctly but paid too much for it.

kai-server root filesystem hit 89.6 percent because the scoped deploy and publisher runner pools were never in the recycle inventory. They carry the same 24Gi emptyDir docker-lib scratch and the same start-time wipe as the general pools, so their scratch grew for the whole pod lifetime. Measured from the kubelet summary API: 96.44 GiB across 14 pods that had not restarted in 8 days, roughly 24 percent of everything on the disk, all of it disposable.

What

Declare the contract once and select on it, instead of enumerating pools.

Each scratch-bearing base carries coilyco.io/runner-scratch: recycle on its StatefulSet metadata.labels. Scoped overlays inherit it through nameSuffix.
The CronJob runs one rollout restart statefulset --selector coilyco.io/runner-scratch=recycle.
The ser8 overlay drops its 89-line patch set and consumes the base unpatched. Each cluster resolves its own runners.
The Role trades resourceNames for namespaced list.

A new deploy, publisher, or general pool is recycled the day it lands, with no edit here.

Why not #769

That change added 13 hand-copied 15-line container blocks and 13 resourceNames. Its body says targets become derived, but the derivation lives only in the test, which then enforces the hand-maintained list. Net effect after merge:

rancher/kubectl digest at 21 copies across two files
every runner enumerated twice, in two separate cluster inventories
every future deploy service becomes a required 16-line copy-paste, gated by infra CI

This branch instead: 97 lines off the recycle manifest, 89 off the ser8 overlay, digest appears once, zero names anywhere.

The list also tracked an inventory the coilyco-bridge/deploy repo owns. Under the config-placement rule, infra should not hold a copy of it. The selector removes the copy without moving the mechanism, which correctly stays in infra.

Coverage

Identical in effect to #769. Verified by rendering deploy/_flux/apps and both ser8 roots:

17 labelled StatefulSets on kai-server - 3 general, 1 image-build, 13 deploy/publisher
4 on ser8 - canary plus three organization pools
forgejo-runner-tap-writer mounts no scratch and stays unlabelled

Rendered diff against main is exactly three things: the RBAC verb change, 17 label insertions, and the container list collapsing 4 to 1. No runner spec is otherwise touched, and no spec.selector changes.

RBAC tradeoff

This is the one place #769 was stronger, so stating it plainly. Kubernetes cannot scope a list by name, and a selector needs list, so resourceNames had to go. The Role stays namespaced to forgejo, whose only other StatefulSet is forgejo-db. Nothing but this CronJob binds the ServiceAccount, the image is distroless with no shell, and the argument vector is fixed at apply time, so the widened verb set is not reachable as a runtime choice.

Test

The recycle test now asserts the contract rather than a copy of the list: docker-lib scratch and the recycle label must imply each other in both directions, across the rendered scoped aggregate and both ser8 roots. That catches the drift the old list could not, a runner gaining scratch without the label, and the inverse, a labelled runner restarted for nothing.

Verification

ward exec forgejo-runner-recycle-test static stage passes: forgejo runner recycle manifest: ok
pre-commit run --all-files passes
The image-entrypoint stage fails on this arm64 macOS host with Exec format error on the amd64 kubectl. Same failure as unmodified main, in validate_image_entrypoint, which this diff does not touch. CI on the amd64 runner exercises it.

Not included

forgejo-runner-deploy-sirens-discord-ops holds ~23 GB and has no manifest in any repo. It carries Flux apps labels, so it was applied once and its source was removed without a prune. Unlabelled here, so still unrecycled. Needs a separate retirement decision, same conclusion #769 reached.
The aggregate ceiling. 14 deploy runners times 24Gi sizeLimit is 336 GiB permitted on a 479.55 GiB filesystem. Nightly recycle bounds the typical case, not a bad day.

Kai reclaimed the 96 GiB by hand before this branch, so the immediate pressure is already resolved. This makes it not recur.

## Why Replaces #769, which diagnosed the same incident correctly but paid too much for it. kai-server root filesystem hit **89.6 percent** because the scoped **deploy** and **publisher** runner pools were never in the recycle inventory. They carry the same 24Gi `emptyDir` `docker-lib` scratch and the same start-time wipe as the general pools, so their scratch grew for the whole pod lifetime. Measured from the kubelet summary API: **96.44 GiB across 14 pods that had not restarted in 8 days**, roughly 24 percent of everything on the disk, all of it disposable. ## What Declare the contract once and select on it, instead of enumerating pools. - Each scratch-bearing base carries `coilyco.io/runner-scratch: recycle` on its StatefulSet `metadata.labels`. Scoped overlays inherit it through `nameSuffix`. - The CronJob runs one `rollout restart statefulset --selector coilyco.io/runner-scratch=recycle`. - The ser8 overlay drops its 89-line patch set and consumes the base unpatched. Each cluster resolves its own runners. - The `Role` trades `resourceNames` for namespaced `list`. A new deploy, publisher, or general pool is recycled the day it lands, with no edit here. ## Why not #769 That change added 13 hand-copied 15-line container blocks and 13 `resourceNames`. Its body says targets become derived, but the derivation lives only in the test, which then *enforces* the hand-maintained list. Net effect after merge: - `rancher/kubectl` digest at **21 copies** across two files - every runner enumerated **twice**, in two separate cluster inventories - every future deploy service becomes a required 16-line copy-paste, gated by infra CI This branch instead: **97 lines off** the recycle manifest, **89 off** the ser8 overlay, digest appears **once**, zero names anywhere. The list also tracked an inventory the `coilyco-bridge/deploy` repo owns. Under the config-placement rule, infra should not hold a copy of it. The selector removes the copy without moving the mechanism, which correctly stays in infra. ## Coverage Identical in effect to #769. Verified by rendering `deploy/_flux/apps` and both ser8 roots: - **17 labelled StatefulSets on kai-server** - 3 general, 1 image-build, 13 deploy/publisher - **4 on ser8** - canary plus three organization pools - `forgejo-runner-tap-writer` mounts no scratch and stays unlabelled Rendered diff against `main` is exactly three things: the RBAC verb change, 17 label insertions, and the container list collapsing 4 to 1. No runner `spec` is otherwise touched, and no `spec.selector` changes. ## RBAC tradeoff This is the one place #769 was stronger, so stating it plainly. Kubernetes cannot scope a `list` by name, and a selector needs `list`, so `resourceNames` had to go. The Role stays namespaced to `forgejo`, whose only other StatefulSet is `forgejo-db`. Nothing but this CronJob binds the ServiceAccount, the image is distroless with no shell, and the argument vector is fixed at apply time, so the widened verb set is not reachable as a runtime choice. ## Test The recycle test now asserts the **contract** rather than a copy of the list: `docker-lib` scratch and the recycle label must imply each other **in both directions**, across the rendered scoped aggregate and both ser8 roots. That catches the drift the old list could not, a runner gaining scratch without the label, and the inverse, a labelled runner restarted for nothing. ## Verification - `ward exec forgejo-runner-recycle-test` static stage passes: `forgejo runner recycle manifest: ok` - `pre-commit run --all-files` passes - The image-entrypoint stage fails on this arm64 macOS host with `Exec format error` on the amd64 kubectl. Same failure as unmodified `main`, in `validate_image_entrypoint`, which this diff does not touch. CI on the amd64 runner exercises it. ## Not included - `forgejo-runner-deploy-sirens-discord-ops` holds ~23 GB and has **no manifest in any repo**. It carries Flux `apps` labels, so it was applied once and its source was removed without a prune. Unlabelled here, so still unrecycled. Needs a separate retirement decision, same conclusion #769 reached. - The aggregate ceiling. 14 deploy runners times 24Gi `sizeLimit` is 336 GiB permitted on a 479.55 GiB filesystem. Nightly recycle bounds the typical case, not a bad day. Kai reclaimed the 96 GiB by hand before this branch, so the immediate pressure is already resolved. This makes it not recur.

coilyco-ops added 1 commit

2026-08-09 06:14:44 +00:00

fix(forgejo): recycle runner scratch by label, not a name list

TruffleHog / Scan for secrets (pull_request) Successful in 5s

Details

CI / lint (pull_request) Failing after 12s

Details

2072a8ae22

kai-server root filesystem hit 89.6 percent because the scoped deploy and
publisher runner pools were never in the recycle inventory. They carry the
same 24Gi emptyDir docker-lib scratch and the same start-time wipe as the
general pools, so their scratch grew for the whole pod lifetime: 96 GiB
across 14 pods that had not restarted in 8 days.

Rather than enumerate those pools, declare the contract once and select on
it. Each scratch-bearing base carries coilyco.io/runner-scratch: recycle,
scoped overlays inherit it through nameSuffix, and the CronJob runs a single
rollout restart against the label. A new deploy, publisher, or general pool
is recycled the day it lands with no edit to the recycle manifest.

This resolves the same incident as #769 without that approach's cost. That
change added 13 hand-copied container blocks and 13 resourceNames, taking
the digest to 21 copies across two files and making every future runner a
required 16-line copy-paste enforced by a test. Here the manifest drops 97
lines, the ser8 overlay drops its 89-line patch set and consumes the base
unpatched, and the digest appears once.

Coverage is unchanged in intent and identical in effect: 17 labelled
StatefulSets on kai-server and 4 on ser8. forgejo-runner-tap-writer mounts
no scratch and stays unlabelled.

The Role trades resourceNames for namespaced list, because Kubernetes cannot
scope a list by name and a selector needs one. The Role stays namespaced to
forgejo, whose only other StatefulSet is forgejo-db. Nothing else binds the
ServiceAccount and the shell-free argument vector is fixed at apply time.

The recycle test now asserts the contract rather than a copy of the list:
docker-lib scratch and the recycle label must imply each other in both
directions, across the rendered scoped aggregate and both ser8 roots. That
catches the drift the old list could not, a runner gaining scratch without
the label.

Not included: forgejo-runner-deploy-sirens-discord-ops holds ~23 GB and has
no manifest in any repo, so it is unlabelled and unrecycled. It needs a
separate retirement decision. The aggregate 24Gi-times-N sizeLimit ceiling
also still exceeds the disk and is untouched here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-09 06:14:44 +00:00

coilyco-ops referenced this pull request

2026-08-09 06:16:05 +00:00

fix(forgejo): recycle deploy and publisher runner scratch nightly #769