coilyco-flight-deck/infrastructure

Watch

Runner recycle misses two consumers: stranded docker-lib PVC and an unlabelled discord-ops runner #868

New issue

Open

opened 2026-08-19 05:22:04 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-19 05:22:04 +00:00

Owner

Found while relieving a live disk emergency on kai-server at 2026-08-19T05:17Z, where kubelet was reporting ImageGCFailed and could not free the 27.7 GB it wanted.

deploy/forgejo-runner-recycle.yml selects on coilyco.io/runner-scratch=recycle. Two scratch consumers sit outside that selector, for two different reasons.

1. `docker-lib-forgejo-runner-build-0` is stranded, 6.95 GB

forgejo-runner-build is spec.replicas: 0 and still declares a docker-lib volumeClaimTemplate. The PVC survives at roughly 6.95 GB with no pod bound to it.

This is not label drift. Adding the recycle label would change nothing, because the recycle action is rollout restart and there is no pod to restart. The scratch is only wiped on pod start, so a scaled-to-zero runner holds its cache indefinitely.

Either the runner is retired, in which case the PVC should go with it, or it is dormant and expected to return, in which case the cache is stale anyway. Deleting the PVC is destructive and irreversible, so it wants an attended operator and a current backup, not a nightly job.

2. `forgejo-runner-deploy-sirens-discord-ops` is genuinely unlabelled

spec.replicas: 1, running a dind sidecar, no coilyco.io/runner-scratch label. Its emptyDir scratch has never been recycled. Its pod had been up since 2026-08-19T01:16Z and survived the manual sweep for that reason.

The manifest header says scoped overlays inherit the label through nameSuffix, so this one appears to have been built off a path that does not. Worth checking whether the overlay is the odd one out or whether the inheritance claim is weaker than the comment states.

forgejo-runner-tap-writer-scoped is also unlabelled and that is correct: it mounts no scratch, exactly as the header documents.

Evidence

kubectl get statefulset -n forgejo --show-labels
kubectl get statefulset forgejo-runner-build -n forgejo \
  -o custom-columns='R:.spec.replicas,V:.spec.volumeClaimTemplates[*].metadata.name'

Filesystem before and after a manual sweep of the labelled set: 84.74 percent used with 1.31 GB to the critical threshold, then 77.56 percent used with 38.30 GB to critical. Roughly 37 GB reclaimed, and the two consumers above contributed none of it.

Acceptance

A decision recorded on the stranded docker-lib-forgejo-runner-build-0 PVC, and the reclaim measured if it is deleted.
forgejo-runner-deploy-sirens-discord-ops either carries the label or has a comment saying why it must not.
A check that catches a scratch-mounting StatefulSet with no recycle label, so the next one does not need an incident to surface it.

Found while relieving a live disk emergency on kai-server at 2026-08-19T05:17Z, where kubelet was reporting `ImageGCFailed` and could not free the 27.7 GB it wanted. `deploy/forgejo-runner-recycle.yml` selects on `coilyco.io/runner-scratch=recycle`. Two scratch consumers sit outside that selector, for two different reasons. ## 1. `docker-lib-forgejo-runner-build-0` is stranded, 6.95 GB `forgejo-runner-build` is `spec.replicas: 0` and still declares a `docker-lib` volumeClaimTemplate. The PVC survives at roughly 6.95 GB with no pod bound to it. This is not label drift. Adding the recycle label would change nothing, because the recycle action is `rollout restart` and there is no pod to restart. The scratch is only wiped on pod start, so a scaled-to-zero runner holds its cache indefinitely. Either the runner is retired, in which case the PVC should go with it, or it is dormant and expected to return, in which case the cache is stale anyway. Deleting the PVC is destructive and irreversible, so it wants an attended operator and a current backup, not a nightly job. ## 2. `forgejo-runner-deploy-sirens-discord-ops` is genuinely unlabelled `spec.replicas: 1`, running a `dind` sidecar, no `coilyco.io/runner-scratch` label. Its emptyDir scratch has never been recycled. Its pod had been up since 2026-08-19T01:16Z and survived the manual sweep for that reason. The manifest header says scoped overlays inherit the label through `nameSuffix`, so this one appears to have been built off a path that does not. Worth checking whether the overlay is the odd one out or whether the inheritance claim is weaker than the comment states. `forgejo-runner-tap-writer-scoped` is also unlabelled and that is correct: it mounts no scratch, exactly as the header documents. ## Evidence kubectl get statefulset -n forgejo --show-labels kubectl get statefulset forgejo-runner-build -n forgejo \ -o custom-columns='R:.spec.replicas,V:.spec.volumeClaimTemplates[*].metadata.name' Filesystem before and after a manual sweep of the labelled set: 84.74 percent used with 1.31 GB to the critical threshold, then 77.56 percent used with 38.30 GB to critical. Roughly 37 GB reclaimed, and the two consumers above contributed none of it. ## Acceptance * A decision recorded on the stranded `docker-lib-forgejo-runner-build-0` PVC, and the reclaim measured if it is deleted. * `forgejo-runner-deploy-sirens-discord-ops` either carries the label or has a comment saying why it must not. * A check that catches a scratch-mounting StatefulSet with no recycle label, so the next one does not need an incident to surface it.

coilyco-ops referenced this issue

2026-08-19 11:42:26 +00:00

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:07 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:58 +00:00

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops referenced this issue

2026-08-29 04:21:22 +00:00

Move disk-heavy scoped Forgejo runners to ser8 #693

coilyco-ops commented

2026-08-29 04:21:52 +00:00

Author

Owner

Both halves still live, ten days on. Verified 2026-08-29 ~04:20Z. This issue now owns the whole unmounted-PVC inventory.

1. `docker-lib-forgejo-runner-build-0` is still stranded

pvc_phase: Bound, pv_phase: Bound, pod_mounts: [], reclaim_policy: Delete, requested 10 GiB. Measured 5,995,282,432 bytes and that is a lower bound, because the scan timed out at 30,449 entries.

Filed at 6.95 GB, measured at 5.99 GB or more today. Nothing has bound to it and nothing has reclaimed it. The reasoning in the body holds unchanged: the recycle action is rollout restart, there is no pod to restart, so the label would still change nothing.

2. `forgejo-runner-deploy-sirens-discord-ops` is still outside the selector

The evidence is now cleaner than a label read, because the recycle itself separates them. Pod ages after the 2026-08-28T16:15Z run:

every labelled deploy runner: 12h2m
forgejo-runner-deploy-sirens-discord-ops-0: 6d3h

Six days without a recycle while twelve siblings were recycled twelve hours ago. forgejo-runner-tap-writer-scoped-0 sits at 9d9h and that remains correct, exactly as the body says, because it mounts no scratch.

3. #846 was closed on half its content, and this is the other half

#846 is titled "kai-server local-path storage is 163 GB, with ~16 GB in unmounted runner PVCs and three PVs stuck Released". It was closed in the #981 sweep on the Released-PV half, which is genuinely fixed: 43 volumes, every one Bound, zero Released.

The ~16 GB unmounted half was never addressed. Measured today across the forgejo namespace, every PVC below is Bound with pod_mounts: []:

docker-lib-forgejo-runner-build-0         5,995,282,432   (lower bound)
data-forgejo-runner-2                     2,048,012,288
data-forgejo-runner-1                     1,956,208,640
data-forgejo-runner-0                     1,951,072,256
data-forgejo-runner-3                     1,921,888,256
data-forgejo-runner-flight-deck-1           719,470,592
data-forgejo-runner-flight-deck-0           614,350,848
data-forgejo-runner-deploy-0                 85,958,656
data-forgejo-runner-bridge-0                 42,893,312
data-forgejo-runner-build-flight-deck-0      42,143,744
data-forgejo-runner-build-0                  42,135,552
                                         --------------
                                         15,419,416,576   (~15.4 GB, lower bound)

The eleven divide into three groups, and they are not one decision:

data-forgejo-runner-{0..3} and -flight-deck-{0,1}, 9.21 GB. These are the scaled-zero general runners the ser8 migration left behind. #693 rollout step 7 explicitly says to retain them until a stated rollback window closes. Not junk, and not this issue's call. Re-anchored on #693.
docker-lib-forgejo-runner-build-0, 5.99 GB. Part 1 above. The single largest reclaim available and the one this issue was filed for.
The five small data-* claims, 255 MB total. Registration state for retired or renamed runners. Cheap to leave, cheap to take.

So the reclaim actually available to an attended operator today is the 5.99 GB in part 1, not the headline 15.4 GB. The rest is either spoken for by #693 or rounding.

Not reopening #846. Its Released-PV finding is genuinely resolved and its 163 GB headline is stale. Its live remainder is the inventory above, which now lives here where the stranded-PVC decision already sat. One issue owns PVC reclaim instead of two.

Acceptance, restated against today

A decision on docker-lib-forgejo-runner-build-0, and the reclaim measured if it is deleted. Deletion is destructive and irreversible, so it wants an attended operator and a current backup. No change made today.
forgejo-runner-deploy-sirens-discord-ops carries the recycle label or a comment saying why it must not.
A check that catches a scratch-mounting StatefulSet with no recycle label. Ten days and one migration later, the gap is still only visible by comparing pod ages by hand.

## Both halves still live, ten days on. Verified 2026-08-29 ~04:20Z. This issue now owns the whole unmounted-PVC inventory. ### 1. `docker-lib-forgejo-runner-build-0` is still stranded `pvc_phase: Bound`, `pv_phase: Bound`, **`pod_mounts: []`**, `reclaim_policy: Delete`, requested 10 GiB. Measured **5,995,282,432 bytes** and that is a lower bound, because the scan timed out at 30,449 entries. Filed at 6.95 GB, measured at 5.99 GB or more today. Nothing has bound to it and nothing has reclaimed it. The reasoning in the body holds unchanged: the recycle action is `rollout restart`, there is no pod to restart, so the label would still change nothing. ### 2. `forgejo-runner-deploy-sirens-discord-ops` is still outside the selector The evidence is now cleaner than a label read, because the recycle itself separates them. Pod ages after the 2026-08-28T16:15Z run: * every labelled deploy runner: **12h2m** * `forgejo-runner-deploy-sirens-discord-ops-0`: **6d3h** Six days without a recycle while twelve siblings were recycled twelve hours ago. `forgejo-runner-tap-writer-scoped-0` sits at 9d9h and that remains correct, exactly as the body says, because it mounts no scratch. ### 3. #846 was closed on half its content, and this is the other half **#846 is titled "kai-server local-path storage is 163 GB, with ~16 GB in unmounted runner PVCs and three PVs stuck Released".** It was closed in the #981 sweep on the Released-PV half, which is genuinely fixed: 43 volumes, every one `Bound`, zero `Released`. **The ~16 GB unmounted half was never addressed.** Measured today across the `forgejo` namespace, every PVC below is `Bound` with `pod_mounts: []`: ``` docker-lib-forgejo-runner-build-0 5,995,282,432 (lower bound) data-forgejo-runner-2 2,048,012,288 data-forgejo-runner-1 1,956,208,640 data-forgejo-runner-0 1,951,072,256 data-forgejo-runner-3 1,921,888,256 data-forgejo-runner-flight-deck-1 719,470,592 data-forgejo-runner-flight-deck-0 614,350,848 data-forgejo-runner-deploy-0 85,958,656 data-forgejo-runner-bridge-0 42,893,312 data-forgejo-runner-build-flight-deck-0 42,143,744 data-forgejo-runner-build-0 42,135,552 -------------- 15,419,416,576 (~15.4 GB, lower bound) ``` The eleven divide into three groups, and they are not one decision: * **`data-forgejo-runner-{0..3}` and `-flight-deck-{0,1}`, 9.21 GB.** These are the scaled-zero general runners the ser8 migration left behind. **#693 rollout step 7 explicitly says to retain them** until a stated rollback window closes. Not junk, and not this issue's call. Re-anchored on #693. * **`docker-lib-forgejo-runner-build-0`, 5.99 GB.** Part 1 above. The single largest reclaim available and the one this issue was filed for. * **The five small `data-*` claims, 255 MB total.** Registration state for retired or renamed runners. Cheap to leave, cheap to take. So the reclaim actually available to an attended operator today is the 5.99 GB in part 1, not the headline 15.4 GB. The rest is either spoken for by #693 or rounding. **Not reopening #846.** Its Released-PV finding is genuinely resolved and its 163 GB headline is stale. Its live remainder is the inventory above, which now lives here where the stranded-PVC decision already sat. One issue owns PVC reclaim instead of two. ### Acceptance, restated against today * A decision on `docker-lib-forgejo-runner-build-0`, and the reclaim measured if it is deleted. Deletion is destructive and irreversible, so it wants an attended operator and a current backup. **No change made today.** * `forgejo-runner-deploy-sirens-discord-ops` carries the recycle label or a comment saying why it must not. * A check that catches a scratch-mounting StatefulSet with no recycle label. Ten days and one migration later, the gap is still only visible by comparing pod ages by hand.

coilyco-ops referenced this issue

2026-08-29 04:22:59 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops referenced this issue

2026-08-29 04:43:32 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops added this to the Forgejo runners milestone

2026-08-29 05:40:51 +00:00