Make Forgejo runner scratch classification and recycle busy-aware #650

New issue

Closed

opened 2026-07-26 05:55:54 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-26 05:55:54 +00:00

Member

Finding

On 2026-07-25, kai-server root pressure reached 83.25% with 80.3 GiB available. The bounded node and PVC views showed Forgejo-related storage but did not expose the acute owner because general-runner Docker scratch lives in emptyDir, not a PVC.

The guarded kubelet stats summary established the missing split:

The four general runners held 41.2 GiB of docker-lib emptyDir scratch only 13 hours after the last successful recycle.
The image-build runner still owned an active agentic-os publish, so restarting the full existing recycle target set would have interrupted live work.
The latest three package-upload cleanup Jobs each reported 1.30 GiB remaining and zero eligible files.
Package versions, upload scratch, Forgejo repositories, LFS, attachments, PostgreSQL, and game data were not the measured acute owner.

The operator verified all four general runners idle through their task and cleanup logs, then restarted only statefulset/forgejo-runner. The four replacement pods returned 2/2 Running. Available root space rose from 80.3 GiB to 123.5 GiB and pressure fell from 83.25% to 74.25%, reclaiming about 43.2 GiB. The operator left the active image-build runner and every durable data surface untouched.

Learning

The current root and local-path classifiers miss material runner emptyDir use.
A nightly recycle is not sufficient during high CI churn. General-runner scratch regrew above 40 GiB in roughly half a day.
The existing recycle targets both the general and image-build StatefulSets, but current activity can make one safe to recycle and the other unsafe.
Forgejo package pressure, failed-upload scratch, and runner scratch can produce the same root-pressure symptom. The classifier must distinguish them before any cleanup path runs.

Work

Extend the read-only disk classifier to report docker-lib use from kubelet stats, aggregated separately for general, image-build, and deploy runners.
Report each pool's last successful recycle and whether current task activity is known idle, active, or ambiguous.
Fail closed when activity cannot be established. An ambiguous or active pool must never be restarted automatically.
Add an attended operator path that can recycle the idle general pool without coupling it to the image-build pool. Keep the deploy runner outside that path unless separate evidence and policy add it later.
Choose a bounded scratch threshold and alert cadence that catches this growth before root pressure reaches 80%.
Preserve before-and-after root bytes, available bytes, pressure percent, scratch bytes, pod readiness, and affected pool in the audit output.
Update the disk-pressure, runner-recycle, and Forgejo storage runbooks with this incident shape.

Acceptance

A read-only report distinguishes general-runner emptyDir, image-build PVC, deploy-runner emptyDir, package-upload scratch, and durable Forgejo data.
A fixture reproducing 41 GiB of general-runner scratch identifies that pool as the owner without misclassifying the package store.
Active and ambiguous runner fixtures block recycle.
An attended general-pool recycle touches only statefulset/forgejo-runner, waits for all four pods to become Ready, and records reclaimed bytes.
The implementation performs no package deletion, PVC deletion, direct filesystem deletion, database mutation, or Forgejo restart.
Monitoring warns early enough that a blocked recycle still leaves operator runway before the 85% critical threshold.
Repository validation passes.

#596
#617
#618
#619
#642
#646
#600
coilyco-bridge/deploy#196

## Finding On 2026-07-25, `kai-server` root pressure reached 83.25% with 80.3 GiB available. The bounded node and PVC views showed Forgejo-related storage but did not expose the acute owner because general-runner Docker scratch lives in `emptyDir`, not a PVC. The guarded kubelet stats summary established the missing split: * The four general runners held 41.2 GiB of `docker-lib` emptyDir scratch only 13 hours after the last successful recycle. * The image-build runner still owned an active `agentic-os` publish, so restarting the full existing recycle target set would have interrupted live work. * The latest three package-upload cleanup Jobs each reported 1.30 GiB remaining and zero eligible files. * Package versions, upload scratch, Forgejo repositories, LFS, attachments, PostgreSQL, and game data were not the measured acute owner. The operator verified all four general runners idle through their task and cleanup logs, then restarted only `statefulset/forgejo-runner`. The four replacement pods returned `2/2 Running`. Available root space rose from 80.3 GiB to 123.5 GiB and pressure fell from 83.25% to 74.25%, reclaiming about 43.2 GiB. The operator left the active image-build runner and every durable data surface untouched. ## Learning * The current root and local-path classifiers miss material runner `emptyDir` use. * A nightly recycle is not sufficient during high CI churn. General-runner scratch regrew above 40 GiB in roughly half a day. * The existing recycle targets both the general and image-build StatefulSets, but current activity can make one safe to recycle and the other unsafe. * Forgejo package pressure, failed-upload scratch, and runner scratch can produce the same root-pressure symptom. The classifier must distinguish them before any cleanup path runs. ## Work * Extend the read-only disk classifier to report `docker-lib` use from kubelet stats, aggregated separately for general, image-build, and deploy runners. * Report each pool's last successful recycle and whether current task activity is known idle, active, or ambiguous. * Fail closed when activity cannot be established. An ambiguous or active pool must never be restarted automatically. * Add an attended operator path that can recycle the idle general pool without coupling it to the image-build pool. Keep the deploy runner outside that path unless separate evidence and policy add it later. * Choose a bounded scratch threshold and alert cadence that catches this growth before root pressure reaches 80%. * Preserve before-and-after root bytes, available bytes, pressure percent, scratch bytes, pod readiness, and affected pool in the audit output. * Update the disk-pressure, runner-recycle, and Forgejo storage runbooks with this incident shape. ## Acceptance * A read-only report distinguishes general-runner emptyDir, image-build PVC, deploy-runner emptyDir, package-upload scratch, and durable Forgejo data. * A fixture reproducing 41 GiB of general-runner scratch identifies that pool as the owner without misclassifying the package store. * Active and ambiguous runner fixtures block recycle. * An attended general-pool recycle touches only `statefulset/forgejo-runner`, waits for all four pods to become Ready, and records reclaimed bytes. * The implementation performs no package deletion, PVC deletion, direct filesystem deletion, database mutation, or Forgejo restart. * Monitoring warns early enough that a blocked recycle still leaves operator runway before the 85% critical threshold. * Repository validation passes. ## Related * #596 * #617 * #618 * #619 * #642 * #646 * #600 * `coilyco-bridge/deploy#196`

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-29 03:46:31 +00:00

coilyco-ops closed this issue

2026-07-29 06:04:38 +00:00