Prevent Forgejo upload staging from exhausting kai-server disk

coilyco-ops commented

2026-07-24 07:35:31 +00:00

Member

Incident

On 2026-07-24, kai-server entered Kubernetes DiskPressure and Forgejo returned 503 because the Forgejo database and application pods could not schedule.

Measured state before remediation:

Root filesystem pressure used was 87.0%, with 66.6 GB available.
K3s local-path PVCs used 257.4 GB.
forgejo-data used about 166 GiB despite a 20 GiB request.
tmp/package-upload held 219 chunked-upload staging files totaling about 65 GiB. Every file was newer than 48 hours.
The build runner Docker PVC used about 56 GiB despite a 10 GiB request. About 42 GiB was BuildKit state.
Git repositories used about 1 GiB. Git history was not the pressure source.

Kai approved emergency removal of only the 171 tmp/package-upload files older than one hour. The agent reclaimed 46.3 GB from that selected set. A separate build-cache reclaim workload released additional space during recovery.

Verified recovery:

Root filesystem pressure used fell to 73.2%.
Kubernetes reported DiskPressure=False and removed the node taint.
Forgejo Postgres became ready.
Forgejo became 1/1 Running after the recovery image-pull surge cleared.
The public /api/healthz endpoint returned HTTP 200.

Problem

The local-path PVC requests do not enforce filesystem quotas. Forgejo chunked-upload staging and the persistent build-runner Docker cache can therefore exceed their requested sizes and exhaust the node. The upload staging tree accumulated tens of gigabytes within hours, matching three observed outages over two days.

Required follow-up

Identify which release or package-upload paths leave chunked uploads behind and determine whether cancellation, timeout, retry, or concurrent publication causes the leak.
Add a reviewed, concurrency-safe cleanup path for abandoned tmp/package-upload data. Normal operations must not depend on manual PVC file deletion.
Define package owner and per-upload limits or retention that fit the node budget.
Replace unbounded persistent BuildKit growth with a supported, measured Docker or BuildKit reclamation policy.
Confirm the disk-pressure watcher alerted before the outage and close any notification gap.
Update the Forgejo PVC retention and disk-pressure runbooks with the managed remediation.
Verify the change live by demonstrating bounded staging and runner-cache growth, root pressure below 85%, intact Forgejo data, and a healthy public edge.

## Incident On 2026-07-24, `kai-server` entered Kubernetes `DiskPressure` and Forgejo returned 503 because the Forgejo database and application pods could not schedule. Measured state before remediation: * Root filesystem pressure used was 87.0%, with 66.6 GB available. * K3s local-path PVCs used 257.4 GB. * `forgejo-data` used about 166 GiB despite a 20 GiB request. * `tmp/package-upload` held 219 chunked-upload staging files totaling about 65 GiB. Every file was newer than 48 hours. * The build runner Docker PVC used about 56 GiB despite a 10 GiB request. About 42 GiB was BuildKit state. * Git repositories used about 1 GiB. Git history was not the pressure source. Kai approved emergency removal of only the 171 `tmp/package-upload` files older than one hour. The agent reclaimed 46.3 GB from that selected set. A separate build-cache reclaim workload released additional space during recovery. Verified recovery: * Root filesystem pressure used fell to 73.2%. * Kubernetes reported `DiskPressure=False` and removed the node taint. * Forgejo Postgres became ready. * Forgejo became `1/1 Running` after the recovery image-pull surge cleared. * The public `/api/healthz` endpoint returned HTTP 200. ## Problem The local-path PVC requests do not enforce filesystem quotas. Forgejo chunked-upload staging and the persistent build-runner Docker cache can therefore exceed their requested sizes and exhaust the node. The upload staging tree accumulated tens of gigabytes within hours, matching three observed outages over two days. ## Required follow-up * Identify which release or package-upload paths leave chunked uploads behind and determine whether cancellation, timeout, retry, or concurrent publication causes the leak. * Add a reviewed, concurrency-safe cleanup path for abandoned `tmp/package-upload` data. Normal operations must not depend on manual PVC file deletion. * Define package owner and per-upload limits or retention that fit the node budget. * Replace unbounded persistent BuildKit growth with a supported, measured Docker or BuildKit reclamation policy. * Confirm the disk-pressure watcher alerted before the outage and close any notification gap. * Update the Forgejo PVC retention and disk-pressure runbooks with the managed remediation. * Verify the change live by demonstrating bounded staging and runner-cache growth, root pressure below 85%, intact Forgejo data, and a healthy public edge.

coilyco-ops added the

labels

2026-07-24 07:36:23 +00:00

coilyco-ops referenced this issue

2026-07-27 03:02:14 +00:00

Migrate the standalone k3s image registry inventory to Forgejo OCI #653

coilyco-ops added

and removed

labels

2026-07-29 06:27:50 +00:00

coilyco-ops commented

2026-07-30 08:21:38 +00:00

Author

Member

Additional live evidence from eco-app run 148 verification on 2026-07-30:

kai-server entered DiskPressure and evicted forgejo-runner-gaming-0 before the run claimed a worker.
Node events reported NodeHasDiskPressure three times over 20 hours and FreeDiskSpaceFailed 204 times over 22 hours.
Kubelet attempted to reclaim about 23.7 GB but reported zero eligible image bytes.
The replacement gaming runner scheduled after DiskPressure cleared, ran init-wipe-docker-lib successfully, returned Running 2/2, and completed the workflow.
The cold replacement stretched the test gate from 2m37s in run 147 to about 14 minutes in run 148, but run 148 ultimately completed fully green.
Current after-state is kai-server Ready with DiskPressure false. No storage mutation was made during this verification.

This confirms the pressure and image-GC failure are recurring even though the immediate node condition self-cleared.

Additional live evidence from eco-app run 148 verification on 2026-07-30: * kai-server entered DiskPressure and evicted forgejo-runner-gaming-0 before the run claimed a worker. * Node events reported NodeHasDiskPressure three times over 20 hours and FreeDiskSpaceFailed 204 times over 22 hours. * Kubelet attempted to reclaim about 23.7 GB but reported zero eligible image bytes. * The replacement gaming runner scheduled after DiskPressure cleared, ran init-wipe-docker-lib successfully, returned Running 2/2, and completed the workflow. * The cold replacement stretched the test gate from 2m37s in run 147 to about 14 minutes in run 148, but run 148 ultimately completed fully green. * Current after-state is kai-server Ready with DiskPressure false. No storage mutation was made during this verification. This confirms the pressure and image-GC failure are recurring even though the immediate node condition self-cleared.

coilyco-ops commented

2026-07-31 08:51:22 +00:00

Author

Member

2026-07-31 recurrence and recovery evidence: kai-server reached DiskPressure=True with the NoSchedule taint. Kubelet logged 235 ImageGCFailed events over 46 hours and found zero eligible image bytes. The read-only classifier measured 224.8 GB in local-path PVCs, 4.0 GB in containerd, 2.2 GB in logs, and 1.0 GB in journals. A concurrent read-only inspector measured 24 Forgejo tmp/package-upload files totaling 17.5 GiB, all inside the existing three-hour safety window, so the cleanup CronJob correctly reclaimed nothing. As the build runner rescheduled, its established init-wipe path coincided with about 17.4 GB returning from local-path storage. Root pressure-use fell from 84.1% to 79.3%, DiskPressure cleared, and 97 workloads rescheduled. Forgejo then remained unavailable because codeberg.org returned HTTP 503 for its image. infrastructure#701 tracks the separate recovery-hardening change to Forgejo's official reachable mirror. After that image-only change, the Forgejo pod became Ready and the public health endpoint passed cache and database checks. No PVC data was deleted. The recurring upload-staging and build-cache root cause remains open here.

coilyco-ops commented

2026-07-31 09:08:22 +00:00

Author

Member

Containment update: after the first recovery, root pressure-use rose again from 76.3% to about 84% while the recovered node repulled images and agentic-os dev-base publish run 2595 occupied the image-build runner. A TTL-bound read-only inspector measured containerd at 23.9 GiB, up about 17.5 GiB from the post-eviction baseline, while the disposable image-build Docker PVC was only 4.0 GiB. The package-upload cleanup reported its staging tree had drained from 17.5 GiB to 12 KiB, so current staging was not the renewed growth source. Ops scaled only statefulset/forgejo-runner-build-flight-deck to zero before the node crossed 85%. The pod is gone and the StatefulSet verifies 0/0. This intentionally interrupts the active dev-base publish, which can be retried. No PVC or application data was deleted. Keep the build runner disabled until headroom is verified stable and the recurring capacity work here or infrastructure#693 supplies a safe execution plane.

coilyco-ops referenced this issue

2026-07-31 09:12:40 +00:00

Keep the kai-server image-build runner disabled after disk pressure #705

coilyco-ops commented

2026-07-31 09:23:40 +00:00

Author

Member

2026-07-31 containment is now landed and reconciled. Commit 317bfbe keeps the Flux-active Flight Deck image-build runner at 0/0 while retaining its registration and Docker claims. Flux reports apps Ready at main@sha1:317bfbe, Forgejo health passes, kai-server is Ready with DiskPressure=False and no taints, and the post-CI runner recycle left root pressure use at 80.0% with 102806388736 bytes available. Push checks lint run 1941 and trufflehog run 1942 both succeeded. No PVC or application data was deleted. This contains the recurrence but does not close the root-cause capacity work in this issue or the ser8 migration in #693.

2026-07-31 containment is now landed and reconciled. Commit 317bfbe keeps the Flux-active Flight Deck image-build runner at 0/0 while retaining its registration and Docker claims. Flux reports apps Ready at main@sha1:317bfbe, Forgejo health passes, kai-server is Ready with DiskPressure=False and no taints, and the post-CI runner recycle left root pressure use at 80.0% with 102806388736 bytes available. Push checks lint run 1941 and trufflehog run 1942 both succeeded. No PVC or application data was deleted. This contains the recurrence but does not close the root-cause capacity work in this issue or the ser8 migration in #693.

coilyco-ops referenced this issue

2026-07-31 09:27:21 +00:00

Add disk-pressure budgets and a guarded CI circuit breaker on kai-server #707

coilyco-ops referenced this issue

2026-07-31 09:30:56 +00:00

Make Forgejo runner recycling job-aware #706

coilyco-ops commented

2026-08-01 06:03:32 +00:00

Author

Member

Observed 2026-08-01T05:53:10Z: Forgejo application storage is 141.5G. Completed packages are 120.2G and attachments are 18.6G. The repository store is 660M and PostgreSQL is 381M, so neither Git repositories nor the database explains current pressure. Node observation at the same incident stage was 88.85% used with 57.4 GB available and 10.02% inode use. Kubernetes still reported DiskPressure=False, but image garbage collection had failed to free its requested space.\n\nThis shifts the durable root cause to completed package or OCI retention. The existing package-upload cleanup only owns stale upload scratch and must not be generalized to the package store. I have not deleted durable data. Per the disk-pressure runbook, relief now requires a current Forgejo and database backup, an exact package-owner/version inventory, and deletion only through a Forgejo-managed UI, API, or admin task.\n\nMeasurement reliability fix 4241976 landed on main and passed lint plus secret scan. It makes pipeline failures visible, replaces GNU find -printf with BusyBox-compatible du, and adds a bounded package-directory breakdown. The next attended capture will supply the ownership split.

Observed 2026-08-01T05:53:10Z: Forgejo application storage is 141.5G. Completed packages are 120.2G and attachments are 18.6G. The repository store is 660M and PostgreSQL is 381M, so neither Git repositories nor the database explains current pressure. Node observation at the same incident stage was 88.85% used with 57.4 GB available and 10.02% inode use. Kubernetes still reported DiskPressure=False, but image garbage collection had failed to free its requested space.\n\nThis shifts the durable root cause to completed package or OCI retention. The existing package-upload cleanup only owns stale upload scratch and must not be generalized to the package store. I have not deleted durable data. Per the disk-pressure runbook, relief now requires a current Forgejo and database backup, an exact package-owner/version inventory, and deletion only through a Forgejo-managed UI, API, or admin task.\n\nMeasurement reliability fix 4241976 landed on main and passed lint plus secret scan. It makes pipeline failures visible, replaces GNU find -printf with BusyBox-compatible du, and adds a bounded package-directory breakdown. The next attended capture will supply the ownership split.

coilyco-ops commented

2026-08-04 18:32:50 +00:00

Author

Member

2026-08-04 recovery evidence:\n\n* A fresh incremental Restic backup completed before package deletion.\n* Supported Forgejo package retention deleted 690 old version records, retained the newest 50 versions plus 13 moving aliases, and a fresh preview reports zero remaining candidates.\n* Root usage fell from 86% to 68-69%. The latest check shows about 154 GB available.\n* Kubernetes reports Ready with DiskPressure=False and no pressure taints.\n* Commit 73c3f33 restored the scoped image-build runner through source and Flux. The pod is 2/2 Running with no restarts, advertises docker-build, and has claimed the queued AOS build.\n* During the live image build, root usage remains steady at 69%.\n\nContainment is lifted. Keeping this issue open for the upload-staging prevention work rather than treating retention as the root-cause fix.

2026-08-04 recovery evidence:\n\n* A fresh incremental Restic backup completed before package deletion.\n* Supported Forgejo package retention deleted 690 old version records, retained the newest 50 versions plus 13 moving aliases, and a fresh preview reports zero remaining candidates.\n* Root usage fell from 86% to 68-69%. The latest check shows about 154 GB available.\n* Kubernetes reports Ready with DiskPressure=False and no pressure taints.\n* Commit 73c3f33 restored the scoped image-build runner through source and Flux. The pod is 2/2 Running with no restarts, advertises docker-build, and has claimed the queued AOS build.\n* During the live image build, root usage remains steady at 69%.\n\nContainment is lifted. Keeping this issue open for the upload-staging prevention work rather than treating retention as the root-cause fix.

coilyco-ops added

and removed

labels

2026-08-05 10:25:59 +00:00

Rows
Columns

Prevent Forgejo upload staging from exhausting kai-server disk #624

Incident

Problem

Required follow-up