coilyco-flight-deck/infrastructure

Watch

Add disk-pressure budgets and a guarded CI circuit breaker on kai-server #707

New issue

Open

opened 2026-07-31 09:27:21 +00:00 by coilyco-ops · 6 comments

coilyco-ops commented

2026-07-31 09:27:21 +00:00

Owner

Outcome

Keep the single-node kai-server cluster out of Kubernetes DiskPressure while Forgejo uploads, container repulls, and CI scratch change which storage category is growing. Add one measured capacity policy, early warning, and a guarded admission path for disk-heavy CI. This issue coordinates the control layer around the source-specific cleanup in #624 and the runner migration in #693 without duplicating either.

What the 2026-07-31 incident taught us

Disk pressure was compound and phase-dependent, not one persistent leak. The first classifier measured 224.8 GB in local-path PVCs and 17.5 GiB of Forgejo package-upload staging. Every staging file was inside the existing three-hour safety window, so the cleanup job correctly reclaimed nothing. The staging tree later drained to 12 KiB without unsafe deletion.
The renewed climb had a different source. After mass eviction and rescheduling, containerd reached 23.9 GiB, about 17.5 GiB above the post-eviction baseline, while root pressure use rose from 76.3% to about 84% during an image-publish workflow. The image-build Docker PVC was only about 4.0 GiB at that point.
Kubelet image GC was not an effective safety valve. The node recorded 235 ImageGCFailed events over roughly 46 hours. One later attempt needed about 20.6 GB but found only about 32 MB eligible.
Recovery can create its own pressure. Rescheduling repulls images, and recycled CI pools rebuild cold caches. The general Flight Deck pool recycle reclaimed about 16.5 GB during containment, while the two cold-cache validation runs temporarily moved root pressure use from 80% to 82% before a final idle recycle returned it to 80%.
Requested local-path PVC sizes are planning metadata, not enforced filesystem quotas. A healthy pod and a nominal 10 GiB claim do not prove bounded host consumption.
DiskPressure is a cluster-wide failure mode. It tainted the only node, evicted unrelated workloads, delayed CI, and left Forgejo recovery dependent on a reachable image source. Commit 661a938 separately hardened that image source, but it does not reduce disk risk.
The safe containment was to keep ordinary workloads and general CI running, disable only the disk-heavy image-build lane, retain rollback data, and verify the node, Flux, Forgejo, and repository checks after each change.

Proposed forward actions

Define one owning capacity policy for kai-server with explicit hysteresis. Start from the observed boundary: warning near 80% root pressure use, block new disk-heavy CI near 82%, and do not re-admit it until pressure remains below a lower recovery threshold for consecutive samples. Include an absolute available-byte floor so percentage rounding cannot hide a narrow buffer. Tune these values from measured peak growth, not intuition.
Extend the existing read-only disk report into a machine-consumable signal for root available bytes, containerd, local-path PVCs grouped by workload, Forgejo package staging, runner scratch, logs, journals, and game saves. Export the same values to the observability plane and alert before kubelet reaches its eviction threshold.
Add a guarded operator workflow for the image-build lane while its tracked default remains zero. It must refuse enablement above the hard threshold, verify DiskPressure is false and the node has no pressure taint, enable only one bounded runner, record a deadline, monitor the run, restore zero afterward, recycle disposable scratch, and report the before and after measurements. An interrupted operator session must fail back to zero.
Add a second guard at workflow start so a queued image publication fails clearly before expensive build inputs are pulled when host headroom is below policy. The failure must name the current measurement and the retry condition.
Complete #624 with concurrency-safe abandoned-upload cleanup, bounded package retention, and measured BuildKit or Docker reclamation. Preserve the active-upload safety window unless evidence supports changing it.
Complete #693 so general, image-build, and eligible publisher runners execute on ser8. Keep the kai-server deploy runner local because its ServiceAccount intentionally controls that cluster.
Add a recovery runbook that distinguishes local-path growth, containerd repulls, active upload staging, disposable runner scratch, and durable application data. Each category needs an approved observation method, safe relief action, rollback statement, and explicit data-retention boundary.
Run a representative four-way image-publish load test on the target execution plane. Capture peak root use, containerd growth, runner scratch, reclaim behavior, queue time, and recovery time. Use that evidence to finalize thresholds and the rollback window.

Acceptance

Warning arrives before DiskPressure, eviction, or a NoSchedule taint.
The hard guard prevents a disk-heavy build from starting above policy and explains how to recover.
Hysteresis prevents rapid enable-disable flapping around one threshold.
A bounded build window returns the image-build runner to zero even when the build fails or the operator session ends.
Active Forgejo uploads survive cleanup and guard verification.
A representative image-build fan-out completes without evicting unrelated pods or crossing the agreed hard threshold.
The node remains Ready, DiskPressure=False, and untainted through the verification window.
The ser8 migration preserves runner registration scope, credential isolation, observable health, and a tested rollback.
The disk-pressure and runner runbooks document the final policy, measurements, and ownership.

#624 tracks upload staging and cache growth.
#693 tracks moving disk-heavy runners to ser8.
#705 landed the tracked 0/0 containment in commit 317bfbe.
#701 landed the Forgejo recovery image mirror in commit 661a938.
Final contained state on 2026-07-31 was 80.0% root pressure use with 102806388736 bytes available, 186.5 GB in local-path PVCs, 25.5 GB in containerd, DiskPressure=False, no taints, Forgejo health passing, ordinary Flight Deck runners 2/2, and the image-build runner 0/0. No PVC or application data was deleted.

## Outcome Keep the single-node kai-server cluster out of Kubernetes DiskPressure while Forgejo uploads, container repulls, and CI scratch change which storage category is growing. Add one measured capacity policy, early warning, and a guarded admission path for disk-heavy CI. This issue coordinates the control layer around the source-specific cleanup in #624 and the runner migration in #693 without duplicating either. ## What the 2026-07-31 incident taught us * Disk pressure was compound and phase-dependent, not one persistent leak. The first classifier measured 224.8 GB in local-path PVCs and 17.5 GiB of Forgejo package-upload staging. Every staging file was inside the existing three-hour safety window, so the cleanup job correctly reclaimed nothing. The staging tree later drained to 12 KiB without unsafe deletion. * The renewed climb had a different source. After mass eviction and rescheduling, containerd reached 23.9 GiB, about 17.5 GiB above the post-eviction baseline, while root pressure use rose from 76.3% to about 84% during an image-publish workflow. The image-build Docker PVC was only about 4.0 GiB at that point. * Kubelet image GC was not an effective safety valve. The node recorded 235 ImageGCFailed events over roughly 46 hours. One later attempt needed about 20.6 GB but found only about 32 MB eligible. * Recovery can create its own pressure. Rescheduling repulls images, and recycled CI pools rebuild cold caches. The general Flight Deck pool recycle reclaimed about 16.5 GB during containment, while the two cold-cache validation runs temporarily moved root pressure use from 80% to 82% before a final idle recycle returned it to 80%. * Requested local-path PVC sizes are planning metadata, not enforced filesystem quotas. A healthy pod and a nominal 10 GiB claim do not prove bounded host consumption. * DiskPressure is a cluster-wide failure mode. It tainted the only node, evicted unrelated workloads, delayed CI, and left Forgejo recovery dependent on a reachable image source. Commit 661a938 separately hardened that image source, but it does not reduce disk risk. * The safe containment was to keep ordinary workloads and general CI running, disable only the disk-heavy image-build lane, retain rollback data, and verify the node, Flux, Forgejo, and repository checks after each change. ## Proposed forward actions 1. Define one owning capacity policy for kai-server with explicit hysteresis. Start from the observed boundary: warning near 80% root pressure use, block new disk-heavy CI near 82%, and do not re-admit it until pressure remains below a lower recovery threshold for consecutive samples. Include an absolute available-byte floor so percentage rounding cannot hide a narrow buffer. Tune these values from measured peak growth, not intuition. 2. Extend the existing read-only disk report into a machine-consumable signal for root available bytes, containerd, local-path PVCs grouped by workload, Forgejo package staging, runner scratch, logs, journals, and game saves. Export the same values to the observability plane and alert before kubelet reaches its eviction threshold. 3. Add a guarded operator workflow for the image-build lane while its tracked default remains zero. It must refuse enablement above the hard threshold, verify DiskPressure is false and the node has no pressure taint, enable only one bounded runner, record a deadline, monitor the run, restore zero afterward, recycle disposable scratch, and report the before and after measurements. An interrupted operator session must fail back to zero. 4. Add a second guard at workflow start so a queued image publication fails clearly before expensive build inputs are pulled when host headroom is below policy. The failure must name the current measurement and the retry condition. 5. Complete #624 with concurrency-safe abandoned-upload cleanup, bounded package retention, and measured BuildKit or Docker reclamation. Preserve the active-upload safety window unless evidence supports changing it. 6. Complete #693 so general, image-build, and eligible publisher runners execute on ser8. Keep the kai-server deploy runner local because its ServiceAccount intentionally controls that cluster. 7. Add a recovery runbook that distinguishes local-path growth, containerd repulls, active upload staging, disposable runner scratch, and durable application data. Each category needs an approved observation method, safe relief action, rollback statement, and explicit data-retention boundary. 8. Run a representative four-way image-publish load test on the target execution plane. Capture peak root use, containerd growth, runner scratch, reclaim behavior, queue time, and recovery time. Use that evidence to finalize thresholds and the rollback window. ## Acceptance * Warning arrives before DiskPressure, eviction, or a NoSchedule taint. * The hard guard prevents a disk-heavy build from starting above policy and explains how to recover. * Hysteresis prevents rapid enable-disable flapping around one threshold. * A bounded build window returns the image-build runner to zero even when the build fails or the operator session ends. * Active Forgejo uploads survive cleanup and guard verification. * A representative image-build fan-out completes without evicting unrelated pods or crossing the agreed hard threshold. * The node remains Ready, DiskPressure=False, and untainted through the verification window. * The ser8 migration preserves runner registration scope, credential isolation, observable health, and a tested rollback. * The disk-pressure and runner runbooks document the final policy, measurements, and ownership. ## Evidence and related work * #624 tracks upload staging and cache growth. * #693 tracks moving disk-heavy runners to ser8. * #705 landed the tracked 0/0 containment in commit 317bfbe. * #701 landed the Forgejo recovery image mirror in commit 661a938. * Final contained state on 2026-07-31 was 80.0% root pressure use with 102806388736 bytes available, 186.5 GB in local-path PVCs, 25.5 GB in containerd, DiskPressure=False, no taints, Forgejo health passing, ordinary Flight Deck runners 2/2, and the image-build runner 0/0. No PVC or application data was deleted.

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-31 09:27:21 +00:00

coilyco-ops commented

2026-08-01 05:21:38 +00:00

Author

Owner

2026-08-01 recurrence evidence

Current read-only classification:

Root pressure use is 88.6% with 58,572,644,352 bytes available and 456,336,941,056 pressure-used bytes. Inodes are 10% used, so this is byte pressure.
The node is Ready, untainted, and reports DiskPressure=False. Kubelet is nevertheless emitting recurring FreeDiskSpaceFailed events. The latest attempt needed about 44.4 GB and found only about 0.3 MB eligible for image GC.
The bounded host report attributes 187.5 GB to local-path PVCs, 75.5 GB to the Kai home, 18.1 GB to containerd, 2.37 GB to logs, 1.06 GB to journald, and 0.87 GB to game saves.
The bounded PVC scanner completed registry-data at 13.9 GB. It timed out on forgejo-data after a 22.0 GB lower bound and on the retired zero-replica unscoped image-builder Docker cache after a 5.9 GB lower bound. The tracked Flight Deck image-builder remains 0/0 and its current Docker cache is negligible.
The nightly Forgejo runner recycle last succeeded and its active general-runner targets are healthy. No runner recycle was triggered during this pass.
The documented restic-backup service and timer are not installed on kai-server. The restic-rest-server service is not installed on ser8. Restic is absent on kai-server, and both restic and rest-server are absent on ser8. There is therefore no fresh rollback artifact for Forgejo or registry data.

No live mutation was made. Durable Forgejo cleanup is blocked by the runbook until an attended operator installs the backup plane, completes an initial backup, and returns the in-pod Forgejo storage split from the disk-pressure measurement reference. Blind PVC, package, Git object, containerd, or log deletion remains unsafe.

## 2026-08-01 recurrence evidence Current read-only classification: * Root pressure use is 88.6% with 58,572,644,352 bytes available and 456,336,941,056 pressure-used bytes. Inodes are 10% used, so this is byte pressure. * The node is Ready, untainted, and reports DiskPressure=False. Kubelet is nevertheless emitting recurring FreeDiskSpaceFailed events. The latest attempt needed about 44.4 GB and found only about 0.3 MB eligible for image GC. * The bounded host report attributes 187.5 GB to local-path PVCs, 75.5 GB to the Kai home, 18.1 GB to containerd, 2.37 GB to logs, 1.06 GB to journald, and 0.87 GB to game saves. * The bounded PVC scanner completed registry-data at 13.9 GB. It timed out on forgejo-data after a 22.0 GB lower bound and on the retired zero-replica unscoped image-builder Docker cache after a 5.9 GB lower bound. The tracked Flight Deck image-builder remains 0/0 and its current Docker cache is negligible. * The nightly Forgejo runner recycle last succeeded and its active general-runner targets are healthy. No runner recycle was triggered during this pass. * The documented restic-backup service and timer are not installed on kai-server. The restic-rest-server service is not installed on ser8. Restic is absent on kai-server, and both restic and rest-server are absent on ser8. There is therefore no fresh rollback artifact for Forgejo or registry data. No live mutation was made. Durable Forgejo cleanup is blocked by the runbook until an attended operator installs the backup plane, completes an initial backup, and returns the in-pod Forgejo storage split from the disk-pressure measurement reference. Blind PVC, package, Git object, containerd, or log deletion remains unsafe.

coilyco-ops commented

2026-08-01 05:51:35 +00:00

Author

Owner

Forgejo measurement follow-up at 2026-08-01T05:50Z:\n\n* attended run reached the cluster and reported PostgreSQL at 381 MB\n* the four application measurements failed before execution because sh -lc loaded a Bash-only Forgejo login-profile fragment\n* corrected wrapper now uses non-login sh -c, with a focused fake-k3s regression test\n* commit 9b67b77 landed on canonical main and Forgejo lint plus secret scan passed\n* live after-state remains pending the attended wrapper rerun\n\nRollback: revert 9b67b77 if the non-login shell causes an unexpected measurement regression. No live storage mutation was made.

Forgejo measurement follow-up at 2026-08-01T05:50Z:\n\n* attended run reached the cluster and reported PostgreSQL at 381 MB\n* the four application measurements failed before execution because sh -lc loaded a Bash-only Forgejo login-profile fragment\n* corrected wrapper now uses non-login sh -c, with a focused fake-k3s regression test\n* commit 9b67b77 landed on canonical main and Forgejo lint plus secret scan passed\n* live after-state remains pending the attended wrapper rerun\n\nRollback: revert 9b67b77 if the non-login shell causes an unexpected measurement regression. No live storage mutation was made.

coilyco-ops commented

2026-08-01 06:03:32 +00:00

Author

Owner

Current disk-pressure evidence: root use is 88.85% with 57.4 GB available, inodes are healthy at 10.02%, and kubelet image garbage collection is repeatedly failing. Forgejo owns 141.5G, dominated by 120.2G of completed package or OCI data and 18.6G of attachments. Its repositories total 660M and PostgreSQL is 381M.\n\nNo durable Forgejo data has been removed. Package retention remains gated on a current backup and an exact owner/version inventory, with deletion through a Forgejo-managed surface only. Measurement fix 4241976 is on main with successful lint and secret-scan runs. A corrected attended capture is the next evidence step.

Current disk-pressure evidence: root use is 88.85% with 57.4 GB available, inodes are healthy at 10.02%, and kubelet image garbage collection is repeatedly failing. Forgejo owns 141.5G, dominated by 120.2G of completed package or OCI data and 18.6G of attachments. Its repositories total 660M and PostgreSQL is 381M.\n\nNo durable Forgejo data has been removed. Package retention remains gated on a current backup and an exact owner/version inventory, with deletion through a Forgejo-managed surface only. Measurement fix 4241976 is on main with successful lint and secret-scan runs. A corrected attended capture is the next evidence step.

coilyco-ops commented

2026-08-01 06:23:20 +00:00

Author

Owner

Follow-up evidence from agentic-os#836 on 2026-08-01:

Commit 08c4992d landed on main and released as aos-v0.144.0.
Promotion 2628, mirror 2627, AOS CLI release 2625, and pre-commit release 2629 succeeded.
Repository CI 2626 and dev-base publication 2630 remained waiting and undispatched from 06:08Z through the final 06:21Z observation.
Bounded SigNoz-only checks found no forgejo-runner ERROR records in the prior 15 minutes and no runner log entry for run ID 14278, consistent with Forgejo not dispatching the queued job.

The native release is complete. Dev-base publication remains queued behind the runner and disk-pressure containment tracked here.

Follow-up evidence from agentic-os#836 on 2026-08-01: * Commit [08c4992d](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/commit/08c4992dfaade82204331061ee60bed526a69c3b) landed on main and released as aos-v0.144.0. * Promotion 2628, mirror 2627, AOS CLI release 2625, and pre-commit release 2629 succeeded. * Repository CI 2626 and dev-base publication 2630 remained waiting and undispatched from 06:08Z through the final 06:21Z observation. * Bounded SigNoz-only checks found no forgejo-runner ERROR records in the prior 15 minutes and no runner log entry for run ID 14278, consistent with Forgejo not dispatching the queued job. The native release is complete. Dev-base publication remains queued behind the runner and disk-pressure containment tracked here.

coilyco-ops commented

2026-08-01 16:45:55 +00:00

Author

Owner

Package attribution is complete. Forgejo has 120 GB of referenced package blobs and only 149 MB expired-unreferenced. coilyco-flight-deck/container owns 117 GB, with agentic-os accounting for 116 GB and 749 external tags. There are zero package cleanup rules. This is persistent referenced data, not a failed garbage-collection backlog.\n\nThe proposed bounded Forgejo rule keeps the newest 50 tags per package, protects every moving release and build-cache alias, and removes all other tags during supported package cleanup. No destructive change has run. A fresh backup and Kai's explicit approval remain the immediate gate.

coilyco-ops commented

2026-08-01 16:55:30 +00:00

Author

Owner

Immediate package-pressure relief is now scripted and landed in 0d1b5a0 (infrastructure#719). The measured source remains 116 GB of referenced agentic-os OCI blobs across 749 external versions. The script is preview-first, preserves the newest 50 versions and moving release/buildcache aliases, and refuses apply without an exact confirmation and a successful restic backup completed within four hours. No durable data has been deleted. Next evidence is the attended preview candidate count, then a fresh backup and explicit destructive approval before apply.

Immediate package-pressure relief is now scripted and landed in 0d1b5a0 (infrastructure#719). The measured source remains 116 GB of referenced agentic-os OCI blobs across 749 external versions. The script is preview-first, preserves the newest 50 versions and moving release/buildcache aliases, and refuses apply without an exact confirmation and a successful restic backup completed within four hours. No durable data has been deleted. Next evidence is the attended preview candidate count, then a fresh backup and explicit destructive approval before apply.

coilyco-ops added the

role/sysadmin

label

2026-08-25 08:02:17 +00:00

coilyco-ops referenced this issue

2026-08-28 21:15:25 +00:00

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

coilyco-ops added this to the Storage and capacity milestone

2026-08-29 06:09:14 +00:00