coilyco-flight-deck/infrastructure

Watch

kai-server rootfs is at 82.8 percent, past the warn threshold, with local-path PVC data at 129 GB the largest consumer #966

New issue

Closed

opened 2026-08-27 10:04:19 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-27 10:04:19 +00:00

Owner

Classification only. No relief run, per the disk-pressure runbook's stop condition.

Found while checking cluster health after coilyco-bridge/deploy#811.

Byte and inode state

/            total 514,909,585,408   used 426,293,997,568   82.79%
             status warning (warn 80, critical 85)
             11.38 GB until critical
inodes       10.38% used

kubelet still reports DiskPressure=False, last transitioned 4d8h ago, so
this is headroom loss rather than an active eviction condition. No
ImageGCFailed or FreeDiskSpaceFailed events.

Measured ownership

/var/lib totals 241.9 GB:

/var/lib/rancher - 196.1 GB
/var/lib/kubelet - 34.9 GB
/var/lib/snapd - 5.6 GB
/var/lib/containerd - 4.6 GB (separate from k3s's own)
/var/lib/apt - 0.41 GB
everything else under 0.11 GB

/var/lib/rancher/k3s breaks down as:

storage (local-path PVC data) - 129.3 GB
agent (containerd image and snapshot store) - 55.4 GB
server - 11.3 GB
data - 0.23 GB

So local-path PVC data is the largest single consumer on the node, ahead of the
image store.

About 184 GB sits outside /var/lib entirely and is unmeasured here. The
root-profile scan had not returned when I stopped. That is the largest remaining
unknown and should be measured before anyone reasons about what to reclaim.

Per-PVC attribution is not yet trustworthy

The bounded volume scan timed out:

timed_out: true
totals_are_lower_bounds: true
volume_results_truncated: true
total_entries_scanned: 101,494 of max 2,000,000

Every per-PVC number below is a floor, not a total, and 28 of the 43 bound PVCs
were not returned at all. The sum of the 15 returned is far short of the 129.3 GB
the directory scan measured, and that gap is scan truncation rather than orphaned
data: unattributed_path_count: 0 across paths_scanned: 43 equal to
volume_count: 43, so every directory under storage maps to a bound PVC.

Largest returned, as lower bounds:

forgejo/forgejo-data - >= 7.58 GiB of 20 GiB
forgejo/docker-lib-forgejo-runner-build-0 - >= 4.95 GiB of 10 GiB
registry/registry-data - >= 1.73 GiB of 20 GiB

A complete attribution needs a scan with a higher entry cap or a longer timeout.

Four runner PVCs are over their request

local-path enforces no quota, and these have overrun theirs:

forgejo/data-forgejo-runner-2 - >= 1.59 GiB against a 1 GiB request
forgejo/data-forgejo-runner-0 - >= 1.52 GiB against 1 GiB
forgejo/data-forgejo-runner-3 - >= 1.52 GiB against 1 GiB
forgejo/data-forgejo-runner-1 - >= 1.39 GiB against 1 GiB

Those are lower bounds too, so the real overrun is larger. Runner scratch
ownership is the runner-failure runbook's territory.

Logs are small but two files are anomalous

Logs total 3.82 GB, which is not the problem, but the shape is odd:

/var/log/syslog - 1.54 GB in the single active file
/var/log/xrdp.log.1.gz - 510 MB compressed, so far larger raw
/var/log/pods - 322 MB
/var/log/syslog.1 - 210 MB
/var/log/journal - 1.14 GB

An xrdp log that compresses to half a gigabyte means something is writing to it
hard. Worth understanding on its own, separately from reclaiming the bytes.

Possible operational relevance

ci / validate on coilyco-bridge/deploy@52716df6 failed at
docker pull ... agentic-os:release forcePull=true with context canceled after
70s, on a node with 11 GB of runway and no runner eviction or restart. That is
correlation, not a proven cause, and it recovered on the next run. Noting it
because a forcePull on every CI job is a recurring large write against thin
headroom.

Not done

The ~184 GB outside /var/lib is unmeasured.
Per-PVC attribution inside the 129.3 GB is incomplete.
No relief action taken, proposed, or approved. No rm, prune, truncation, or
PVC deletion.

Classification only. No relief run, per the disk-pressure runbook's stop condition. Found while checking cluster health after coilyco-bridge/deploy#811. ## Byte and inode state ``` / total 514,909,585,408 used 426,293,997,568 82.79% status warning (warn 80, critical 85) 11.38 GB until critical inodes 10.38% used ``` `kubelet` still reports `DiskPressure=False`, last transitioned 4d8h ago, so this is headroom loss rather than an active eviction condition. No `ImageGCFailed` or `FreeDiskSpaceFailed` events. ## Measured ownership `/var/lib` totals **241.9 GB**: * `/var/lib/rancher` - **196.1 GB** * `/var/lib/kubelet` - 34.9 GB * `/var/lib/snapd` - 5.6 GB * `/var/lib/containerd` - 4.6 GB (separate from k3s's own) * `/var/lib/apt` - 0.41 GB * everything else under 0.11 GB `/var/lib/rancher/k3s` breaks down as: * `storage` (local-path PVC data) - **129.3 GB** * `agent` (containerd image and snapshot store) - **55.4 GB** * `server` - 11.3 GB * `data` - 0.23 GB So local-path PVC data is the largest single consumer on the node, ahead of the image store. **About 184 GB sits outside `/var/lib` entirely** and is unmeasured here. The root-profile scan had not returned when I stopped. That is the largest remaining unknown and should be measured before anyone reasons about what to reclaim. ## Per-PVC attribution is not yet trustworthy The bounded volume scan **timed out**: ``` timed_out: true totals_are_lower_bounds: true volume_results_truncated: true total_entries_scanned: 101,494 of max 2,000,000 ``` Every per-PVC number below is a floor, not a total, and 28 of the 43 bound PVCs were not returned at all. The sum of the 15 returned is far short of the 129.3 GB the directory scan measured, and that gap is scan truncation rather than orphaned data: `unattributed_path_count: 0` across `paths_scanned: 43` equal to `volume_count: 43`, so every directory under `storage` maps to a bound PVC. Largest returned, as lower bounds: * `forgejo/forgejo-data` - >= 7.58 GiB of 20 GiB * `forgejo/docker-lib-forgejo-runner-build-0` - >= 4.95 GiB of 10 GiB * `registry/registry-data` - >= 1.73 GiB of 20 GiB A complete attribution needs a scan with a higher entry cap or a longer timeout. ## Four runner PVCs are over their request local-path enforces no quota, and these have overrun theirs: * `forgejo/data-forgejo-runner-2` - >= 1.59 GiB against a 1 GiB request * `forgejo/data-forgejo-runner-0` - >= 1.52 GiB against 1 GiB * `forgejo/data-forgejo-runner-3` - >= 1.52 GiB against 1 GiB * `forgejo/data-forgejo-runner-1` - >= 1.39 GiB against 1 GiB Those are lower bounds too, so the real overrun is larger. Runner scratch ownership is the runner-failure runbook's territory. ## Logs are small but two files are anomalous Logs total 3.82 GB, which is not the problem, but the shape is odd: * `/var/log/syslog` - **1.54 GB** in the single active file * `/var/log/xrdp.log.1.gz` - **510 MB compressed**, so far larger raw * `/var/log/pods` - 322 MB * `/var/log/syslog.1` - 210 MB * `/var/log/journal` - 1.14 GB An `xrdp` log that compresses to half a gigabyte means something is writing to it hard. Worth understanding on its own, separately from reclaiming the bytes. ## Possible operational relevance `ci / validate` on `coilyco-bridge/deploy@52716df6` failed at `docker pull ... agentic-os:release forcePull=true` with `context canceled` after 70s, on a node with 11 GB of runway and no runner eviction or restart. That is correlation, not a proven cause, and it recovered on the next run. Noting it because a `forcePull` on every CI job is a recurring large write against thin headroom. ## Not done * The ~184 GB outside `/var/lib` is unmeasured. * Per-PVC attribution inside the 129.3 GB is incomplete. * No relief action taken, proposed, or approved. No `rm`, prune, truncation, or PVC deletion.

coilyco-ops commented

2026-08-28 21:13:35 +00:00

Author

Owner

The reading no longer holds. Closing as a stale measurement.

Measured on kai-server 2026-08-28:

total  514,909,585,408 bytes   (514.9 GB)
used   380,762,824,704 bytes   (380.8 GB)
free   107,915,534,336 bytes   (107.9 GB)
percent 77.9

77.9 percent, against the 82.8 percent this issue was filed on. Roughly 5 points below the figure that triggered it, and below the warn threshold it names.

Why this closes rather than stays open

This issue is a point-in-time reading with a threshold breach in its title. That specific breach is over, so the issue as written no longer describes the system. Keeping it open makes the backlog assert something untrue about kai-server.

The underlying concern, that kai-server has no headroom plan, is real and belongs somewhere permanent rather than in a snapshot. #870 is the standing kai-server capacity issue and I have updated it with today's numbers, so nothing is lost by closing this.

Not established

What reclaimed the space. I did not find the change that moved it, and 5 points is well within the range that ordinary runner and image churn produces on this node. So treat this as "the number moved" rather than "a fix landed". If it climbs back through 82 percent, that is #870's business and this issue does not need reopening.

The 129 GB local-path PVC observation in the body was not re-measured and may still be accurate. #846 covers local-path storage specifically.

## The reading no longer holds. Closing as a stale measurement. Measured on kai-server 2026-08-28: ``` total 514,909,585,408 bytes (514.9 GB) used 380,762,824,704 bytes (380.8 GB) free 107,915,534,336 bytes (107.9 GB) percent 77.9 ``` **77.9 percent**, against the 82.8 percent this issue was filed on. Roughly 5 points below the figure that triggered it, and below the warn threshold it names. ### Why this closes rather than stays open This issue is a point-in-time reading with a threshold breach in its title. That specific breach is over, so the issue as written no longer describes the system. Keeping it open makes the backlog assert something untrue about kai-server. The underlying concern, that kai-server has no headroom plan, is real and belongs somewhere permanent rather than in a snapshot. **#870 is the standing kai-server capacity issue** and I have updated it with today's numbers, so nothing is lost by closing this. ### Not established What reclaimed the space. I did not find the change that moved it, and 5 points is well within the range that ordinary runner and image churn produces on this node. So treat this as "the number moved" rather than "a fix landed". If it climbs back through 82 percent, that is #870's business and this issue does not need reopening. The 129 GB local-path PVC observation in the body was not re-measured and may still be accurate. #846 covers local-path storage specifically.

coilyco-ops

2026-08-28 21:13:40 +00:00