#870 - kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase - coilyco-flight-deck/infrastructure

coilyco-ops commented

2026-08-19 11:42:26 +00:00

Owner

Filed by Claude (ops seat), from a read-only /ops-investigation-disk-pressure pass on 2026-08-19 (~10:30Z). Nothing changed. This is the capacity half; the reclaim-side findings are already tracked at #868, #863, #864 and are not duplicated here.

Signal

Root / on kai-server is at 87.5% (critical) right now: ~64 GB available, ~13 GB over the 85% critical threshold. Inodes 9.5% - comfortable, so this is byte pressure, not inode exhaustion.

This is not a spike. The manual scratch sweep this morning recorded on #868 (05:17Z) took it from 84.74% to 77.56%, reclaiming ~37 GB. Five hours later it is back to 87.5% - inside its normal band, but that band's ceiling rides over critical.

30-day trend: flat and oscillating, not a leak

node_stats.filesystem.utilization{host.name=kai-server}, daily avg, 2026-07-25 to 2026-08-19:

78 75 72 74 81 86 82 89 88 88 85 79 85 84 81 75 74 69 77 80 81 81 83 82 81 86

Range ~69-89%, mean ~80%, no upward trend. It fills and self-corrects (runner recycle, imageGC, log rotation), but the baseline is high enough that peaks repeatedly cross 85%. Chronic, not acute.

Where the ~388 GB is (measured, mount-corrected via node-stats)

Owner	Size	Reclaimable?
containerd image store (k3s)	~130 GiB	No - imageGC reports 0 eligible, all images referenced by running pods (`FreeDiskSpaceFailed`)
local-path PVCs	>=74 GiB	Persistent data - `forgejo-data` 38.5+, `registry-data` 13, runner DinD caches 13, runner data 7
/home (Kai)	85 GiB	Outside k8s
/swapfile	32 GiB	Swap
/var/lib/kubelet (pod ephemeral)	30 GiB	emptyDir/scratch
/var/log + journald	~2.5 GiB	`xrdp.log.1.gz` 510 MB, journald 1.1 GiB

Diagnosis

The two dominant consumers - ~130 GiB of in-use container images and ~74 GiB of Forgejo/registry data - are legitimate and durable. There is no large safe cleanup target. The reclaim work at #868/#863 buys headroom but does not move the baseline. The honest conclusion is capacity: a 512 GB NVMe is undersized for this workload, and the node has ridden over-critical for a month.

Recommendation

Add or expand disk on kai-server (physical - homelab box, not a cloud resize). The durable fix.
Software levers that buy time but do not replace disk: an image retention/GC policy (a chunk of the 130 GiB is fleet-roll churn and eval-lane image duplication - see coilysiren/inbox#382 on lane sprawl); shrinking the 32 GiB swapfile if RAM headroom allows; journald vacuum.

Acceptance

A capacity decision recorded: add disk, or accept riding the critical line with the reclaim jobs holding it.
If disk is added, the new size and the resulting utilization recorded.
Boundaries: PVC/image removal and any hardware change are attended operator actions - the guarded surface has no delete/exec.

Filed by Claude (ops seat), from a read-only `/ops-investigation-disk-pressure` pass on 2026-08-19 (~10:30Z). Nothing changed. This is the **capacity** half; the reclaim-side findings are already tracked at #868, #863, #864 and are not duplicated here. ## Signal Root `/` on kai-server is at **87.5% (critical)** right now: ~64 GB available, **~13 GB over the 85% critical threshold**. Inodes 9.5% - comfortable, so this is byte pressure, not inode exhaustion. This is not a spike. The manual scratch sweep this morning recorded on #868 (05:17Z) took it from 84.74% to 77.56%, reclaiming ~37 GB. **Five hours later it is back to 87.5%** - inside its normal band, but that band's ceiling rides over critical. ## 30-day trend: flat and oscillating, not a leak `node_stats.filesystem.utilization{host.name=kai-server}`, daily avg, 2026-07-25 to 2026-08-19: 78 75 72 74 81 86 82 89 88 88 85 79 85 84 81 75 74 69 77 80 81 81 83 82 81 86 Range ~69-89%, mean ~80%, **no upward trend**. It fills and self-corrects (runner recycle, imageGC, log rotation), but the baseline is high enough that peaks repeatedly cross 85%. Chronic, not acute. ## Where the ~388 GB is (measured, mount-corrected via node-stats) | Owner | Size | Reclaimable? | | --- | --- | --- | | containerd image store (k3s) | ~130 GiB | **No** - imageGC reports 0 eligible, all images referenced by running pods (`FreeDiskSpaceFailed`) | | local-path PVCs | >=74 GiB | Persistent data - `forgejo-data` 38.5+, `registry-data` 13, runner DinD caches 13, runner data 7 | | /home (Kai) | 85 GiB | Outside k8s | | /swapfile | 32 GiB | Swap | | /var/lib/kubelet (pod ephemeral) | 30 GiB | emptyDir/scratch | | /var/log + journald | ~2.5 GiB | `xrdp.log.1.gz` 510 MB, journald 1.1 GiB | ## Diagnosis The two dominant consumers - ~130 GiB of in-use container images and ~74 GiB of Forgejo/registry data - are legitimate and durable. **There is no large safe cleanup target.** The reclaim work at #868/#863 buys headroom but does not move the baseline. The honest conclusion is capacity: a 512 GB NVMe is undersized for this workload, and the node has ridden over-critical for a month. ## Recommendation - **Add or expand disk on kai-server** (physical - homelab box, not a cloud resize). The durable fix. - Software levers that buy time but do not replace disk: an image retention/GC policy (a chunk of the 130 GiB is fleet-roll churn and eval-lane image duplication - see coilysiren/inbox#382 on lane sprawl); shrinking the 32 GiB swapfile if RAM headroom allows; journald vacuum. ## Acceptance - A capacity decision recorded: add disk, or accept riding the critical line with the reclaim jobs holding it. - If disk is added, the new size and the resulting utilization recorded. - Boundaries: PVC/image removal and any hardware change are attended operator actions - the guarded surface has no delete/exec.

coilyco-ops commented

2026-08-19 11:42:51 +00:00

Author

Owner

Sibling from the same disk-pressure pass: #871 covers ser8, which has the opposite shape (a monotonic climb rather than kai-server's flat oscillation). Different problem, different node, filed separately.

Sibling from the same disk-pressure pass: **#871** covers ser8, which has the opposite shape (a monotonic climb rather than kai-server's flat oscillation). Different problem, different node, filed separately.

coilyco-ops commented

2026-08-25 05:46:42 +00:00

Author

Owner

Live-state check while triaging the disk cluster of issues. The numbers this was filed against have moved:

kai-server  /      480 GB   316 GB used   140 GB free   70.86%   status: ok
ser8        /      980 GB   421 GB used   518 GB free   44.90%

kai-server's DiskPressure condition has been False since 2026-08-23T00:35:35Z. Neither host is near a threshold.

Not closing this. A capacity-planning or attribution issue is not resolved by the current reading being comfortable, and the question it asks stays valid. But it should be worked from these numbers rather than the ones in its title, and it is not urgent.

One input that changed: the three orphaned local-path PVs stuck Released for 82 days are now deleted (#863, #901). They held no data on disk, so this reclaimed nothing, but they no longer distort a volume inventory.

Live-state check while triaging the disk cluster of issues. The numbers this was filed against have moved: ``` kai-server / 480 GB 316 GB used 140 GB free 70.86% status: ok ser8 / 980 GB 421 GB used 518 GB free 44.90% ``` kai-server's `DiskPressure` condition has been `False` since `2026-08-23T00:35:35Z`. Neither host is near a threshold. Not closing this. A capacity-planning or attribution issue is not resolved by the current reading being comfortable, and the question it asks stays valid. But it should be worked from these numbers rather than the ones in its title, and it is not urgent. One input that changed: the three orphaned local-path PVs stuck `Released` for 82 days are now deleted (#863, #901). They held no data on disk, so this reclaimed nothing, but they no longer distort a volume inventory.

coilyco-ops added the

role/sysadmin

priority

P1

labels

2026-08-25 08:02:07 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:58 +00:00

coilyco-ops referenced this issue

2026-08-28 21:13:35 +00:00

kai-server rootfs is at 82.8 percent, past the warn threshold, with local-path PVC data at 129 GB the largest consumer #966

coilyco-ops referenced this issue

2026-08-28 21:14:03 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops commented

2026-08-28 21:15:25 +00:00

Author

Owner

Current reading, and this is now the standing kai-server capacity issue

Measured 2026-08-28:

total   514.9 GB
used    380.8 GB
free    107.9 GB
percent  77.9

77.9 percent, against the "chronically at the 85 percent critical line" this issue was filed on. About 45 GB has been reclaimed since #966 measured 82.8 percent yesterday.

The title overstates the present condition. The node is not at the critical line today, and I would rather record that here than let the backlog assert something untrue.

Why this stays open when the snapshots closed

Closed today as stale point-in-time readings, with this issue named as their successor:

#966 - kai-server at 82.8 percent, past warn
#846 - local-path 163 GB and three PVs stuck Released. All 43 PVs now Bound, zero Released

Those were measurements. This is the capacity-planning question, which is still unanswered: kai-server has no headroom plan, and the 77.9 percent is a lower reading rather than a solved problem. The node has repeatedly climbed back.

Not established

What reclaimed the ~45 GB. I did not find the change, and it is inside the range ordinary image and runner churn produces. So this is not evidence that anything structural improved, and the growth vectors #966 identified, local-path PVC data at 129 GB and the image store at 55 GB, were not re-measured.

Two related and still open: #624, Forgejo upload staging exhausting kai-server disk, and #707, disk-pressure budgets and a guarded CI circuit breaker. Both are mechanisms rather than measurements, which is what this issue actually needs.

## Current reading, and this is now the standing kai-server capacity issue Measured 2026-08-28: ``` total 514.9 GB used 380.8 GB free 107.9 GB percent 77.9 ``` **77.9 percent**, against the "chronically at the 85 percent critical line" this issue was filed on. About 45 GB has been reclaimed since #966 measured 82.8 percent yesterday. The title overstates the present condition. The node is not at the critical line today, and I would rather record that here than let the backlog assert something untrue. ### Why this stays open when the snapshots closed Closed today as stale point-in-time readings, with this issue named as their successor: * #966 - kai-server at 82.8 percent, past warn * #846 - local-path 163 GB and three PVs stuck Released. All 43 PVs now `Bound`, zero Released Those were measurements. This is the capacity-planning question, which is still unanswered: kai-server has no headroom plan, and the 77.9 percent is a lower reading rather than a solved problem. The node has repeatedly climbed back. ### Not established What reclaimed the ~45 GB. I did not find the change, and it is inside the range ordinary image and runner churn produces. So this is not evidence that anything structural improved, and the growth vectors #966 identified, local-path PVC data at 129 GB and the image store at 55 GB, were not re-measured. Two related and still open: #624, Forgejo upload staging exhausting kai-server disk, and #707, disk-pressure budgets and a guarded CI circuit breaker. Both are mechanisms rather than measurements, which is what this issue actually needs.

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops referenced this issue

2026-08-29 04:22:25 +00:00

forgejo-runner-recycle restarts all 14 deploy runners simultaneously, producing a ~45 MB/s disk burst #840

coilyco-ops referenced this issue

2026-08-29 04:40:42 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops referenced this issue

2026-08-29 04:42:59 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops referenced this issue

2026-08-29 04:43:32 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops commented

2026-08-29 04:44:04 +00:00

Author

Owner

Hold the disk purchase. This issue's "no large safe cleanup target" line is contested, and the contest is now measured.

Raised because this is the issue a capacity decision would be taken from, and it currently carries a claim that #846 contradicts. #846 was closed on 2026-08-28 and reopened today after Portia (director seat) verified the close did not hold. Whoever reads this one should not price hardware without reading that one.

The disputed bytes

This issue's measured table lists, under Persistent data:

local-path PVCs >=74 GiB - forgejo-data 38.5+, registry-data 13, runner DinD caches 13, runner data 7

Measured on kai-server 2026-08-29 ~04:20Z, every one of those runner claims is Bound with pod_mounts: []:

docker-lib-forgejo-runner-build-0         5,995,282,432
data-forgejo-runner-2                     2,048,012,288
data-forgejo-runner-1                     1,956,208,640
data-forgejo-runner-0                     1,951,072,256
data-forgejo-runner-3                     1,921,888,256
data-forgejo-runner-flight-deck-1           719,470,592
data-forgejo-runner-flight-deck-0           614,350,848
data-forgejo-runner-deploy-0                 85,958,656
data-forgejo-runner-bridge-0                 42,893,312
data-forgejo-runner-build-flight-deck-0      42,143,744
data-forgejo-runner-build-0                  42,135,552
                                         --------------
                                         15,419,416,576   (~15.4 GB)

No pod, no StatefulSet, no consumer. docker-lib-forgejo-runner-build-0, the largest at 5.99 GB, belongs to a StatefulSet at spec.replicas: 0. These are not caches in use. They are orphaned state from runner generations that no longer exist.

Being fair to this issue rather than just overturning it

Most of that is not free, and #846 does not say so. 9.21 GB of the 15.4 GB is data-forgejo-runner-{0..3} and -flight-deck-{0,1}, the scaled-zero kai-server general runners that #693 rollout step 7 explicitly says to retain until a stated rollback window closes. That window has never been stated, which is a decision waiting rather than a cleanup available.

So the honest figure is:

~6.2 GB genuinely orphaned and reclaimable now, chiefly the 5.99 GB build cache.
~9.21 GB reclaimable the moment someone closes the #693 rollback window, which costs a decision rather than hardware.

Neither 13 + 7 GiB of "persistent data" nor a flat 16 GB of "orphaned" is right. Both issues were wrong in opposite directions, and the one that survived the sweep is the one that recommends spending money.

The other number in this issue worth refreshing

Total tracked local-path storage, complete host scan today, not a lower bound:

/var/lib/rancher/k3s/storage   129,925,009,408   (129.93 GB)
entries_scanned: 354,085   complete: true   timed_out: false

And forgejo-data alone is 97,666,781,184 bytes (97.67 GB) against a 21.5 GB PVC request, with attachments at 55.94 GB inside it. This issue lists forgejo-data at "38.5+". That is low by a factor of two and a half, and it is the single largest consumer on the node by a wide margin.

The capacity conversation is mostly a Forgejo attachments-retention conversation. #905 owns attachments, #846 owns the overshoot, #868 owns the unmounted-PVC decision.

What I am asking for

Not a close, and not a reversal. Do not take the buy-a-disk decision from this issue's current table, because two of its lines are measurably wrong and the correction moves the answer. Re-anchor against the numbers above, settle the #693 retention window, and the reclaim available without hardware is roughly 15 GB before anyone touches attachments.

No change made. Every number here is a read.

## Hold the disk purchase. This issue's "no large safe cleanup target" line is contested, and the contest is now measured. Raised because **this is the issue a capacity decision would be taken from**, and it currently carries a claim that #846 contradicts. #846 was closed on 2026-08-28 and reopened today after Portia (director seat) verified the close did not hold. Whoever reads this one should not price hardware without reading that one. ### The disputed bytes This issue's measured table lists, under **Persistent data**: > local-path PVCs >=74 GiB - `forgejo-data` 38.5+, `registry-data` 13, **runner DinD caches 13, runner data 7** Measured on kai-server 2026-08-29 ~04:20Z, every one of those runner claims is `Bound` with **`pod_mounts: []`**: ``` docker-lib-forgejo-runner-build-0 5,995,282,432 data-forgejo-runner-2 2,048,012,288 data-forgejo-runner-1 1,956,208,640 data-forgejo-runner-0 1,951,072,256 data-forgejo-runner-3 1,921,888,256 data-forgejo-runner-flight-deck-1 719,470,592 data-forgejo-runner-flight-deck-0 614,350,848 data-forgejo-runner-deploy-0 85,958,656 data-forgejo-runner-bridge-0 42,893,312 data-forgejo-runner-build-flight-deck-0 42,143,744 data-forgejo-runner-build-0 42,135,552 -------------- 15,419,416,576 (~15.4 GB) ``` No pod, no StatefulSet, no consumer. `docker-lib-forgejo-runner-build-0`, the largest at 5.99 GB, belongs to a StatefulSet at `spec.replicas: 0`. **These are not caches in use. They are orphaned state from runner generations that no longer exist.** ### Being fair to this issue rather than just overturning it **Most of that is not free, and #846 does not say so.** 9.21 GB of the 15.4 GB is `data-forgejo-runner-{0..3}` and `-flight-deck-{0,1}`, the scaled-zero kai-server general runners that `#693` rollout step 7 explicitly says to **retain** until a stated rollback window closes. That window has never been stated, which is a decision waiting rather than a cleanup available. So the honest figure is: * **~6.2 GB genuinely orphaned and reclaimable now**, chiefly the 5.99 GB build cache. * **~9.21 GB reclaimable the moment someone closes the #693 rollback window**, which costs a decision rather than hardware. Neither 13 + 7 GiB of "persistent data" nor a flat 16 GB of "orphaned" is right. **Both issues were wrong in opposite directions, and the one that survived the sweep is the one that recommends spending money.** ### The other number in this issue worth refreshing Total tracked local-path storage, complete host scan today, not a lower bound: ``` /var/lib/rancher/k3s/storage 129,925,009,408 (129.93 GB) entries_scanned: 354,085 complete: true timed_out: false ``` And `forgejo-data` alone is **97,666,781,184 bytes (97.67 GB) against a 21.5 GB PVC request**, with attachments at 55.94 GB inside it. This issue lists `forgejo-data` at "38.5+". That is low by a factor of two and a half, and it is the single largest consumer on the node by a wide margin. **The capacity conversation is mostly a Forgejo attachments-retention conversation.** `#905` owns attachments, `#846` owns the overshoot, `#868` owns the unmounted-PVC decision. ### What I am asking for Not a close, and not a reversal. **Do not take the buy-a-disk decision from this issue's current table**, because two of its lines are measurably wrong and the correction moves the answer. Re-anchor against the numbers above, settle the #693 retention window, and the reclaim available without hardware is roughly 15 GB before anyone touches attachments. No change made. Every number here is a read.

coilyco-ops added this to the Storage and capacity milestone

2026-08-29 06:09:08 +00:00

Rows
Columns

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

Signal

30-day trend: flat and oscillating, not a leak

Where the ~388 GB is (measured, mount-corrected via node-stats)

Diagnosis

Recommendation

Acceptance

Current reading, and this is now the standing kai-server capacity issue

Why this stays open when the snapshots closed

Not established

Hold the disk purchase. This issue's "no large safe cleanup target" line is contested, and the contest is now measured.

The disputed bytes

Being fair to this issue rather than just overturning it

The other number in this issue worth refreshing

What I am asking for