scale up forgejo CI #560

New issue

Open

opened 2026-07-14 16:44:31 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-07-14 16:44:31 +00:00

Owner

pending an analysis of whats going on with disk pressure

and a memory usage audit

I want forgejo CI to have 6 standard runners, 2 deployers, and 2 tap writers

pending an analysis of whats going on with disk pressure and a memory usage audit I want forgejo CI to have 6 standard runners, 2 deployers, and 2 tap writers

coilyco-ops commented

2026-07-15 07:25:24 +00:00

Member

Current live read says do not scale up the kai-server build runners yet.

Snapshot from 2026-07-15 00:10 PDT:

/ is 480G total, 365G used, 91G available, 81%.
Kubelet sees nodefs/imagefs on the same filesystem: 479Gi capacity, 366Gi used, 88Gi available.
Image storage itself is only 18Gi, so this remains a general rootfs pressure problem, not an image-cache problem.
Inodes are fine at 9% used.
Node taints are clear and DiskPressure=False.
No fresh FreeDiskSpaceFailed after the runner rollout. Last observed event was about 19m old at the time of the check.

The scale-up blocker is the margin, not current health. The node only has about 16Gi of buffer above kubelet’s likely 85% image-GC high watermark, and one runner was just evicted because docker-lib exceeded the new 12Gi emptyDir limit:

Usage of EmptyDir volume "docker-lib" exceeds the limit "12Gi".

A fifth kai-server build runner would add roughly another 13Gi of worst-case ephemeral limit. That leaves almost no margin, while the current four runners can already push the node back toward pressure if several builds fill DinD at once.

Recommended path for this issue: do not increase kai-server replicas until one of these lands first:

Move build runners to ser8 or another node with isolated disk.
Reduce per-job DinD footprint.
Add reliable job-end Docker pruning / cleanup.
Add materially more rootfs runway on kai-server.

The 12Gi cap is doing the right protective thing now. Scaling before reducing scratch usage or adding disk would trade queue time for cancelled/evicted jobs and another node-pressure loop.

Current live read says do not scale up the kai-server build runners yet. Snapshot from 2026-07-15 00:10 PDT: * `/` is 480G total, 365G used, 91G available, 81%. * Kubelet sees nodefs/imagefs on the same filesystem: 479Gi capacity, 366Gi used, 88Gi available. * Image storage itself is only 18Gi, so this remains a general rootfs pressure problem, not an image-cache problem. * Inodes are fine at 9% used. * Node taints are clear and `DiskPressure=False`. * No fresh `FreeDiskSpaceFailed` after the runner rollout. Last observed event was about 19m old at the time of the check. The scale-up blocker is the margin, not current health. The node only has about 16Gi of buffer above kubelet’s likely 85% image-GC high watermark, and one runner was just evicted because `docker-lib` exceeded the new 12Gi emptyDir limit: `Usage of EmptyDir volume "docker-lib" exceeds the limit "12Gi".` A fifth kai-server build runner would add roughly another 13Gi of worst-case ephemeral limit. That leaves almost no margin, while the current four runners can already push the node back toward pressure if several builds fill DinD at once. Recommended path for this issue: do not increase kai-server replicas until one of these lands first: * Move build runners to ser8 or another node with isolated disk. * Reduce per-job DinD footprint. * Add reliable job-end Docker pruning / cleanup. * Add materially more rootfs runway on kai-server. The 12Gi cap is doing the right protective thing now. Scaling before reducing scratch usage or adding disk would trade queue time for cancelled/evicted jobs and another node-pressure loop.