scale up forgejo CI

coilysiren commented

2026-07-14 16:44:31 +00:00

Owner

pending an analysis of whats going on with disk pressure

and a memory usage audit

I want forgejo CI to have 6 standard runners, 2 deployers, and 2 tap writers

pending an analysis of whats going on with disk pressure and a memory usage audit I want forgejo CI to have 6 standard runners, 2 deployers, and 2 tap writers

coilyco-ops commented

2026-07-15 07:25:24 +00:00

Owner

Current live read says do not scale up the kai-server build runners yet.

Snapshot from 2026-07-15 00:10 PDT:

/ is 480G total, 365G used, 91G available, 81%.
Kubelet sees nodefs/imagefs on the same filesystem: 479Gi capacity, 366Gi used, 88Gi available.
Image storage itself is only 18Gi, so this remains a general rootfs pressure problem, not an image-cache problem.
Inodes are fine at 9% used.
Node taints are clear and DiskPressure=False.
No fresh FreeDiskSpaceFailed after the runner rollout. Last observed event was about 19m old at the time of the check.

The scale-up blocker is the margin, not current health. The node only has about 16Gi of buffer above kubelet’s likely 85% image-GC high watermark, and one runner was just evicted because docker-lib exceeded the new 12Gi emptyDir limit:

Usage of EmptyDir volume "docker-lib" exceeds the limit "12Gi".

A fifth kai-server build runner would add roughly another 13Gi of worst-case ephemeral limit. That leaves almost no margin, while the current four runners can already push the node back toward pressure if several builds fill DinD at once.

Recommended path for this issue: do not increase kai-server replicas until one of these lands first:

Move build runners to ser8 or another node with isolated disk.
Reduce per-job DinD footprint.
Add reliable job-end Docker pruning / cleanup.
Add materially more rootfs runway on kai-server.

The 12Gi cap is doing the right protective thing now. Scaling before reducing scratch usage or adding disk would trade queue time for cancelled/evicted jobs and another node-pressure loop.

Current live read says do not scale up the kai-server build runners yet. Snapshot from 2026-07-15 00:10 PDT: * `/` is 480G total, 365G used, 91G available, 81%. * Kubelet sees nodefs/imagefs on the same filesystem: 479Gi capacity, 366Gi used, 88Gi available. * Image storage itself is only 18Gi, so this remains a general rootfs pressure problem, not an image-cache problem. * Inodes are fine at 9% used. * Node taints are clear and `DiskPressure=False`. * No fresh `FreeDiskSpaceFailed` after the runner rollout. Last observed event was about 19m old at the time of the check. The scale-up blocker is the margin, not current health. The node only has about 16Gi of buffer above kubelet’s likely 85% image-GC high watermark, and one runner was just evicted because `docker-lib` exceeded the new 12Gi emptyDir limit: `Usage of EmptyDir volume "docker-lib" exceeds the limit "12Gi".` A fifth kai-server build runner would add roughly another 13Gi of worst-case ephemeral limit. That leaves almost no margin, while the current four runners can already push the node back toward pressure if several builds fill DinD at once. Recommended path for this issue: do not increase kai-server replicas until one of these lands first: * Move build runners to ser8 or another node with isolated disk. * Reduce per-job DinD footprint. * Add reliable job-end Docker pruning / cleanup. * Add materially more rootfs runway on kai-server. The 12Gi cap is doing the right protective thing now. Scaling before reducing scratch usage or adding disk would trade queue time for cancelled/evicted jobs and another node-pressure loop.

coilyco-ops added the

priority

P1

label

2026-07-16 00:48:21 +00:00

coilyco-ops commented

2026-07-23 04:28:34 +00:00

Owner

Incident update, 2026-07-22 PDT: kai-server entered DiskPressure and evicted the Forgejo application pod plus three general runners. The forgejo Service had zero endpoints and returned 503. Kubelet reported repeated NodeHasDiskPressure transitions over 9 days and FreeDiskSpaceFailed over 11 days; image GC attempted to free about 24 GB but found only about 462 MB eligible. Evictions raised rootfs free space to 75 GB on the 480 GB filesystem. After kubelet pressure-transition grace elapsed, DiskPressure cleared, the taint was removed, Forgejo rescheduled, endpoint 10.42.0.169:3000 became ready, and the API recovered. This reinforces the existing recommendation: do not scale kai-server runners up until runner scratch is isolated, pruned reliably, or moved off-node.

coilyco-ops commented

2026-07-23 04:30:49 +00:00

Owner

Additional root cause: the daily forgejo-runner-recycle CronJob has failed every observed run. Its image is rancher/kubectl:v1.31.4, but the pod command is /bin/sh -c; that image contains no /bin/sh, so each job terminates StartError before restarting either runner StatefulSet. Four failed recycle pods remain from the last two schedules. After todays eviction, the general runners recovered and free rootfs fell from 75 GiB to 62 GiB immediately, confirming runner scratch rapidly consumes the recovered margin. The recycle job needs an image/command pairing that actually starts, plus verification that its restart behavior clears the intended scratch stores.

coilyco-ops referenced this issue

2026-07-23 04:35:41 +00:00

fix: make the Forgejo runner recycle CronJob executable #587

coilyco-ops referenced this issue

2026-07-23 04:35:50 +00:00

investigate: recurring probe timeouts across stateful services #592

coilyco-ops referenced this issue

2026-07-23 04:36:21 +00:00

kai-server unhealthy: forgejo bounce, runner crashloop, missing /github/pat, signoz/metrics 500s #308

coilyco-ops referenced this issue

2026-07-23 06:35:47 +00:00

Incident: kai-server root filesystem is at 94.1% pressure #596