coilyco-flight-deck/infrastructure

Watch

forgejo-runner-recycle restarts all 14 deploy runners simultaneously, producing a ~45 MB/s disk burst #840

New issue

Open

opened 2026-08-15 16:52:51 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 16:52:51 +00:00

Owner

Summary

The nightly forgejo-runner-recycle CronJob restarts every scratch-recycled runner StatefulSet at once. Fourteen dind daemons then cold-start together and saturate the node's NVMe for several minutes. This is the direct cause of a user-visible slowdown reported on 2026-08-15.

Timeline, 2026-08-15

16:15:00Z forgejo-runner-recycle-29780175-wqqlm runs and succeeds
16:20:56Z to 16:21:18Z all 14 forgejo-runner-deploy-* StatefulSets restart, within a 22 second window
16:33 to 16:38Z NVMe write burst
16:39:43Z burst over

Disk measurements

Block device counters on nvme0n1p7 across the burst window (about 270 seconds):

writes: 12.1 GB, roughly 45 MB/s sustained
reads: 3.9 GB, roughly 15 MB/s
device busy time: about 84% utilization

IO pressure stall on the node, full line:

during burst   avg10 2.21   avg60 6.86   avg300 4.69
after burst    avg10 2.12   avg60 3.47   avg300 4.11

Following window measured only 2.2 MB/s, confirming the burst was the runner restart and not steady state.

Why it hurts

Everything on kai-server shares this disk, including Forgejo's git repositories and its Postgres. During the burst, Forgejo's own request handling stayed under one second for all 17,430 requests it served that hour, but git operations and anything else touching disk pay the queueing cost.

Suggested direction

The recycle itself is correct and the 10Gi scratch cap that motivated it should stay. The problem is that it is simultaneous. Options:

Stagger the restarts, for example a short sleep between each StatefulSet rollout so dind cold starts do not overlap
Recycle in batches rather than all fourteen at once
Spread pools across a wider window instead of one 16:15Z tick

Acceptance

A recycle run no longer drives sustained NVMe utilization above a chosen threshold, and IO pressure stall full avg60 stays in single digits through the window.

## Summary The nightly `forgejo-runner-recycle` CronJob restarts every scratch-recycled runner StatefulSet at once. Fourteen dind daemons then cold-start together and saturate the node's NVMe for several minutes. This is the direct cause of a user-visible slowdown reported on 2026-08-15. ## Timeline, 2026-08-15 - 16:15:00Z `forgejo-runner-recycle-29780175-wqqlm` runs and succeeds - 16:20:56Z to 16:21:18Z all 14 `forgejo-runner-deploy-*` StatefulSets restart, within a 22 second window - 16:33 to 16:38Z NVMe write burst - 16:39:43Z burst over ## Disk measurements Block device counters on `nvme0n1p7` across the burst window (about 270 seconds): - writes: 12.1 GB, roughly 45 MB/s sustained - reads: 3.9 GB, roughly 15 MB/s - device busy time: about 84% utilization IO pressure stall on the node, `full` line: ``` during burst avg10 2.21 avg60 6.86 avg300 4.69 after burst avg10 2.12 avg60 3.47 avg300 4.11 ``` Following window measured only 2.2 MB/s, confirming the burst was the runner restart and not steady state. ## Why it hurts Everything on kai-server shares this disk, including Forgejo's git repositories and its Postgres. During the burst, Forgejo's own request handling stayed under one second for all 17,430 requests it served that hour, but git operations and anything else touching disk pay the queueing cost. ## Suggested direction The recycle itself is correct and the 10Gi scratch cap that motivated it should stay. The problem is that it is simultaneous. Options: - Stagger the restarts, for example a short sleep between each StatefulSet rollout so dind cold starts do not overlap - Recycle in batches rather than all fourteen at once - Spread pools across a wider window instead of one 16:15Z tick ## Acceptance A recycle run no longer drives sustained NVMe utilization above a chosen threshold, and IO pressure stall `full` avg60 stays in single digits through the window.

coilyco-ops referenced this issue

2026-08-15 22:00:13 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:10 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:00 +00:00

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops commented

2026-08-29 04:22:25 +00:00

Author

Owner

Still live, and the blast radius doubled. Verified 2026-08-29 ~04:20Z.

The simultaneity this issue describes is unchanged on kai-server and now also exists on ser8, which did not run general pools when this was filed.

kai-server, after the 2026-08-28T16:15Z run

Twelve forgejo-runner-deploy-* pods all report age 12h2m, meaning they restarted inside the same window. Down from the fourteen in the original timeline, because the general pools left for ser8 under #693, not because anything staggered.

ser8, after the 2026-08-28T09:15Z run

Eleven runner pods, created 09:15:49Z to 09:17:24Z, a 95 second spread:

forgejo-runner-ser8-flight-deck-0 through -5
forgejo-runner-ser8-gaming-0, -1, -2
forgejo-runner-ser8-bridge-0
forgejo-runner-build-ser8-flight-deck-0

Every one carries a dind sidecar, which is the cold start that produced the original 45 MB/s burst.

So the fleet went from one 14-runner thundering herd to two herds of twelve and eleven, on two different disks, seven hours apart. The seven-hour offset is not a stagger. It is the timezone bug in #869, and it splits the herd by accident rather than by design.

What this changes about the fix

The body offers three directions: stagger the restarts, batch them, or spread pools across a wider window. The third one partly happened by accident and it is worth noticing that it helped. kai-server's herd is smaller now and its disk is the one under chronic pressure, per #870.

What has not happened is any deliberate control. A recycle still restarts every labelled StatefulSet in its cluster at once, and the ser8 herd is eleven dind cold starts against a disk that also carries the ClickHouse telemetry store.

Not re-measured

No block-device counters or IO pressure stalls were taken for either cluster today. The original 45 MB/s, 84% busy, and full avg60 6.86 figures are from 2026-08-15 on kai-server and should not be read as current. What is verified today is the restart topology, not its cost. The acceptance threshold in this issue still needs a fresh measurement on ser8 before anyone can say whether eleven concurrent dind starts hurt there the way fourteen hurt on kai-server.

## Still live, and the blast radius doubled. Verified 2026-08-29 ~04:20Z. The simultaneity this issue describes is unchanged on kai-server and **now also exists on ser8**, which did not run general pools when this was filed. ### kai-server, after the 2026-08-28T16:15Z run Twelve `forgejo-runner-deploy-*` pods all report age `12h2m`, meaning they restarted inside the same window. Down from the fourteen in the original timeline, because the general pools left for ser8 under #693, not because anything staggered. ### ser8, after the 2026-08-28T09:15Z run Eleven runner pods, created **09:15:49Z to 09:17:24Z**, a 95 second spread: * `forgejo-runner-ser8-flight-deck-0` through `-5` * `forgejo-runner-ser8-gaming-0`, `-1`, `-2` * `forgejo-runner-ser8-bridge-0` * `forgejo-runner-build-ser8-flight-deck-0` Every one carries a `dind` sidecar, which is the cold start that produced the original 45 MB/s burst. **So the fleet went from one 14-runner thundering herd to two herds of twelve and eleven, on two different disks, seven hours apart.** The seven-hour offset is not a stagger. It is the timezone bug in #869, and it splits the herd by accident rather than by design. ### What this changes about the fix The body offers three directions: stagger the restarts, batch them, or spread pools across a wider window. **The third one partly happened by accident and it is worth noticing that it helped.** kai-server's herd is smaller now and its disk is the one under chronic pressure, per #870. What has not happened is any deliberate control. A recycle still restarts every labelled StatefulSet in its cluster at once, and the ser8 herd is eleven dind cold starts against a disk that also carries the ClickHouse telemetry store. ### Not re-measured No block-device counters or IO pressure stalls were taken for either cluster today. The original 45 MB/s, 84% busy, and `full avg60 6.86` figures are from 2026-08-15 on kai-server and **should not be read as current**. What is verified today is the restart topology, not its cost. The acceptance threshold in this issue still needs a fresh measurement on ser8 before anyone can say whether eleven concurrent dind starts hurt there the way fourteen hurt on kai-server.

coilyco-ops referenced this issue

2026-08-29 04:22:27 +00:00

forgejo-runner-recycle comment says 09:15 UTC, the job actually fires 09:15 local #869

coilyco-ops referenced this issue

2026-08-29 04:22:59 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops added this to the Forgejo runners milestone

2026-08-29 05:40:53 +00:00