forgejo CI outage: runner local-path PVCs stranded on dead WSL node (fixed + prevention) #304

New issue

Closed

opened 2026-06-12 09:13:31 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-12 09:13:31 +00:00

Owner

Summary

Forgejo CI was fully down for ~6 days (no runners able to schedule). Root cause: the forgejo-runner StatefulSet's local-path PVCs (data-forgejo-runner-0, data-forgejo-runner-1) were provisioned on kai-desktop-tower-wsl and pinned there via kubernetes.io/hostname node affinity. That node went NotReady/unreachable (and kai-macbook-pro-vm was cordoned), leaving only kai-server Ready. kai-server then rejected forgejo-runner-0 with volume node affinity conflict, and StatefulSet ordering meant forgejo-runner-1 never started. Net: 0/2 runners, every Actions job queued forever. 1,533 FailedScheduling events over 5d7h.

Fix applied (2026-06-12)

Scaled forgejo-runner to 0, deleted the stranded PVCs (PVs cascaded, reclaimPolicy=Delete), scaled back to 2 → local-path re-provisioned on kai-server. Both runners now 2/2 Running and registered (labels [docker], v12.10.1); backlog draining.
Deleted the two ghost nodes kai-desktop-tower-wsl and kai-macbook-pro-vm. Cluster is single-node kai-server again.

Prevention (the real meta-fix)

Ephemeral WSL/VM nodes must not be allowed to host durable workloads on this single-node cluster, or local-path will strand them again the moment the node drops off. Options:

Don't join transient WSL/VM nodes to the kai-server k3s cluster at all, or
Taint them NoSchedule on join so local-path never provisions a durable PV there, or
Pin stateful workloads to kai-server with a nodeSelector / required node affinity.

Document the trap and chosen guard in docs/k3s-deploy-notes.md §7.

## Summary Forgejo CI was fully down for ~6 days (no runners able to schedule). Root cause: the `forgejo-runner` StatefulSet's `local-path` PVCs (`data-forgejo-runner-0`, `data-forgejo-runner-1`) were provisioned on `kai-desktop-tower-wsl` and pinned there via `kubernetes.io/hostname` node affinity. That node went `NotReady`/unreachable (and `kai-macbook-pro-vm` was cordoned), leaving only `kai-server` `Ready`. `kai-server` then rejected `forgejo-runner-0` with `volume node affinity conflict`, and StatefulSet ordering meant `forgejo-runner-1` never started. Net: 0/2 runners, every Actions job queued forever. 1,533 `FailedScheduling` events over 5d7h. ## Fix applied (2026-06-12) - Scaled `forgejo-runner` to 0, deleted the stranded PVCs (PVs cascaded, `reclaimPolicy=Delete`), scaled back to 2 → local-path re-provisioned on `kai-server`. Both runners now `2/2 Running` and registered (labels `[docker]`, v12.10.1); backlog draining. - Deleted the two ghost nodes `kai-desktop-tower-wsl` and `kai-macbook-pro-vm`. Cluster is single-node `kai-server` again. ## Prevention (the real meta-fix) Ephemeral WSL/VM nodes must not be allowed to host durable workloads on this single-node cluster, or local-path will strand them again the moment the node drops off. Options: - Don't join transient WSL/VM nodes to the kai-server k3s cluster at all, or - Taint them `NoSchedule` on join so local-path never provisions a durable PV there, or - Pin stateful workloads to `kai-server` with a nodeSelector / required node affinity. Document the trap and chosen guard in `docs/k3s-deploy-notes.md` §7.

coilysiren referenced this issue from a commit

2026-06-12 09:29:39 +00:00

docs(k3s): record 2026-06-12 forgejo CI recovery + tap-writer trap

coilysiren closed this issue

2026-06-12 09:29:39 +00:00

coilysiren referenced this issue

2026-06-12 11:13:08 +00:00

k3s.service pkill -9 containerd-shim orphans container payloads on every restart, causing recurring lock-blocked crashloops (CI, o11y) #311