k3s.service pkill -9 containerd-shim orphans container payloads on every restart, causing recurring lock-blocked crashloops (CI, o11y)

coilysiren commented

2026-06-12 11:13:08 +00:00

Owner

Summary

The recurring "CI is down again" failures and the long-running observability outage share one root cause: k3s.service carries ExecStartPre=-/usr/bin/pkill -9 containerd-shim. Every k3s restart SIGKILLs all containerd-shims, the container payload processes survive as orphans reparented to init, and they keep holding exclusive file locks on their volumes. Fresh pods after the restart can never acquire those locks and crashloop until someone manually reaps the orphans.

Evidence (2026-06-12, read-only shadow from ser8)

k3s restarted 03:11 PDT (ActiveEnterTimestamp), matching the restart age on forgejo pods.
Two orphaned dockerd pairs from the runner pods' first incarnation (PIDs 1366354/1366725 and 1368189/1368559, etime ~117m at observation) still hold the dind boltdb. Runner docker-lib is an emptyDir, so container restarts within the pod reuse it, and every new dind hangs on waiting for response from boltdb open then times out. forgejo-runner-0 and -1 are 0/2 CrashLoopBackOff (29-30 restarts).
Same mechanism on the observability stack from an earlier k3s restart (~5.4d ago): orphaned victoria-metrics (PID 15980) holds /storage/flock.lock (45d of crashloops, 521 restarts), orphaned clickhouse-server (PIDs 22658/22678) holds the ClickHouse status file (20d, 806 restarts, exit 76), and a leaked listener holds :9100 blocking node-exporter (25d, 529 restarts).
Node itself healthy: no memory/disk/PID pressure.

Fixes

Object-level - reap the orphans (the two dockerd pairs, vmsingle, clickhouse, the :9100 holder), then delete the crashlooping pods so they reschedule onto free locks. A reboot achieves the same wholesale.
Confirm before reaping - orphans should show PPID 1 (ps -o ppid= -p 1366725 1368559), and fuser on the emptyDir containerd/daemon/meta.db should name them.
Meta-fix - remove or replace the pkill -9 containerd-shim ExecStartPre in k3s.service. If it exists to clear some earlier hang, the correct shape is k3s-killall.sh semantics (kills whole container cgroups, payloads included) or letting pods survive the restart. Shim-only SIGKILL is precisely the orphan generator.
Fold into the k3s.service drift cleanup in #306 - both findings point at the unit file being hand-drifted from stock.

#297, #288 (CI not running), #308 (kai-server unhealthy umbrella), #306 (k3s.service stale checkout), #304 (runner PVCs stranded, closed).
An agent on kai-server is actively remediating today. This issue records the root-cause analysis and the meta-fix so the recurrence stops.

## Summary The recurring "CI is down again" failures and the long-running observability outage share one root cause: `k3s.service` carries `ExecStartPre=-/usr/bin/pkill -9 containerd-shim`. Every k3s restart SIGKILLs all containerd-shims, the container payload processes survive as orphans reparented to init, and they keep holding exclusive file locks on their volumes. Fresh pods after the restart can never acquire those locks and crashloop until someone manually reaps the orphans. ## Evidence (2026-06-12, read-only shadow from ser8) - k3s restarted 03:11 PDT (`ActiveEnterTimestamp`), matching the restart age on forgejo pods. - Two orphaned dockerd pairs from the runner pods' first incarnation (PIDs 1366354/1366725 and 1368189/1368559, etime ~117m at observation) still hold the dind boltdb. Runner `docker-lib` is an emptyDir, so container restarts within the pod reuse it, and every new dind hangs on `waiting for response from boltdb open` then times out. `forgejo-runner-0` and `-1` are 0/2 CrashLoopBackOff (29-30 restarts). - Same mechanism on the observability stack from an earlier k3s restart (~5.4d ago): orphaned `victoria-metrics` (PID 15980) holds `/storage/flock.lock` (45d of crashloops, 521 restarts), orphaned `clickhouse-server` (PIDs 22658/22678) holds the ClickHouse `status` file (20d, 806 restarts, exit 76), and a leaked listener holds `:9100` blocking node-exporter (25d, 529 restarts). - Node itself healthy: no memory/disk/PID pressure. ## Fixes 1. Object-level - reap the orphans (the two dockerd pairs, vmsingle, clickhouse, the `:9100` holder), then delete the crashlooping pods so they reschedule onto free locks. A reboot achieves the same wholesale. 2. Confirm before reaping - orphans should show PPID 1 (`ps -o ppid= -p 1366725 1368559`), and `fuser` on the emptyDir `containerd/daemon/meta.db` should name them. 3. Meta-fix - remove or replace the `pkill -9 containerd-shim` ExecStartPre in `k3s.service`. If it exists to clear some earlier hang, the correct shape is `k3s-killall.sh` semantics (kills whole container cgroups, payloads included) or letting pods survive the restart. Shim-only SIGKILL is precisely the orphan generator. 4. Fold into the `k3s.service` drift cleanup in #306 - both findings point at the unit file being hand-drifted from stock. ## Related - #297, #288 (CI not running), #308 (kai-server unhealthy umbrella), #306 (k3s.service stale checkout), #304 (runner PVCs stranded, closed). - An agent on kai-server is actively remediating today. This issue records the root-cause analysis and the meta-fix so the recurrence stops.

coilysiren commented

2026-06-12 11:14:23 +00:00

Author

Owner

Recovery confirmed from the ser8 read-only watch: both runner pods 2/2 Running as fresh incarnations at 2026-06-12T11:13:23Z, and the runner log shows real task pickup and execution (tasks 271, 272 for coilyco-gaming/eco-app, with clean job-network teardown). The dormant queue from the outage window is draining.

The object-level fix is done. Items 3 and 4 (remove or replace the pkill -9 containerd-shim ExecStartPre, fold into the #306 unit-file cleanup) remain open, and they are what stops the recurrence.

Recovery confirmed from the ser8 read-only watch: both runner pods 2/2 Running as fresh incarnations at 2026-06-12T11:13:23Z, and the runner log shows real task pickup and execution (tasks 271, 272 for coilyco-gaming/eco-app, with clean job-network teardown). The dormant queue from the outage window is draining. The object-level fix is done. Items 3 and 4 (remove or replace the `pkill -9 containerd-shim` ExecStartPre, fold into the #306 unit-file cleanup) remain open, and they are what stops the recurrence.

coilysiren referenced this issue

2026-06-12 11:20:02 +00:00

Evacuate CI runners from kai-server to the ser8 cluster #314

coilysiren referenced this issue

2026-06-12 11:20:02 +00:00

ser8 read-only watcher: alert on the lock-class crashloop signature on kai-server #315

coilysiren referenced this issue

2026-06-12 11:20:02 +00:00

Orphan-reaper for kai-server: reap PPID-1 container payloads holding stale locks #316

coilysiren referenced this issue

2026-06-12 11:20:02 +00:00

dind init-wipe: clear the emptyDir docker-lib on runner pod start so CI self-heals #317