coilyco-flight-deck/infrastructure

Fork 0

forgejo-runner pods don't self-heal after k3s restart (dind containerd boltdb timeout -> CrashLoopBackOff) #313

New issue

Closed

opened 2026-06-12 11:15:36 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-12 11:15:36 +00:00

Owner

Summary

The two forgejo-runner-{0,1} StatefulSet pods cannot self-heal after a k3s restart. On 2026-06-12 ~03:11 PDT k3s restarted on kai-server, bouncing every pod in the cluster. Everything recovered on its own except the two dind-backed Actions runners, which sat in CrashLoopBackOff for ~55 minutes until a manual kubectl delete pod kicked them. CI was effectively down that whole window.

Root-cause chain

runner container dies with cannot ping the docker daemon ... Cannot connect to the Docker daemon at tcp://localhost:2375.
It can't reach docker because its dind sidecar never comes up. dind's containerd hangs on waiting for response from boltdb open and then failed to start containerd: timeout waiting for containerd to start (the default ~10s containerd startup timeout).
That boltdb-open stall is the classic signature of transient disk-I/O contention - exactly what you get during the I/O storm of every pod restarting at once after a k3s bounce.
forgejo-runner-tap-writer-0 (no dind sidecar) recovered fine. Only the dind runners wedged, and CrashLoopBackOff's exponential backoff kept them down well past the point the host had gone idle again.

Host was healthy throughout the stuck window: disk 57% (197G free), 18Gi RAM available, load ~2.0, zero processes in I/O-wait. The contention had cleared - the pods were just stuck in backoff.

Remediation applied

coily ops kubectl delete pod forgejo-runner-0 forgejo-runner-1 -n forgejo

Both came back 2/2 Running within ~2 minutes (3 harmless early restarts each as the runner container races ahead of dind on first boot, same pattern tap-writer shows). This confirmed the crash-loop was transient, not a persistent fault.

Proposed fix (the reliability gap)

A node-restart should not require a human to manually delete pods to restore CI. Options, roughly in order of preference:

Raise dind's containerd startup timeout so a slow boltdb open during an I/O storm doesn't trip the 10s deadline and CrashLoop. This targets the actual failure directly.
Add a readiness/startup probe + restart ordering so the runner container waits for dind to be reachable instead of dying on first ping and burning into backoff.
Consider whether the dind docker-lib EmptyDir and StatefulSet shape are still the right design vs. a host-docker or rootless-runner approach.

Also unexplained

Why k3s itself restarted at 03:11 PDT is unknown - reading sudo journalctl -u k3s requires a password not in the agent's passwordless sudo allowlist. If k3s restarts turn out to be anything but rare, that is its own investigation and raises the priority of the self-heal fix above.

Acceptance

After a k3s restart (or a manual systemctl restart k3s), both forgejo-runner-{0,1} reach 2/2 Running with no manual intervention.

## Summary The two `forgejo-runner-{0,1}` StatefulSet pods cannot self-heal after a k3s restart. On 2026-06-12 ~03:11 PDT k3s restarted on kai-server, bouncing every pod in the cluster. Everything recovered on its own **except** the two dind-backed Actions runners, which sat in `CrashLoopBackOff` for ~55 minutes until a manual `kubectl delete pod` kicked them. CI was effectively down that whole window. ## Root-cause chain - `runner` container dies with `cannot ping the docker daemon ... Cannot connect to the Docker daemon at tcp://localhost:2375`. - It can't reach docker because its `dind` sidecar never comes up. dind's containerd hangs on `waiting for response from boltdb open` and then `failed to start containerd: timeout waiting for containerd to start` (the default ~10s containerd startup timeout). - That boltdb-open stall is the classic signature of transient disk-I/O contention - exactly what you get during the I/O storm of every pod restarting at once after a k3s bounce. - `forgejo-runner-tap-writer-0` (no dind sidecar) recovered fine. Only the dind runners wedged, and CrashLoopBackOff's exponential backoff kept them down well past the point the host had gone idle again. Host was healthy throughout the stuck window: disk 57% (197G free), 18Gi RAM available, load ~2.0, zero processes in I/O-wait. The contention had cleared - the pods were just stuck in backoff. ## Remediation applied ``` coily ops kubectl delete pod forgejo-runner-0 forgejo-runner-1 -n forgejo ``` Both came back `2/2 Running` within ~2 minutes (3 harmless early restarts each as the runner container races ahead of dind on first boot, same pattern tap-writer shows). This confirmed the crash-loop was transient, not a persistent fault. ## Proposed fix (the reliability gap) A node-restart should not require a human to manually delete pods to restore CI. Options, roughly in order of preference: 1. **Raise dind's containerd startup timeout** so a slow boltdb open during an I/O storm doesn't trip the 10s deadline and CrashLoop. This targets the actual failure directly. 2. **Add a readiness/startup probe + restart ordering** so the `runner` container waits for dind to be reachable instead of dying on first ping and burning into backoff. 3. Consider whether the dind `docker-lib` EmptyDir and StatefulSet shape are still the right design vs. a host-docker or rootless-runner approach. ## Also unexplained *Why* k3s itself restarted at 03:11 PDT is unknown - reading `sudo journalctl -u k3s` requires a password not in the agent's passwordless sudo allowlist. If k3s restarts turn out to be anything but rare, that is its own investigation and raises the priority of the self-heal fix above. ## Acceptance - After a k3s restart (or a manual `systemctl restart k3s`), both `forgejo-runner-{0,1}` reach `2/2 Running` with no manual intervention.

coilysiren added the

label

2026-06-17 07:06:48 +00:00

coilysiren added

and removed

labels

2026-06-17 08:39:25 +00:00

coilysiren added

headless

and removed

labels

2026-06-18 06:16:58 +00:00

coilyco-ops commented

2026-07-08 17:10:12 +00:00

Member

🔒 Reserved by ward agent --harness codex — container engineer-codex-infrastructure-313 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-08T17:10:10Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#313 · branch issue-313 · harness codex · workflow direct-main
Run: engineer-codex-infrastructure-313 · ward v0.451.0 · dispatched 2026-07-08T17:10:10Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Issue body as seeded:

## Summary

The two `forgejo-runner-{0,1}` StatefulSet pods cannot self-heal after a k3s restart. On 2026-06-12 ~03:11 PDT k3s restarted on kai-server, bouncing every pod in the cluster. Everything recovered on its own **except** the two dind-backed Actions runners, which sat in `CrashLoopBackOff` for ~55 minutes until a manual `kubectl delete pod` kicked them. CI was effectively down that whole window.

## Root-cause chain

- `runner` container dies with `cannot ping the docker daemon ... Cannot connect to the Docker daemon at tcp://localhost:2375`.
- It can't reach docker because its `dind` sidecar never comes up. dind's containerd hangs on `waiting for response from boltdb open` and then `failed to start containerd: timeout waiting for containerd to start` (the default ~10s containerd startup timeout).
- That boltdb-open stall is the classic signature of transient disk-I/O contention - exactly what you get during the I/O storm of every pod restarting at once after a k3s bounce.
- `forgejo-runner-tap-writer-0` (no dind sidecar) recovered fine. Only the dind runners wedged, and CrashLoopBackOff's exponential backoff kept them down well past the point the host had gone idle again.

Host was healthy throughout the stuck window: disk 57% (197G free), 18Gi RAM available, load ~2.0, zero processes in I/O-wait. The contention had cleared - the pods were just stuck in backoff.

## Remediation applied

` ` `
coily ops kubectl delete pod forgejo-runner-0 forgejo-runner-1 -n forgejo
` ` `

Both came back `2/2 Running` within ~2 minutes (3 harmless early restarts each as the runner container races ahead of dind on first boot, same pattern tap-writer shows). This confirmed the crash-loop was transient, not a persistent fault.

## Proposed fix (the reliability gap)

A node-restart should not require a human to manually delete pods to restore CI. Options, roughly in order of preference:

1. **Raise dind's containerd startup timeout** so a slow boltdb open during an I/O storm doesn't

… (truncated to 2000 chars; full body is on this issue)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.451.0).

— Codex, via ward agent

🔒 Reserved by `ward agent --harness codex` — container `engineer-codex-infrastructure-313` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-08T17:10:10Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#313` · branch `issue-313` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-313` · ward `v0.451.0` · dispatched `2026-07-08T17:10:10Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). **Issue body as seeded:** ``` ## Summary The two `forgejo-runner-{0,1}` StatefulSet pods cannot self-heal after a k3s restart. On 2026-06-12 ~03:11 PDT k3s restarted on kai-server, bouncing every pod in the cluster. Everything recovered on its own **except** the two dind-backed Actions runners, which sat in `CrashLoopBackOff` for ~55 minutes until a manual `kubectl delete pod` kicked them. CI was effectively down that whole window. ## Root-cause chain - `runner` container dies with `cannot ping the docker daemon ... Cannot connect to the Docker daemon at tcp://localhost:2375`. - It can't reach docker because its `dind` sidecar never comes up. dind's containerd hangs on `waiting for response from boltdb open` and then `failed to start containerd: timeout waiting for containerd to start` (the default ~10s containerd startup timeout). - That boltdb-open stall is the classic signature of transient disk-I/O contention - exactly what you get during the I/O storm of every pod restarting at once after a k3s bounce. - `forgejo-runner-tap-writer-0` (no dind sidecar) recovered fine. Only the dind runners wedged, and CrashLoopBackOff's exponential backoff kept them down well past the point the host had gone idle again. Host was healthy throughout the stuck window: disk 57% (197G free), 18Gi RAM available, load ~2.0, zero processes in I/O-wait. The contention had cleared - the pods were just stuck in backoff. ## Remediation applied ` ` ` coily ops kubectl delete pod forgejo-runner-0 forgejo-runner-1 -n forgejo ` ` ` Both came back `2/2 Running` within ~2 minutes (3 harmless early restarts each as the runner container races ahead of dind on first boot, same pattern tap-writer shows). This confirmed the crash-loop was transient, not a persistent fault. ## Proposed fix (the reliability gap) A node-restart should not require a human to manually delete pods to restore CI. Options, roughly in order of preference: 1. **Raise dind's containerd startup timeout** so a slow boltdb open during an I/O storm doesn't ``` … (truncated to 2000 chars; full body is on this issue) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.451.0). </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-08 17:17:19 +00:00

fix: self-heal forgejo runner dind startup

coilyco-ops closed this issue

2026-07-08 17:17:19 +00:00

coilyco-ops commented

2026-07-08 17:23:19 +00:00

Member

WARD-OUTCOME: done - forgejo-runner now waits for dind before starting the daemon, so transient post-restart containerd stalls no longer cascade into CrashLoopBackOff.

This went more cleanly once I stopped treating the containerd timeout as the thing to tune and instead fixed the runner's failure mode. The only friction was the repo's local uv cache path, which wanted a writable Python install dir for pre-commit. Confidence is high, but if dind ever starts taking materially longer to come up again, the next place to look is the image-side startup path.

WARD-OUTCOME: done - forgejo-runner now waits for dind before starting the daemon, so transient post-restart containerd stalls no longer cascade into CrashLoopBackOff. This went more cleanly once I stopped treating the containerd timeout as the thing to tune and instead fixed the runner's failure mode. The only friction was the repo's local `uv` cache path, which wanted a writable Python install dir for pre-commit. Confidence is high, but if dind ever starts taking materially longer to come up again, the next place to look is the image-side startup path.

coilyco-ops referenced this issue

2026-07-10 16:58:42 +00:00

Post-reboot recovery incomplete: Forgejo Actions runners + DNS/tailnet degraded on kai-server, landing-blocking fleet burndown #538