#215 - forgejo-actions runners wedged in Terminating - no CI on main pushes - coilyco-flight-deck/infrastructure

coilysiren commented

2026-06-04 08:50:56 +00:00

Owner

Both Forgejo Actions runner pods are wedged in Terminating, so no CI task is being claimed for pushes to main.

Observed

$ coily ops kubectl get pods -n forgejo
forgejo-runner-0   1/2   Terminating   10 (5h21m ago)   5h32m
forgejo-runner-1   1/2   Terminating    6 (5h20m ago)   5h30m

Both runner pods sit at 1/2 ready, Terminating for ~5.5h with 6-10 restarts. coily ops forgejo actions task list --repo coilyco-flight-deck/infrastructure shows the newest registered run is 8eb40580 (a commit not even in current main history). Every push to main today - including #208 (02909ac) and #209 (fc50503) - has zero CI runs.

Impact

The lint + Scan for secrets jobs (.forgejo/workflows/lint.yml, trufflehog.yml) never fire, so the per-push CI verification in AGENTS.md cannot complete. Code still lands clean (pre-commit mirrors the CI lint job exactly), but the CI signal is dark.

Likely fix

Force-delete the stuck Terminating runner pods so the StatefulSet respawns them:

coily ops kubectl delete pod -n forgejo forgejo-runner-0 forgejo-runner-1 --force --grace-period=0

Then re-run a no-op push or re-trigger to confirm tasks are claimed again. If they re-wedge, check deploy/forgejo-runner.yml (the 1/2 ready suggests one sidecar - likely the dind or the runner container - is failing its probe / crash-looping; kubectl describe/logs on the new pods will say which).

Verify after fix

coily ops forgejo actions task list --repo coilyco-flight-deck/infrastructure should show lint + Scan for secrets runs for commit fc50503.

Both Forgejo Actions runner pods are wedged in `Terminating`, so no CI task is being claimed for pushes to `main`. ## Observed ``` $ coily ops kubectl get pods -n forgejo forgejo-runner-0 1/2 Terminating 10 (5h21m ago) 5h32m forgejo-runner-1 1/2 Terminating 6 (5h20m ago) 5h30m ``` Both runner pods sit at 1/2 ready, `Terminating` for ~5.5h with 6-10 restarts. `coily ops forgejo actions task list --repo coilyco-flight-deck/infrastructure` shows the newest registered run is `8eb40580` (a commit not even in current `main` history). Every push to `main` today - including #208 (`02909ac`) and #209 (`fc50503`) - has zero CI runs. ## Impact The `lint` + `Scan for secrets` jobs (`.forgejo/workflows/lint.yml`, `trufflehog.yml`) never fire, so the per-push CI verification in AGENTS.md cannot complete. Code still lands clean (pre-commit mirrors the CI lint job exactly), but the CI signal is dark. ## Likely fix Force-delete the stuck `Terminating` runner pods so the StatefulSet respawns them: ``` coily ops kubectl delete pod -n forgejo forgejo-runner-0 forgejo-runner-1 --force --grace-period=0 ``` Then re-run a no-op push or re-trigger to confirm tasks are claimed again. If they re-wedge, check `deploy/forgejo-runner.yml` (the `1/2` ready suggests one sidecar - likely the dind or the runner container - is failing its probe / crash-looping; `kubectl describe`/`logs` on the new pods will say which). ## Verify after fix `coily ops forgejo actions task list --repo coilyco-flight-deck/infrastructure` should show `lint` + `Scan for secrets` runs for commit `fc50503`.

coilysiren commented

2026-06-04 09:05:27 +00:00

Author

Owner

Update: the runner fix is blocked on a kube-apiserver outage. coily ops kubectl get pods -n forgejo succeeded ~2 min before this, but the cluster API now returns Unable to connect to the server: Bad Gateway on every call (3+ retries + the force-delete). The Bad Gateway means the ingress/tailscale proxy is up but the upstream apiserver is unreachable - so this is broader than the runners, and likely the same root cause that wedged them.

The planned fix (coily ops kubectl delete pod -n forgejo forgejo-runner-0 forgejo-runner-1 --force --grace-period=0) could not run. Once the apiserver is reachable again, retry the delete, then verify lint + Scan for secrets runs appear for fc50503.

This probably needs hands-on kai-server access (SSH kai@kai-server, check k3s/systemctl status k3s, apiserver logs) rather than remote kubectl.

Update: the runner fix is blocked on a kube-apiserver outage. `coily ops kubectl get pods -n forgejo` succeeded ~2 min before this, but the cluster API now returns `Unable to connect to the server: Bad Gateway` on every call (3+ retries + the force-delete). The Bad Gateway means the ingress/tailscale proxy is up but the upstream apiserver is unreachable - so this is broader than the runners, and likely the same root cause that wedged them. The planned fix (`coily ops kubectl delete pod -n forgejo forgejo-runner-0 forgejo-runner-1 --force --grace-period=0`) could not run. Once the apiserver is reachable again, retry the delete, then verify `lint` + `Scan for secrets` runs appear for `fc50503`. This probably needs hands-on kai-server access (SSH `kai@kai-server`, check `k3s`/`systemctl status k3s`, apiserver logs) rather than remote kubectl.

coilysiren referenced this issue

2026-06-05 22:08:08 +00:00

Stand up a self-hosted health-check service (Gatus), tailnet-reachable #232

coilysiren referenced this issue

2026-06-06 22:40:13 +00:00

tap-writer runner CrashLoopBackOff (exit 99) - missing workingDir: /data #249

coilysiren referenced this issue from a commit

2026-06-06 22:41:49 +00:00

fix(forgejo-runner): set workingDir /data on tap-writer runner

coilysiren referenced this issue

2026-06-06 22:42:03 +00:00

fix(forgejo-runner): set workingDir /data on tap-writer runner #250

coilysiren commented

2026-06-06 22:42:23 +00:00

Author

Owner

Status update (2026-06-06) - the symptom has moved on.

The apiserver Bad Gateway cleared after Kai's 06-05 power cycle of kai-server (coily ops kubectl calls succeed again). The runners are no longer Terminating either - but they still don't run, for a new reason:

forgejo-runner-0   0/2   Pending   16h
forgejo-runner-1   0/2   Pending   16h

describe reason:

FailedScheduling: 0/3 nodes available: 1 node(s) had untolerated taint
{node.kubernetes.io/unreachable}, 1 node(s) had volume node affinity
conflict, 1 node(s) were unschedulable.

Node state:

kai-server - Ready (control-plane)
kai-macbook-pro-vm - NotReady,SchedulingDisabled
kai-desktop-tower-wsl - NotReady

So the regular runners' local-path PVC is pinned to a node that is now NotReady, and they can't reschedule onto kai-server (volume node affinity conflict). The blocker is now node recovery, not the pod wedge.

Suggested fix

Either bring kai-macbook-pro-vm / kai-desktop-tower-wsl back to Ready, or (if that data is disposable) delete + recreate the stranded forgejo-runner PVC so it binds on kai-server. Both are on-box / hands-on.

Note

A separate fault, the dedicated tap-writer runner, was found during this: forgejo-runner-tap-writer-0 CrashLoopBackOff exit 99, tracked in #249 with fix PR #250. Not the same root cause as this issue.

Current diagnosis is also on o2r channel FQT7 (event 126).

**Status update (2026-06-06) - the symptom has moved on.** The apiserver `Bad Gateway` cleared after Kai's 06-05 power cycle of kai-server (`coily ops kubectl` calls succeed again). The runners are no longer `Terminating` either - but they still don't run, for a new reason: ``` forgejo-runner-0 0/2 Pending 16h forgejo-runner-1 0/2 Pending 16h ``` `describe` reason: ``` FailedScheduling: 0/3 nodes available: 1 node(s) had untolerated taint {node.kubernetes.io/unreachable}, 1 node(s) had volume node affinity conflict, 1 node(s) were unschedulable. ``` Node state: * `kai-server` - Ready (control-plane) * `kai-macbook-pro-vm` - NotReady,SchedulingDisabled * `kai-desktop-tower-wsl` - NotReady So the regular runners' `local-path` PVC is pinned to a node that is now NotReady, and they can't reschedule onto kai-server (volume node affinity conflict). The blocker is now **node recovery**, not the pod wedge. ## Suggested fix Either bring `kai-macbook-pro-vm` / `kai-desktop-tower-wsl` back to Ready, or (if that data is disposable) delete + recreate the stranded `forgejo-runner` PVC so it binds on kai-server. Both are on-box / hands-on. ## Note A *separate* fault, the dedicated tap-writer runner, was found during this: `forgejo-runner-tap-writer-0` CrashLoopBackOff exit 99, tracked in #249 with fix PR #250. Not the same root cause as this issue. Current diagnosis is also on o2r channel FQT7 (event 126).

coilysiren referenced this issue

2026-06-06 23:06:30 +00:00

kai-server SSH sessions hang / drop - can't reliably get on-box #251

coilysiren

2026-06-17 07:05:03 +00:00