Debug job: kai-server k3s control-plane API flapping 502 (o2r 9KP5) #2
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Debug job: kai-server k3s control-plane API flapping 502
Run as an o2r ansible debug job in this repo:
source=kais-macbook-pro,target=kai-server, probes throughscripts/step.py, public diffs only, posted to o2r channel9KP5(already open - "debug kais-macbook-pro -> kai-server").Symptom
During Forgejo runner-tier remediation (coilysiren/inbox#65), the kai-server k3s API at
https://kai-server:6443degraded from brief transient502 Bad Gateway(self-recovered in seconds) to sustained 502 across 5+ minutes, failing both reads (get pods) and writes (apply). A502(not connection-refused/timeout) means a proxy in front of:6443is up but the apiserver backend is unhealthy. The host itself isactiveon Tailscale.Corroborating signal: a fleet-wide pod restart spike ~24h ago (
forgejo-db16 restarts,forgejo/ts-forgejo/null-db8-10 each), consistent with a kai-server reboot or resource-pressure event a day ago that left the control plane flapping.Probe plan (each via STEP, author parser on first run)
k3s service state-systemctl status k3s/is-activeapiserver readyz-k3s kubectl get --raw=/readyz(run locally on the box, bypasses the proxy)k3s journal- grepapiserver|etcd|oom|panic|leader|503|502|unavailablehost pressure-uptime,free -h,df -h / /var/lib/rancherrecent reboots-last -x reboot shutdown(does it explain the 24h-ago restart spike?)Acceptance
Root cause of the 502 flapping identified (OOM / etcd / proxy / reboot loop), object-level fix applied or filed, and the relevant probe baselines blessed once the apiserver holds steady.
Links
Live findings moving runners to kai-server (unpin from #241)
API recovered (502 was a ~2min flap, not permanent this round). Both runner StatefulSets re-applied unpinned; deleted the stale pinned pods to force recreation. Results:
registerinit succeeded ("Runner registered successfully"). Its PVC bound fresh on kai-server because local-path usesWaitForFirstConsumer(it had never bound to the tower). BUT the mainrunnercontainer isCrashLoopBackOff, exit 99:Unable to open log: Permission denied. Host-executor runs user-mode; the mounted work/log path is not writable by the runner user. Fix: manifest - set fsGroup/securityContext or a writable workdir.volume node affinity conflict. Its PVCdata-forgejo-runner-0is Bound (2d20h) to a local-path PV pinned to the deadkai-desktop-tower-wsl. Unpinning the pod doesn't move its storage. Fix: delete PVCdata-forgejo-runner-0(+data-forgejo-runner-1) so they reprovision on kai-server; the 1Gi volume is ephemeral DinD docker cache. Destructive write - needs sign-off per infra safety rules.Net: unpin is correct and tap-writer reached kai-server, but two follow-ups remain before CI is green on kai-server: (1) runner log-path permissions, (2) discard tower-bound DinD PVCs.
Resolved by recovery - control plane healthy, 502 not reproducing
Ran the laptop-side battery through
scripts/step.py(o2r channel9KP5, events 107-109). Kai authorized the laptop-kubectl-first path with the rule "escalate to on-box SSH only if still flapping."Results
kubectl-readyz:rc=0,readyz check passed, no change vs the blessed healthy baseline. The apiserver holds steady - a502would surface here as a proxy-up / backend-unhealthy failure, and it did not. Baseline blessed (apiserver steady).kubectl-pods(forgejons):forgejo-db/forgejoweb /ts-forgejounchanged andRunning. The only diffs are on the runner pods -forgejo-runner-0 -> Pending,forgejo-runner-tap-writer-0Runningbut not-ready. Not blessed - that is a degraded pending state, not a healthy baseline.Root cause
Transient control-plane blip during the inbox#65 runner-tier remediation (force-deleting the wedged runner pods), layered on the ~24h-ago restart event. It self-recovered. The sustained
502no longer reproduces from laptop-side probes, so the on-box SSH battery (systemctl/journal/ on-boxreadyz/free/df/last) was not needed and was not run.Residual (not a control-plane fault)
The runner tier stays degraded because
kai-desktop-tower-wslis down - the single node all Forgejo CI is pinned to. That is the SPOF tracked in coilysiren/inbox#65 (needs the tower powered on +/forgejo/tap-bump-tokenprovisioned), out of scope for this control-plane debug job.Closing as resolved-by-recovery. Runner-tier restoration follow-up lives in inbox#65.
Closing. o2r is archived in the June 2026 surface reduction - an optional agent channel, unused autonomously. Handover doctrine moves to human-mediated.