Post-reboot recovery incomplete: Forgejo Actions runners + DNS/tailnet degraded on kai-server, landing-blocking fleet burndown #538
Labels
No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
coilyco-flight-deck/infrastructure#538
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fleet-wide landing blocker: post-reboot recovery on kai-server is incomplete
The ~08:28 UTC 2026-07-10 kai-server reboot (root cause documented in infrastructure#537 for the eco-server symptom) left two host-level services only partially recovered. Those two gaps are now landing-blocking the entire warded-agent burndown across all three orgs. The code work is fine. The engineers cannot land it because CI cannot run/merge and live acceptance cannot pass.
Symptom 1 - Forgejo Actions runners degraded
waiting/queuedfor minutes with no pickup (deploy#130, tracked for unstick in infrastructure#531, which has itself blocked twice - the fix needs host access an engineer container does not have).ci / gatejobs fail afterRepo test gate, and the Forgejo job-log route returns 404 for the failing runs (observed on aos PR #437 runs 918/922, and on the ward#1009 dispatch). Localward exec test+ward exec pre-commit-allpass on the same trees, so this is runner/job-log plumbing, not the code.Net effect:
WARD-OUTCOME: merge-readyPRs (aos#434, aos#435, aos#268) cannot verify or land, and the ward "replace unmergeable PR" chain keeps regenerating blocked PRs.Symptom 2 - DNS / tailnet resolution broken for deployed services
Post-reboot, the director surface cannot resolve deployed hostnames:
factory-game.coilysiren.me-Name or service not known(deploy#130).comfyui.coilysiren.me- does not resolve from a tailnet client (deploy#134, reopened after a prematuredone).Engineers report
WARD-OUTCOME: doneon the code change, then the live acceptance probe fails on DNS/tailnet, so the issues reopen. The recurring reopen loop is a recovery gap, not repeated engineer error.Why this needs a human / host action (not a headless engineer)
Both fixes are host-level on kai-server - restart/re-register the Forgejo Actions runner daemon(s), and restore the DNS/tailnet resolution path (or the ansible converge that does both). A sandboxed warded engineer container has no host access, and infrastructure#531 has already blocked twice proving exactly that. This is filed
consultfor that reason - it is a wall worth a human, not headless-dispatchable.Do
merge-readyPRs land.Acceptance
merge-readyPR (e.g. aos#434 or aos#435) lands green.Context
Surfaced from a read-only director surface (
director-claude-...) on 2026-07-10 while burning down the guard/ward/aos backlog. The flight-deck code lanes are already saturated by sibling directors (2 Claude + 1 Codex, ~7 engineers near the 12 cap), all producingblocked/salvagedoutcomes for this one root cause. Clearing this unblocks the whole fleet's burndown at once.Refs: infrastructure#537 (eco-server face of the same reboot), infrastructure#531 (bridge-runner unstick, blocked twice), deploy#130/#133/#134/#135/#137, aos#434/#435/#268 (merge-ready, cannot land).
🔎 ward agent advisor
ward agent advisor --harness clauderan a one-shot standard research pass on this question:Advisory: post-reboot kai-server recovery is one attended operator pass, ordered runners-first
Bottom line. This is correctly filed
consult. Both symptoms are host/cluster-level on kai-server and need an attended operator (Kai) with kubectl-write,ward ops forgejo, and tailnet/host access - a sandboxed engineer container cannot reach the host, which is exactly why infrastructure#531 blocked twice. This issue supersedes #531 for the bridge-runner unstick. Run one recovery pass in an attended shell on kai-server, ordered so the fleet unblocks in the right sequence: fix the Forgejo runners first (that unblocks CI and the merge gate for the whole burndown), then DNS/tailnet (that unblocks live acceptance), then rerun and land the queued/blocked work. Do not headless-dispatch this.The two symptoms decompose into 4 distinct failure paths
The issue lumps things, but the fixes differ per path, so diagnose before acting:
ci / gatefails afterRepo test gateand the job-log route 404s, while localward exec test/ward exec pre-commit-allpass. A 404 job-log means the runner never uploaded logs, so the job died at infra level (registration, dind, or cross-node reachability), not in the code.waiting/queued- run 6276 never picked up (deploy#130). Runners register per-org, so the bridge runner and the flight-deck runner are separate registrations that can fail independently.factory-game.coilysiren.meName or service not known. ExternalDNS (deploy/external_dns.yml,ward exec external-dns) writes Route53 records from Ingress objects. It is--policy=upsert-only, so it never deletes - a missing record for a new service (factory-game is new in deploy#130) means the record was never created, which points at the app Ingress or the controller, not a lost record.comfyui.coilysiren.me(deploy#134) resolves via the tailscale-operator ts-proxy sidecar (comfyui carries ats-authkeysidecar), and forgejo-issues / agent-proxy / skillsmp (deploy#137/#135/#133) resolve via the host-native CaddyHost: kai-servershortcut matcher plus the skillsmp NodePort bypass (caddy/sites-manual/skillsmp.caddy). Post-reboot, either the ts-proxy sidecars did not re-auth onto the tailnet or host Caddy did not reload its shortcut config.Ordered recovery runbook (attended, on kai-server)
Phase 0 - baseline (2 min).
systemctl status k3sandkubectl get nodes(nodeReady). Note trap #170:systemctl restart k3scan hang inactivatingeven when k3s is healthy - if you must restart, use--no-block. Confirm the tailnet is up withtailscale status; if SSH/MagicDNS itself is degraded, re-assert the host-side prereqsudo tailscale set --ssh(not managed by the terraform tailscale module, so a reboot can drop it).Phase 1 - Forgejo runners (unblocks the merge gate).
kubectl -n forgejo get pods -o wide | grep runner- this is the first diagnostic fork:CrashLoopBackOffonforgejo-runner-{0,1}after restart with dind logs showingwaiting for response from boltdb open-> this is trap #313 and the wrapper self-heals once dind answershttp://localhost:2375/_ping; check dind logs and node I/O pressure rather than editing the spec.Pendingwithvolume node affinity conflict-> stalelocal-pathPVCs (data-forgejo-runner-{0,1}) bound to the wrong node (trap #163). Delete them and let the StatefulSet recreate.Init:CrashLoopBackOffon the worker (kai-desktop-tower-wsl) -> cross-node flannel VXLAN to the forgejo ClusterIP is dead (trap #163). Do not repin the runner onto kai-server as a workaround (trap #151 forbids it) - fix the routable-plane reachability instead.kubectl -n forgejo apply -f deploy/forgejo-runner.yml. The pods re-register from SSM/forgejo/runner-registration-token(minted viaward ops forgejo actions generate-runner-token), so confirm that param is present and current if registration fails.Phase 2 - DNS/tailnet (unblocks live acceptance).
kubectl -n external-dns get podsand tail its logs for reconcile/AccessDenied.dig +short factory-game.coilysiren.me. If empty, check the app's Ingress in coilyco-bridge/deploy carries all three: theexternal-dns.coilysiren.me/managed: "true"label, the.../hostnameannotation, and the.../target= home public IP annotation (a public host without the target annotation resolves to the unroutable100.xtailnet IP). That last one is an app-repo fix, not infra - see Open questions.kubectl -n tailscale get podsandtailscale statusfor the service nodes; restart the ts-proxy sidecar/operator if comfyui dropped off the tailnet. For the Caddy shortcuts, on the host:systemctl status caddyandsystemctl status caddy-config-deploy.path; re-runsudo bash scripts/install-caddy-config.shto reinstall and reload the shortcut config (bootstrap the path unit withsudo bash scripts/install-caddy-config-deploy.shif it is absent).ward exec ansible-sync local=1on kai-server restores host-level config (caddy deploy path unit, node tuning, etc). Confirm before cluster writes per the repo Safety rule.Phase 3 - rerun and land. Re-dispatch deploy run 6276, re-run the aos gate runs (918/922) and the ward#1009 dispatch, then confirm a
merge-readyPR (aos#434 or aos#435) lands green. That green landing is the acceptance signal for the whole issue.Decisions and open questions
kubectl -n forgejo get pods -o wideoutput plus dind logs decide it. Everything downstream branches here, so resolve it before touching manifests.digshows no record or the unroutable tailnet IP, the fix is on the coilyco-bridge app Ingress (thetargetannotation), not this infra repo. That is the one genuinely cross-repo thread - if confirmed, it wants a small coilyco-bridge/deploy issue, but only after the operator confirms the annotation is the actual gap (the controller may simply have been down).forgejonamespace separately.docs/fleet-reachability.md) and confirmation that the ts-proxy sidecars and caddy path unit come back on boot without hand-holding, so the next reboot does not re-block the fleet.Risks and safety
.runnerregistration; it re-registers from the SSM token, so verify/forgejo/runner-registration-tokenis valid first or the pods come up unregistered.dockerd --iptables=false(trap).Acceptance mapping
merge-readyPR lands green -> Phase 3, using aos#434/#435 as the proof.This is one attended pass, not a headless dispatch. Recommend Kai runs Phases 0-3 in sequence on kai-server, then files the reboot-self-recovery hardening follow-up so the fleet does not re-block on the next bounce.
Researched and posted automatically by
ward agent advisor --harness claude(ward#179). This is one-shot research, not a carried change - verify before acting on it.— Claude (she/her), via
ward agentFolded under the resilience epic #541 - post-bounce runner + DNS/tailnet recovery is the incident umbrella for this epic