Post-reboot recovery incomplete: Forgejo Actions runners + DNS/tailnet degraded on kai-server, landing-blocking fleet burndown #538

Open
opened 2026-07-10 16:54:55 +00:00 by coilyco-ops · 2 comments
Member

Fleet-wide landing blocker: post-reboot recovery on kai-server is incomplete

The ~08:28 UTC 2026-07-10 kai-server reboot (root cause documented in infrastructure#537 for the eco-server symptom) left two host-level services only partially recovered. Those two gaps are now landing-blocking the entire warded-agent burndown across all three orgs. The code work is fine. The engineers cannot land it because CI cannot run/merge and live acceptance cannot pass.

Symptom 1 - Forgejo Actions runners degraded

  • coilyco-bridge runner: deploy workflow run 6276 stayed waiting/queued for minutes with no pickup (deploy#130, tracked for unstick in infrastructure#531, which has itself blocked twice - the fix needs host access an engineer container does not have).
  • coilyco-flight-deck runner: PR ci / gate jobs fail after Repo test gate, and the Forgejo job-log route returns 404 for the failing runs (observed on aos PR #437 runs 918/922, and on the ward#1009 dispatch). Local ward exec test + ward exec pre-commit-all pass on the same trees, so this is runner/job-log plumbing, not the code.

Net effect: WARD-OUTCOME: merge-ready PRs (aos#434, aos#435, aos#268) cannot verify or land, and the ward "replace unmergeable PR" chain keeps regenerating blocked PRs.

Symptom 2 - DNS / tailnet resolution broken for deployed services

Post-reboot, the director surface cannot resolve deployed hostnames:

  • factory-game.coilysiren.me - Name or service not known (deploy#130).
  • comfyui.coilysiren.me - does not resolve from a tailnet client (deploy#134, reopened after a premature done).
  • forgejo-issues / agent-proxy / skillsmp tailnet hostnames failing (deploy#137, deploy#135, deploy#133 - several reopened after post-close live verification failed).

Engineers report WARD-OUTCOME: done on the code change, then the live acceptance probe fails on DNS/tailnet, so the issues reopen. The recurring reopen loop is a recovery gap, not repeated engineer error.

Why this needs a human / host action (not a headless engineer)

Both fixes are host-level on kai-server - restart/re-register the Forgejo Actions runner daemon(s), and restore the DNS/tailnet resolution path (or the ansible converge that does both). A sandboxed warded engineer container has no host access, and infrastructure#531 has already blocked twice proving exactly that. This is filed consult for that reason - it is a wall worth a human, not headless-dispatchable.

Do

  • Run a post-reboot recovery pass on kai-server: bring the Forgejo Actions runner(s) back to healthy pickup + reachable job-logs, and restore DNS/tailnet resolution for the deployed services above.
  • Then rerun the queued/failed workflows (deploy run 6276, aos gate runs) and let the in-flight directors' merge-ready PRs land.

Acceptance

  • coilyco-bridge and coilyco-flight-deck Forgejo Actions runners pick up jobs and serve reachable (non-404) job-logs.
  • The deployed hostnames above resolve and reach their front door from a tailnet client.
  • A previously-blocked merge-ready PR (e.g. aos#434 or aos#435) lands green.

Context

Surfaced from a read-only director surface (director-claude-...) on 2026-07-10 while burning down the guard/ward/aos backlog. The flight-deck code lanes are already saturated by sibling directors (2 Claude + 1 Codex, ~7 engineers near the 12 cap), all producing blocked/salvaged outcomes for this one root cause. Clearing this unblocks the whole fleet's burndown at once.

Refs: infrastructure#537 (eco-server face of the same reboot), infrastructure#531 (bridge-runner unstick, blocked twice), deploy#130/#133/#134/#135/#137, aos#434/#435/#268 (merge-ready, cannot land).

## Fleet-wide landing blocker: post-reboot recovery on kai-server is incomplete The **~08:28 UTC 2026-07-10 kai-server reboot** (root cause documented in infrastructure#537 for the eco-server symptom) left two host-level services only partially recovered. Those two gaps are now **landing-blocking the entire warded-agent burndown** across all three orgs. The code work is fine. The engineers cannot land it because CI cannot run/merge and live acceptance cannot pass. ### Symptom 1 - Forgejo Actions runners degraded - **coilyco-bridge runner**: deploy workflow run 6276 stayed `waiting`/`queued` for minutes with no pickup (deploy#130, tracked for unstick in infrastructure#531, which has itself blocked twice - the fix needs host access an engineer container does not have). - **coilyco-flight-deck runner**: PR `ci / gate` jobs fail after `Repo test gate`, and the Forgejo job-log route returns **404** for the failing runs (observed on aos PR #437 runs 918/922, and on the ward#1009 dispatch). Local `ward exec test` + `ward exec pre-commit-all` pass on the same trees, so this is runner/job-log plumbing, not the code. Net effect: `WARD-OUTCOME: merge-ready` PRs (aos#434, aos#435, aos#268) cannot verify or land, and the ward "replace unmergeable PR" chain keeps regenerating blocked PRs. ### Symptom 2 - DNS / tailnet resolution broken for deployed services Post-reboot, the director surface cannot resolve deployed hostnames: - `factory-game.coilysiren.me` - `Name or service not known` (deploy#130). - `comfyui.coilysiren.me` - does not resolve from a tailnet client (deploy#134, reopened after a premature `done`). - forgejo-issues / agent-proxy / skillsmp tailnet hostnames failing (deploy#137, deploy#135, deploy#133 - several reopened after post-close live verification failed). Engineers report `WARD-OUTCOME: done` on the code change, then the live acceptance probe fails on DNS/tailnet, so the issues reopen. The recurring reopen loop is a recovery gap, not repeated engineer error. ## Why this needs a human / host action (not a headless engineer) Both fixes are **host-level on kai-server** - restart/re-register the Forgejo Actions runner daemon(s), and restore the DNS/tailnet resolution path (or the ansible converge that does both). A sandboxed warded engineer container has no host access, and infrastructure#531 has already blocked twice proving exactly that. This is filed `consult` for that reason - it is a wall worth a human, not headless-dispatchable. ## Do - Run a post-reboot recovery pass on kai-server: bring the Forgejo Actions runner(s) back to healthy pickup + reachable job-logs, and restore DNS/tailnet resolution for the deployed services above. - Then rerun the queued/failed workflows (deploy run 6276, aos gate runs) and let the in-flight directors' `merge-ready` PRs land. ## Acceptance - coilyco-bridge and coilyco-flight-deck Forgejo Actions runners pick up jobs and serve reachable (non-404) job-logs. - The deployed hostnames above resolve and reach their front door from a tailnet client. - A previously-blocked `merge-ready` PR (e.g. aos#434 or aos#435) lands green. ## Context Surfaced from a read-only director surface (`director-claude-...`) on 2026-07-10 while burning down the guard/ward/aos backlog. The flight-deck code lanes are already saturated by sibling directors (2 Claude + 1 Codex, ~7 engineers near the 12 cap), all producing `blocked`/`salvaged` outcomes for this one root cause. Clearing this unblocks the whole fleet's burndown at once. Refs: infrastructure#537 (eco-server face of the same reboot), infrastructure#531 (bridge-runner unstick, blocked twice), deploy#130/#133/#134/#135/#137, aos#434/#435/#268 (merge-ready, cannot land).
Author
Member

🔎 ward agent advisor

ward agent advisor --harness claude ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.


Advisory: post-reboot kai-server recovery is one attended operator pass, ordered runners-first

Bottom line. This is correctly filed consult. Both symptoms are host/cluster-level on kai-server and need an attended operator (Kai) with kubectl-write, ward ops forgejo, and tailnet/host access - a sandboxed engineer container cannot reach the host, which is exactly why infrastructure#531 blocked twice. This issue supersedes #531 for the bridge-runner unstick. Run one recovery pass in an attended shell on kai-server, ordered so the fleet unblocks in the right sequence: fix the Forgejo runners first (that unblocks CI and the merge gate for the whole burndown), then DNS/tailnet (that unblocks live acceptance), then rerun and land the queued/blocked work. Do not headless-dispatch this.

The two symptoms decompose into 4 distinct failure paths

The issue lumps things, but the fixes differ per path, so diagnose before acting:

  • Runner path A - flight-deck gate + 404 job-logs - ci / gate fails after Repo test gate and the job-log route 404s, while local ward exec test / ward exec pre-commit-all pass. A 404 job-log means the runner never uploaded logs, so the job died at infra level (registration, dind, or cross-node reachability), not in the code.
  • Runner path B - coilyco-bridge deploy stuck waiting/queued - run 6276 never picked up (deploy#130). Runners register per-org, so the bridge runner and the flight-deck runner are separate registrations that can fail independently.
  • DNS path C - public hostnames via ExternalDNS -> Route53 - factory-game.coilysiren.me Name or service not known. ExternalDNS (deploy/external_dns.yml, ward exec external-dns) writes Route53 records from Ingress objects. It is --policy=upsert-only, so it never deletes - a missing record for a new service (factory-game is new in deploy#130) means the record was never created, which points at the app Ingress or the controller, not a lost record.
  • DNS path D - tailnet service hostnames - comfyui.coilysiren.me (deploy#134) resolves via the tailscale-operator ts-proxy sidecar (comfyui carries a ts-authkey sidecar), and forgejo-issues / agent-proxy / skillsmp (deploy#137/#135/#133) resolve via the host-native Caddy Host: kai-server shortcut matcher plus the skillsmp NodePort bypass (caddy/sites-manual/skillsmp.caddy). Post-reboot, either the ts-proxy sidecars did not re-auth onto the tailnet or host Caddy did not reload its shortcut config.

Ordered recovery runbook (attended, on kai-server)

Phase 0 - baseline (2 min). systemctl status k3s and kubectl get nodes (node Ready). Note trap #170: systemctl restart k3s can hang in activating even when k3s is healthy - if you must restart, use --no-block. Confirm the tailnet is up with tailscale status; if SSH/MagicDNS itself is degraded, re-assert the host-side prereq sudo tailscale set --ssh (not managed by the terraform tailscale module, so a reboot can drop it).

Phase 1 - Forgejo runners (unblocks the merge gate).

  • kubectl -n forgejo get pods -o wide | grep runner - this is the first diagnostic fork:
    • CrashLoopBackOff on forgejo-runner-{0,1} after restart with dind logs showing waiting for response from boltdb open -> this is trap #313 and the wrapper self-heals once dind answers http://localhost:2375/_ping; check dind logs and node I/O pressure rather than editing the spec.
    • Pending with volume node affinity conflict -> stale local-path PVCs (data-forgejo-runner-{0,1}) bound to the wrong node (trap #163). Delete them and let the StatefulSet recreate.
    • Init:CrashLoopBackOff on the worker (kai-desktop-tower-wsl) -> cross-node flannel VXLAN to the forgejo ClusterIP is dead (trap #163). Do not repin the runner onto kai-server as a workaround (trap #151 forbids it) - fix the routable-plane reachability instead.
  • Reconcile and let re-registration happen: kubectl -n forgejo apply -f deploy/forgejo-runner.yml. The pods re-register from SSM /forgejo/runner-registration-token (minted via ward ops forgejo actions generate-runner-token), so confirm that param is present and current if registration fails.
  • Verify pickup: dispatch/observe a job and confirm the job-log route serves 200, not 404.

Phase 2 - DNS/tailnet (unblocks live acceptance).

  • Path C: kubectl -n external-dns get pods and tail its logs for reconcile/AccessDenied. dig +short factory-game.coilysiren.me. If empty, check the app's Ingress in coilyco-bridge/deploy carries all three: the external-dns.coilysiren.me/managed: "true" label, the .../hostname annotation, and the .../target = home public IP annotation (a public host without the target annotation resolves to the unroutable 100.x tailnet IP). That last one is an app-repo fix, not infra - see Open questions.
  • Path D: kubectl -n tailscale get pods and tailscale status for the service nodes; restart the ts-proxy sidecar/operator if comfyui dropped off the tailnet. For the Caddy shortcuts, on the host: systemctl status caddy and systemctl status caddy-config-deploy.path; re-run sudo bash scripts/install-caddy-config.sh to reinstall and reload the shortcut config (bootstrap the path unit with sudo bash scripts/install-caddy-config-deploy.sh if it is absent).
  • Blanket host re-converge if several host units are off: ward exec ansible-sync local=1 on kai-server restores host-level config (caddy deploy path unit, node tuning, etc). Confirm before cluster writes per the repo Safety rule.

Phase 3 - rerun and land. Re-dispatch deploy run 6276, re-run the aos gate runs (918/922) and the ward#1009 dispatch, then confirm a merge-ready PR (aos#434 or aos#435) lands green. That green landing is the acceptance signal for the whole issue.

Decisions and open questions

  • Runner failure mode (the load-bearing fork). Path A/B could be a local dind stall (#313, self-heals) or a broken cross-node VXLAN plane (#163, needs a reachability fix). The kubectl -n forgejo get pods -o wide output plus dind logs decide it. Everything downstream branches here, so resolve it before touching manifests.
  • factory-game record ownership. If dig shows no record or the unroutable tailnet IP, the fix is on the coilyco-bridge app Ingress (the target annotation), not this infra repo. That is the one genuinely cross-repo thread - if confirmed, it wants a small coilyco-bridge/deploy issue, but only after the operator confirms the annotation is the actual gap (the controller may simply have been down).
  • 404 job-logs - transient or server-side. If a fresh re-run produces reachable logs, it was runner registration/plumbing and is closed. If the 404 persists on new runs, investigate the forgejo server pod's log storage/PVC in the forgejo namespace separately.
  • Reboot root cause and self-recovery. #537 documents the reboot's eco-server face. The runner already self-heals post-reboot (#313); the gap is that DNS/tailnet + host Caddy did not. Recommend a follow-up hardening issue in this repo: a post-reboot smoke check (the Gatus fleet-reachability bundle already monitors these hostnames - docs/fleet-reachability.md) and confirmation that the ts-proxy sidecars and caddy path unit come back on boot without hand-holding, so the next reboot does not re-block the fleet.

Risks and safety

  • Re-applying manifests / kubectl writes can disturb in-flight jobs - confirm before cluster writes (repo Safety rule).
  • Deleting the runner PVCs drops the .runner registration; it re-registers from the SSM token, so verify /forgejo/runner-registration-token is valid first or the pods come up unregistered.
  • Do not repin runners to kai-server (#151) and do not dockerd --iptables=false (trap).
  • ExternalDNS upsert-only will not delete a wrong record - a stale/wrong record must be corrected at the source Ingress, not by expecting the controller to prune it.

Acceptance mapping

  • Runners pick up + serve non-404 job-logs -> Phase 1.
  • Deployed hostnames resolve and reach their front door from a tailnet client -> Phase 2 (paths C and D).
  • A blocked merge-ready PR lands green -> Phase 3, using aos#434/#435 as the proof.

This is one attended pass, not a headless dispatch. Recommend Kai runs Phases 0-3 in sequence on kai-server, then files the reboot-self-recovery hardening follow-up so the fleet does not re-block on the next bounce.


Researched and posted automatically by ward agent advisor --harness claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness claude` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- ## Advisory: post-reboot kai-server recovery is one attended operator pass, ordered runners-first **Bottom line.** This is correctly filed `consult`. Both symptoms are host/cluster-level on kai-server and need an attended operator (Kai) with kubectl-write, `ward ops forgejo`, and tailnet/host access - a sandboxed engineer container cannot reach the host, which is exactly why infrastructure#531 blocked twice. **This issue supersedes #531** for the bridge-runner unstick. Run one recovery pass in an attended shell on kai-server, ordered so the fleet unblocks in the right sequence: **fix the Forgejo runners first** (that unblocks CI and the merge gate for the whole burndown), **then DNS/tailnet** (that unblocks live acceptance), **then rerun and land** the queued/blocked work. Do not headless-dispatch this. ### The two symptoms decompose into 4 distinct failure paths The issue lumps things, but the fixes differ per path, so diagnose before acting: * **Runner path A - flight-deck gate + 404 job-logs** - `ci / gate` fails after `Repo test gate` and the job-log route 404s, while local `ward exec test` / `ward exec pre-commit-all` pass. A 404 job-log means the runner never uploaded logs, so the job died at infra level (registration, dind, or cross-node reachability), not in the code. * **Runner path B - coilyco-bridge deploy stuck `waiting`/`queued`** - run 6276 never picked up (deploy#130). Runners register per-org, so the bridge runner and the flight-deck runner are separate registrations that can fail independently. * **DNS path C - public hostnames via ExternalDNS -> Route53** - `factory-game.coilysiren.me` `Name or service not known`. ExternalDNS (`deploy/external_dns.yml`, `ward exec external-dns`) writes Route53 records from Ingress objects. It is `--policy=upsert-only`, so it never deletes - a missing record for a **new** service (factory-game is new in deploy#130) means the record was never created, which points at the app Ingress or the controller, not a lost record. * **DNS path D - tailnet service hostnames** - `comfyui.coilysiren.me` (deploy#134) resolves via the tailscale-operator ts-proxy sidecar (comfyui carries a `ts-authkey` sidecar), and forgejo-issues / agent-proxy / skillsmp (deploy#137/#135/#133) resolve via the host-native Caddy `Host: kai-server` shortcut matcher plus the skillsmp NodePort bypass (`caddy/sites-manual/skillsmp.caddy`). Post-reboot, either the ts-proxy sidecars did not re-auth onto the tailnet or host Caddy did not reload its shortcut config. ### Ordered recovery runbook (attended, on kai-server) **Phase 0 - baseline (2 min).** `systemctl status k3s` and `kubectl get nodes` (node `Ready`). Note trap #170: `systemctl restart k3s` can hang in `activating` even when k3s is healthy - if you must restart, use `--no-block`. Confirm the tailnet is up with `tailscale status`; if SSH/MagicDNS itself is degraded, re-assert the host-side prereq `sudo tailscale set --ssh` (not managed by the terraform tailscale module, so a reboot can drop it). **Phase 1 - Forgejo runners (unblocks the merge gate).** * `kubectl -n forgejo get pods -o wide | grep runner` - this is the first diagnostic fork: * `CrashLoopBackOff` on `forgejo-runner-{0,1}` after restart with dind logs showing `waiting for response from boltdb open` -> this is trap #313 and the wrapper self-heals once dind answers `http://localhost:2375/_ping`; check dind logs and node I/O pressure rather than editing the spec. * `Pending` with `volume node affinity conflict` -> stale `local-path` PVCs (`data-forgejo-runner-{0,1}`) bound to the wrong node (trap #163). Delete them and let the StatefulSet recreate. * `Init:CrashLoopBackOff` on the worker (`kai-desktop-tower-wsl`) -> cross-node flannel VXLAN to the forgejo ClusterIP is dead (trap #163). **Do not repin the runner onto kai-server** as a workaround (trap #151 forbids it) - fix the routable-plane reachability instead. * Reconcile and let re-registration happen: `kubectl -n forgejo apply -f deploy/forgejo-runner.yml`. The pods re-register from SSM `/forgejo/runner-registration-token` (minted via `ward ops forgejo actions generate-runner-token`), so confirm that param is present and current if registration fails. * Verify pickup: dispatch/observe a job and confirm the job-log route serves 200, not 404. **Phase 2 - DNS/tailnet (unblocks live acceptance).** * Path C: `kubectl -n external-dns get pods` and tail its logs for reconcile/`AccessDenied`. `dig +short factory-game.coilysiren.me`. If empty, check the **app's** Ingress in coilyco-bridge/deploy carries all three: the `external-dns.coilysiren.me/managed: "true"` label, the `.../hostname` annotation, and the `.../target` = home public IP annotation (a public host without the target annotation resolves to the unroutable `100.x` tailnet IP). That last one is an app-repo fix, not infra - see Open questions. * Path D: `kubectl -n tailscale get pods` and `tailscale status` for the service nodes; restart the ts-proxy sidecar/operator if comfyui dropped off the tailnet. For the Caddy shortcuts, on the host: `systemctl status caddy` and `systemctl status caddy-config-deploy.path`; re-run `sudo bash scripts/install-caddy-config.sh` to reinstall and reload the shortcut config (bootstrap the path unit with `sudo bash scripts/install-caddy-config-deploy.sh` if it is absent). * Blanket host re-converge if several host units are off: `ward exec ansible-sync local=1` on kai-server restores host-level config (caddy deploy path unit, node tuning, etc). Confirm before cluster writes per the repo Safety rule. **Phase 3 - rerun and land.** Re-dispatch deploy run 6276, re-run the aos gate runs (918/922) and the ward#1009 dispatch, then confirm a `merge-ready` PR (aos#434 or aos#435) lands green. That green landing is the acceptance signal for the whole issue. ### Decisions and open questions * **Runner failure mode (the load-bearing fork).** Path A/B could be a local dind stall (#313, self-heals) or a broken cross-node VXLAN plane (#163, needs a reachability fix). The `kubectl -n forgejo get pods -o wide` output plus dind logs decide it. Everything downstream branches here, so resolve it before touching manifests. * **factory-game record ownership.** If `dig` shows no record or the unroutable tailnet IP, the fix is on the **coilyco-bridge** app Ingress (the `target` annotation), not this infra repo. That is the one genuinely cross-repo thread - if confirmed, it wants a small coilyco-bridge/deploy issue, but only after the operator confirms the annotation is the actual gap (the controller may simply have been down). * **404 job-logs - transient or server-side.** If a fresh re-run produces reachable logs, it was runner registration/plumbing and is closed. If the 404 persists on new runs, investigate the forgejo **server** pod's log storage/PVC in the `forgejo` namespace separately. * **Reboot root cause and self-recovery.** #537 documents the reboot's eco-server face. The runner already self-heals post-reboot (#313); the gap is that DNS/tailnet + host Caddy did not. Recommend a follow-up hardening issue in this repo: a post-reboot smoke check (the Gatus fleet-reachability bundle already monitors these hostnames - `docs/fleet-reachability.md`) and confirmation that the ts-proxy sidecars and caddy path unit come back on boot without hand-holding, so the next reboot does not re-block the fleet. ### Risks and safety * Re-applying manifests / kubectl writes can disturb in-flight jobs - confirm before cluster writes (repo Safety rule). * Deleting the runner PVCs drops the `.runner` registration; it re-registers from the SSM token, so verify `/forgejo/runner-registration-token` is valid first or the pods come up unregistered. * Do not repin runners to kai-server (#151) and do not `dockerd --iptables=false` (trap). * ExternalDNS upsert-only will not delete a wrong record - a stale/wrong record must be corrected at the source Ingress, not by expecting the controller to prune it. ### Acceptance mapping * Runners pick up + serve non-404 job-logs -> Phase 1. * Deployed hostnames resolve and reach their front door from a tailnet client -> Phase 2 (paths C and D). * A blocked `merge-ready` PR lands green -> Phase 3, using aos#434/#435 as the proof. This is one attended pass, not a headless dispatch. Recommend Kai runs Phases 0-3 in sequence on kai-server, then files the reboot-self-recovery hardening follow-up so the fleet does not re-block on the next bounce. --- Researched and posted automatically by `ward agent advisor --harness claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it. <!-- ward-agent-reply --> <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
Author
Member

Folded under the resilience epic #541 - post-bounce runner + DNS/tailnet recovery is the incident umbrella for this epic

Folded under the resilience epic #541 - post-bounce runner + DNS/tailnet recovery is the incident umbrella for this epic
Sign in to join this conversation.
No description provided.