[epic] Eliminate the recurring post-bounce fleet full-stop (self-recovery + drift gate + blast-radius) #541

New issue

Open

opened 2026-07-10 18:12:52 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-10 18:12:52 +00:00

Member

[epic] Eliminate the recurring post-bounce fleet full-stop

Five full work stops in one week (2026-07-10 week) share one shape: kai-server bounces (reboot or k3s restart) and a service does not self-recover, and because everything funnels through the one cluster, it halts landing fleet-wide instead of degrading one service. The failures are independent surfaces of the same gap:

eco-server boot path + login (infrastructure#537)
Forgejo runner registration token missing from SSM (infrastructure#539)
containerd registry mirror alias missing from the host (infrastructure#540)
Forgejo runners + DNS/tailnet degraded, landing-blocking the whole fleet (infrastructure#538)

Each was fixed reactively and by hand. Per-incident patches do not lower the count - the next bounce finds the next un-hardened service. This epic owns the systemic fix so a bounce becomes a shrug, not a stop.

Two things make each of these a full stop

No post-bounce self-recovery or health gate. Nothing verifies the fleet came back healthy after a reboot/restart. Services fail independently and silently until an agent trips over one mid-work.
One-host blast radius, and config changes cost a bounce. Everything routes through one k3s/Forgejo/registry on kai-server, so one bounce halts landing everywhere - and even the fix for a config drift (the registries.yaml edit in #540) required a full k3s restart, i.e. another full stop.

Systemic work (children of this epic)

Post-bounce health gate + self-recovery pass. A boot-triggered check that asserts the fleet's load-bearing state and heals-or-alerts before an agent hits it: ExternalSecrets synced (runner token, api-token present), Forgejo runners registered and serving non-404 job-logs, the kai-registry.local mirror present, ExternalDNS + tailnet resolving the deployed hostnames, eco-server authenticated. The Gatus fleet-reachability bundle (docs/fleet-reachability.md) already monitors the hostnames - extend it into a boot gate.
Drift self-checks at bring-up, generalizing the point-checks landed in #539/#540: fail loud when a load-bearing SSM param, mirror alias, or ExternalSecret key is absent, at boot rather than mid-incident.
Decouple config reload from full k3s restart so a registries.yaml / mirror / config change does not cost a cluster bounce.
Reduce single-host blast radius - the longer arc already ticketed as the ser8 second plane (infrastructure#506 / #507 / #518). Track the dependency here.

Folded reactive issues (symptoms of this epic)

infrastructure#538 (runners + DNS/tailnet post-reboot) - the incident umbrella, keep open under this epic.
infrastructure#537 (eco-server boot path) - open follow-ups fold here.
infrastructure#539 (token-bootstrap script) - landed, precedent for the drift self-check pattern.
infrastructure#540 (registry mirror converge + drift check) - landed, precedent for #2.

Acceptance

A kai-server reboot returns the fleet to a landing-capable state with no hand-holding, verified by the boot gate going green unattended.
A missing load-bearing param/alias/secret is caught at bring-up and named, not discovered by an agent mid-run.
A routine config/mirror change no longer requires a full cluster restart.
The count of post-bounce full-stops trends to zero.

Context

Filed at Kai's direction after the 5th full stop of the week, to stop the reactive whack-a-mole and own the recurrence. Paired epic for the other full-stop layer: the ward-workflow anti-churn epic (ward). Refs: infrastructure#537/#538/#539/#540, #506/#507/#518, docs/fleet-reachability.md.

## [epic] Eliminate the recurring post-bounce fleet full-stop Five full work stops in one week (2026-07-10 week) share one shape: **kai-server bounces (reboot or k3s restart) and a service does not self-recover**, and because everything funnels through the one cluster, it halts *landing* fleet-wide instead of degrading one service. The failures are independent surfaces of the same gap: - eco-server boot path + login (infrastructure#537) - Forgejo runner registration token missing from SSM (infrastructure#539) - containerd registry mirror alias missing from the host (infrastructure#540) - Forgejo runners + DNS/tailnet degraded, landing-blocking the whole fleet (infrastructure#538) Each was fixed reactively and by hand. Per-incident patches do not lower the count - the next bounce finds the next un-hardened service. This epic owns the systemic fix so a bounce becomes a shrug, not a stop. ## Two things make each of these a *full* stop 1. **No post-bounce self-recovery or health gate.** Nothing verifies the fleet came back healthy after a reboot/restart. Services fail independently and silently until an agent trips over one mid-work. 2. **One-host blast radius, and config changes cost a bounce.** Everything routes through one k3s/Forgejo/registry on kai-server, so one bounce halts landing everywhere - and even the *fix* for a config drift (the registries.yaml edit in #540) required a full `k3s restart`, i.e. another full stop. ## Systemic work (children of this epic) 1. **Post-bounce health gate + self-recovery pass.** A boot-triggered check that asserts the fleet's load-bearing state and heals-or-alerts *before* an agent hits it: ExternalSecrets synced (runner token, api-token present), Forgejo runners registered and serving non-404 job-logs, the `kai-registry.local` mirror present, ExternalDNS + tailnet resolving the deployed hostnames, eco-server authenticated. The Gatus fleet-reachability bundle (`docs/fleet-reachability.md`) already monitors the hostnames - extend it into a boot gate. 2. **Drift self-checks at bring-up**, generalizing the point-checks landed in #539/#540: fail loud when a load-bearing SSM param, mirror alias, or ExternalSecret key is absent, at boot rather than mid-incident. 3. **Decouple config reload from full k3s restart** so a registries.yaml / mirror / config change does not cost a cluster bounce. 4. **Reduce single-host blast radius** - the longer arc already ticketed as the ser8 second plane (infrastructure#506 / #507 / #518). Track the dependency here. ## Folded reactive issues (symptoms of this epic) - infrastructure#538 (runners + DNS/tailnet post-reboot) - the incident umbrella, keep open under this epic. - infrastructure#537 (eco-server boot path) - open follow-ups fold here. - infrastructure#539 (token-bootstrap script) - **landed**, precedent for the drift self-check pattern. - infrastructure#540 (registry mirror converge + drift check) - **landed**, precedent for #2. ## Acceptance - A kai-server reboot returns the fleet to a landing-capable state with **no hand-holding**, verified by the boot gate going green unattended. - A missing load-bearing param/alias/secret is caught at bring-up and named, not discovered by an agent mid-run. - A routine config/mirror change no longer requires a full cluster restart. - The count of post-bounce full-stops trends to zero. ## Context Filed at Kai's direction after the 5th full stop of the week, to stop the reactive whack-a-mole and own the recurrence. Paired epic for the other full-stop layer: the ward-workflow anti-churn epic (ward). Refs: infrastructure#537/#538/#539/#540, #506/#507/#518, `docs/fleet-reachability.md`.

coilyco-ops added the

consult

labels

2026-07-10 18:12:52 +00:00

coilyco-ops referenced this issue

2026-07-10 18:13:08 +00:00

Post-reboot recovery incomplete: Forgejo Actions runners + DNS/tailnet degraded on kai-server, landing-blocking fleet burndown #538

coilyco-ops referenced this issue

2026-07-10 18:13:09 +00:00

eco-server: harden boot path + resolve -userToken leak and DiscordLink error (post-reboot incident) #537