Cross-site warm standby for k3s control plane (operational resilience, not HA)

coilysiren commented

2026-05-24 00:15:40 +00:00

Owner

Title: Cross-site warm standby for k3s control plane (operational resilience, not HA)

Context

Today the cluster has one always-on failure domain, the home-1 server, which holds the control plane, the datastore, and the orchestrator. The failure mode being designed against is the home-1 site going dark for days with no advance notice. A second home is available as a genuinely independent failure domain with separate power, ISP, and physical location.

Decision

Pursue operational resilience via a cross-site warm standby with manual promotion, explicitly not first-class k3s HA. HA is a failure-domain-count problem, and with a single always-on domain it is unachievable at any spend. A two-domain etcd is also worse for writes than a single node, since losing either member drops below quorum. A public-cloud third domain is off the table by preference, so automated quorum is out, and a manual promote step is the accepted trade for keeping everything inside the homelab.

Topology

Home 1 server - primary - always-on control plane, datastore, and orchestrator.
Home 2 box, to purchase - warm standby and DR target - wired, NVMe, sized to actually run the orchestrator and critical workloads on promotion rather than idle as a witness.
Radxa Zero 3W - worker only - WiFi and SD/eMMC storage make it unfit to hold state, fine for running scheduled workloads.
Tailscale - transport - carries cross-home replication and control traffic and keeps it off the open internet.

Replication and recovery

State replication - scheduled snapshot ship from home 1 to home 2 over Tailscale - RPO equals the snapshot interval.
Recovery - manual promotion of the home-2 standby, which restores the latest snapshot, brings up a single-node control plane, and repoints workers - RTO equals restore time plus the manual trigger.
Cloud - none - all snapshots and replication stay between the two homes.

Why this is acceptable

Running workloads survive a dead control plane because the kubelet keeps existing pods alive without the apiserver. A home-1 outage therefore degrades the cluster to cannot deploy, reschedule, or scale, rather than everything stops, and the manual promote restores write capability when it is actually needed. The result doubles as a demonstrable DR story with concrete RTO and RPO numbers.

Next action

Purchase the home-2 mini-PC. Everything downstream targets a host that does not exist yet, so the build is blocked on the box being bought and joined to Tailscale.

Follow-ups, file once the box exists

Snapshot-ship automation - home 1 to home 2 on a schedule.
Promote runbook plus a single mobile-triggerable recovery path.
Warm-standby orchestrator failover - so the orchestrator has a live home when home 1 is dark.
Measured RTO and RPO captured from a tested restore.

Title: Cross-site warm standby for k3s control plane (operational resilience, not HA) ## Context Today the cluster has one always-on failure domain, the home-1 server, which holds the control plane, the datastore, and the orchestrator. The failure mode being designed against is the home-1 site going dark for days with no advance notice. A second home is available as a genuinely independent failure domain with separate power, ISP, and physical location. ## Decision Pursue operational resilience via a cross-site warm standby with manual promotion, explicitly not first-class k3s HA. HA is a failure-domain-count problem, and with a single always-on domain it is unachievable at any spend. A two-domain etcd is also worse for writes than a single node, since losing either member drops below quorum. A public-cloud third domain is off the table by preference, so automated quorum is out, and a manual promote step is the accepted trade for keeping everything inside the homelab. ## Topology * Home 1 server - primary - always-on control plane, datastore, and orchestrator. * Home 2 box, to purchase - warm standby and DR target - wired, NVMe, sized to actually run the orchestrator and critical workloads on promotion rather than idle as a witness. * Radxa Zero 3W - worker only - WiFi and SD/eMMC storage make it unfit to hold state, fine for running scheduled workloads. * Tailscale - transport - carries cross-home replication and control traffic and keeps it off the open internet. ## Replication and recovery * State replication - scheduled snapshot ship from home 1 to home 2 over Tailscale - RPO equals the snapshot interval. * Recovery - manual promotion of the home-2 standby, which restores the latest snapshot, brings up a single-node control plane, and repoints workers - RTO equals restore time plus the manual trigger. * Cloud - none - all snapshots and replication stay between the two homes. ## Why this is acceptable Running workloads survive a dead control plane because the kubelet keeps existing pods alive without the apiserver. A home-1 outage therefore degrades the cluster to cannot deploy, reschedule, or scale, rather than everything stops, and the manual promote restores write capability when it is actually needed. The result doubles as a demonstrable DR story with concrete RTO and RPO numbers. ## Next action Purchase the home-2 mini-PC. Everything downstream targets a host that does not exist yet, so the build is blocked on the box being bought and joined to Tailscale. ## Follow-ups, file once the box exists * Snapshot-ship automation - home 1 to home 2 on a schedule. * Promote runbook plus a single mobile-triggerable recovery path. * Warm-standby orchestrator failover - so the orchestrator has a live home when home 1 is dark. * Measured RTO and RPO captured from a tested restore.

coilysiren commented

2026-05-30 05:43:24 +00:00

Author

Owner

Iceboxed in the 2026-05-29 backlog burn-down: cross-site warm standby, far-future resilience play. Reopen anytime if it becomes real.

coilysiren closed this issue

2026-05-30 05:43:24 +00:00

coilysiren referenced this issue

2026-06-07 05:20:11 +00:00

ser8: join kai-server k3s cluster as a worker node (warm-standby promotion later) #258

coilysiren referenced this issue

2026-06-07 05:27:12 +00:00