Buy two Beelink SER8 mini PCs, expand K3s to a 3-node HA control plane #11

New issue

Closed

opened 2026-05-23 20:54:26 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:26 +00:00

Owner

Originally filed by @coilysiren on 2026-05-22T13:50:37Z - https://github.com/coilysiren/infrastructure/issues/278

Problem

kai-server is currently the only K3s control-plane node. If it goes down, the whole control plane is down and nothing reschedules anywhere. It is a single point of failure that cannot tolerate any downtime.

Plan

Buy two Beelink SER8 mini PCs and join them as control-plane (server) nodes alongside kai-server. That makes a 3-node etcd cluster: N=3, quorum 2, tolerates one machine failure. Any single box, including kai-server, can go down and the cluster keeps scheduling. The two SER8s also host the public-facing services so blue-green deploys stop being manual.

Hardware

Beelink SER8 (AMD Ryzen 7 8845HS, 8C/16T, 32GB DDR5, 1TB NVMe, 2x M.2 slots, 2.5GbE): https://www.bee-link.com/products/beelink-ser8-8845hs
Buy two. Each well under the ~$1k/machine budget.

What this gives, and what it does not

Control-plane HA: yes. The 3-node cluster removes kai-server as a single point of failure.
Both SER8s live at the current house, kai-server is at the other house. The single point of failure moves from "one machine" to "the 2-node site." If that house goes fully dark, kai-server alone cannot hold quorum and scheduling stops. Still a real upgrade, since single-box failures and routine reboots are far more common than a whole-house outage.
Physical-distance backup is a separate goal. kai-server should still hold independent data backups and replicas of the public services so a true house-level disaster leaves it able to serve, even if degraded and manual.

Notes

etcd Raft heartbeats will cross the inter-house link. Run them over the Tailscale tailnet. Expect kai-server to briefly drop out of quorum and rejoin whenever the link flaps. That is churn, not data loss.
Consider a small UPS per site. etcd dislikes unclean shutdowns.

Out of scope

The Beelink GTR9 Pro (Ryzen AI Max+ 395, ~$1,799) for local LLM compute. Possible separate purchase in a year or two.

Kai purchases the two SER8s at her convenience, then the cluster-join work happens.

_Originally filed by @coilysiren on 2026-05-22T13:50:37Z - [https://github.com/coilysiren/infrastructure/issues/278](https://github.com/coilysiren/infrastructure/issues/278)_ **Problem** kai-server is currently the only K3s control-plane node. If it goes down, the whole control plane is down and nothing reschedules anywhere. It is a single point of failure that cannot tolerate any downtime. **Plan** Buy two Beelink SER8 mini PCs and join them as control-plane (server) nodes alongside kai-server. That makes a 3-node etcd cluster: N=3, quorum 2, tolerates one machine failure. Any single box, including kai-server, can go down and the cluster keeps scheduling. The two SER8s also host the public-facing services so blue-green deploys stop being manual. **Hardware** - Beelink SER8 (AMD Ryzen 7 8845HS, 8C/16T, 32GB DDR5, 1TB NVMe, 2x M.2 slots, 2.5GbE): https://www.bee-link.com/products/beelink-ser8-8845hs - Buy two. Each well under the ~$1k/machine budget. **What this gives, and what it does not** - Control-plane HA: yes. The 3-node cluster removes kai-server as a single point of failure. - Both SER8s live at the current house, kai-server is at the other house. The single point of failure moves from "one machine" to "the 2-node site." If that house goes fully dark, kai-server alone cannot hold quorum and scheduling stops. Still a real upgrade, since single-box failures and routine reboots are far more common than a whole-house outage. - Physical-distance backup is a separate goal. kai-server should still hold independent data backups and replicas of the public services so a true house-level disaster leaves it able to serve, even if degraded and manual. **Notes** - etcd Raft heartbeats will cross the inter-house link. Run them over the Tailscale tailnet. Expect kai-server to briefly drop out of quorum and rejoin whenever the link flaps. That is churn, not data loss. - Consider a small UPS per site. etcd dislikes unclean shutdowns. **Out of scope** - The Beelink GTR9 Pro (Ryzen AI Max+ 395, ~$1,799) for local LLM compute. Possible separate purchase in a year or two. Kai purchases the two SER8s at her convenience, then the cluster-join work happens.

coilysiren commented

2026-05-25 05:38:33 +00:00

Author

Owner

Probably gonna buy from this vendor - but with a goal of warm standby instead of HA

coilysiren closed this issue

2026-05-25 05:38:35 +00:00

coilysiren referenced this issue

2026-06-18 02:52:56 +00:00

feat(ansible): full-auto Goose job to converge fleet agentic-os pre-commit pins #372

coilysiren referenced this issue

2026-07-04 10:38:26 +00:00