Migrate kai-server systemd units into the k3s cluster #17

New issue

Closed

opened 2026-05-23 20:54:27 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:27 +00:00

Owner

Originally filed by @coilysiren on 2026-05-22T10:53:44Z - https://github.com/coilysiren/infrastructure/issues/261

Goal - Move the always-on services currently running as systemd units on kai-server into the k3s homelab cluster, so supervision lives on one control plane.

Why - kai-server already runs k3s. Today supervision is split-brain: some workloads in systemd, some in k3s, two control planes and two places to look when something is down. Consolidating removes the split. Any workload that wants Ingress, cert-manager TLS, or ExternalSecrets belongs in the cluster regardless.

Priority - Low. Parked behind current work, filed so it does not get lost. Not starting now.

Steps

Enumerate current state on kai-server: systemctl list-units --type=service plus systemctl list-timers. Record each unit's ExecStart args verbatim - the k3s-upgrade playbook already flags that custom ExecStart args are easy to lose.
Classify each unit:
- always-on service -> Deployment with replicas: 1
- scheduled timer -> CronJob
- genuinely host-level (k3s itself, node bootstrap) -> stays in systemd
Containerize anything not already shipping as an image.
Write plain manifests (plain over Kustomize, per the kubernetes skill).
Wire secrets via ExternalSecrets, add Ingress + cert-manager where a unit needs a hostname or TLS.
Cut over one unit at a time: bring the k8s workload up, confirm healthy, then systemctl disable --now the old unit.

Out of scope - k3s itself and host-level units stay as systemd. The goal is consolidating app workloads, not making the node itself a pod.

_Originally filed by @coilysiren on 2026-05-22T10:53:44Z - [https://github.com/coilysiren/infrastructure/issues/261](https://github.com/coilysiren/infrastructure/issues/261)_ **Goal** - Move the always-on services currently running as systemd units on kai-server into the k3s homelab cluster, so supervision lives on one control plane. **Why** - kai-server already runs k3s. Today supervision is split-brain: some workloads in systemd, some in k3s, two control planes and two places to look when something is down. Consolidating removes the split. Any workload that wants Ingress, cert-manager TLS, or ExternalSecrets belongs in the cluster regardless. **Priority** - Low. Parked behind current work, filed so it does not get lost. Not starting now. **Steps** 1. Enumerate current state on kai-server: `systemctl list-units --type=service` plus `systemctl list-timers`. Record each unit's `ExecStart` args verbatim - the k3s-upgrade playbook already flags that custom `ExecStart` args are easy to lose. 2. Classify each unit: - always-on service -> `Deployment` with `replicas: 1` - scheduled timer -> `CronJob` - genuinely host-level (k3s itself, node bootstrap) -> stays in systemd 3. Containerize anything not already shipping as an image. 4. Write plain manifests (plain over Kustomize, per the kubernetes skill). 5. Wire secrets via ExternalSecrets, add Ingress + cert-manager where a unit needs a hostname or TLS. 6. Cut over one unit at a time: bring the k8s workload up, confirm healthy, then `systemctl disable --now` the old unit. **Out of scope** - k3s itself and host-level units stay as systemd. The goal is consolidating app workloads, not making the node itself a pod.

coilysiren added the

label

2026-06-04 08:16:57 +00:00

coilysiren commented

2026-06-17 08:22:58 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:22:58 +00:00