k3s.service runs a stale pre-migration checkout; canonical repo commits don't deploy #306

New issue

Open

opened 2026-06-12 10:06:12 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-12 10:06:12 +00:00

Owner

Summary

The live k3s.service on kai-server runs from a stale, pre-org-migration checkout, not the canonical org-dir repo. This means commits to coilyco-flight-deck/infrastructure do not reach the running k3s until the stale script is manually synced.

Detail

Unit: ExecStart=bash -c /home/kai/projects/infrastructure/scripts/k3s-start.sh (and an ExecStartPre dos2unix on the same path).
~/projects/infrastructure is HEAD 6742868, remote git@github.com:coilysiren/infrastructure.git (old GitHub remote).
Canonical is ~/projects/coilyco-flight-deck/infrastructure, remote forgejo.coilysiren.me/coilyco-flight-deck/infrastructure, currently HEAD fb57f2d.

So any k3s startup change (e.g. the kubelet reservation in fb57f2d) is authored in the canonical repo but silently not deployed.

Fix

Re-point the unit at the canonical org-dir checkout and retire the stale clone:

Edit k3s.service ExecStart + ExecStartPre to /home/kai/projects/coilyco-flight-deck/infrastructure/scripts/k3s-start.sh.
systemctl daemon-reload and restart k3s with --no-block (Type=notify hang, #170).
Remove ~/projects/infrastructure once nothing else references it.

Until then, the stopgap is cp the canonical k3s-start.sh into the stale path before each k3s restart (see /tmp/apply-kubelet-reservation.sh).

Surfaced 2026-06-12 while wiring kubelet CPU/memory reservations (#151).

## Summary The live `k3s.service` on kai-server runs from a **stale, pre-org-migration checkout**, not the canonical org-dir repo. This means commits to `coilyco-flight-deck/infrastructure` do not reach the running k3s until the stale script is manually synced. ## Detail - Unit: `ExecStart=bash -c /home/kai/projects/infrastructure/scripts/k3s-start.sh` (and an `ExecStartPre` `dos2unix` on the same path). - `~/projects/infrastructure` is HEAD `6742868`, remote `git@github.com:coilysiren/infrastructure.git` (old GitHub remote). - Canonical is `~/projects/coilyco-flight-deck/infrastructure`, remote `forgejo.coilysiren.me/coilyco-flight-deck/infrastructure`, currently HEAD `fb57f2d`. So any k3s startup change (e.g. the kubelet reservation in fb57f2d) is authored in the canonical repo but silently not deployed. ## Fix Re-point the unit at the canonical org-dir checkout and retire the stale clone: 1. Edit `k3s.service` `ExecStart` + `ExecStartPre` to `/home/kai/projects/coilyco-flight-deck/infrastructure/scripts/k3s-start.sh`. 2. `systemctl daemon-reload` and restart k3s with `--no-block` (Type=notify hang, #170). 3. Remove `~/projects/infrastructure` once nothing else references it. Until then, the stopgap is `cp` the canonical `k3s-start.sh` into the stale path before each k3s restart (see `/tmp/apply-kubelet-reservation.sh`). Surfaced 2026-06-12 while wiring kubelet CPU/memory reservations (#151).

coilysiren referenced this issue

2026-06-12 11:13:08 +00:00

k3s.service pkill -9 containerd-shim orphans container payloads on every restart, causing recurring lock-blocked crashloops (CI, o11y) #311

coilysiren referenced this issue

2026-06-12 11:14:23 +00:00

k3s.service pkill -9 containerd-shim orphans container payloads on every restart, causing recurring lock-blocked crashloops (CI, o11y) #311

coilysiren added the

label

2026-06-17 07:06:36 +00:00

coilysiren added the

interactive

label