Backfill ansible role for kai-server systemd units (post checkout-move) #355

New issue

Open

opened 2026-06-17 07:55:12 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-17 07:55:12 +00:00

Owner

Backfill an ansible role for kai-server's hand-installed systemd units

On 2026-06-17 the live infra checkout was moved from the retired top-level
~/projects/infrastructure to the canonical-tracking
~/projects/coilyco-flight-deck/infrastructure. Six systemd units on
kai-server hardcoded the old path and were hand-edited on-box to unblock
the move (backed up to /etc/systemd/system/.pre-move-bak-20260617-004800):

k3s.service (ExecStartPre dos2unix + ExecStart)
eco-server.service
factorio-server.service
factorio-backup.service (timer-driven)
icarus-server.service
core-keeper-server.service

These units are not managed by ansible today, so a fresh kai-server
rebuild would not reproduce them. Per the authoring-vs-rollout doctrine, the
unit definitions belong in an ansible role (rollout), templating the checkout
path rather than hardcoding it.

Scope

New ansible role (kai-server scoped) that installs the 6 unit files with the
scripts path templated to the canonical checkout.
Converge, confirm it renders byte-equivalent to the current hand-edited
units, then the on-box .pre-move-bak-* backup can be dropped.

k3s wedge fix (system-reserved/kube-reserved) that motivated the watched
restart: landed in scripts/k3s-start.sh.
o2r FQT7 (apiserver Bad Gateway / power-cycle recovery) is the incident this
hardening closes out.

Side-finding (separate, low priority)

During the k3s restart, journald logged Failed to allocate directory watch: Too many open files. Current limits: fs.inotify.max_user_watches=65536,
max_user_instances=128. k3s + containerd can exhaust instances; worth a
sysctl bump (e.g. instances 512, watches 524288) if it recurs.

## Backfill an ansible role for kai-server's hand-installed systemd units On 2026-06-17 the live infra checkout was moved from the retired top-level `~/projects/infrastructure` to the canonical-tracking `~/projects/coilyco-flight-deck/infrastructure`. Six systemd units on kai-server hardcoded the old path and were **hand-edited on-box** to unblock the move (backed up to `/etc/systemd/system/.pre-move-bak-20260617-004800`): - `k3s.service` (ExecStartPre dos2unix + ExecStart) - `eco-server.service` - `factorio-server.service` - `factorio-backup.service` (timer-driven) - `icarus-server.service` - `core-keeper-server.service` These units are **not** managed by ansible today, so a fresh kai-server rebuild would not reproduce them. Per the authoring-vs-rollout doctrine, the unit definitions belong in an ansible role (rollout), templating the checkout path rather than hardcoding it. ### Scope - New ansible role (kai-server scoped) that installs the 6 unit files with the scripts path templated to the canonical checkout. - Converge, confirm it renders byte-equivalent to the current hand-edited units, then the on-box `.pre-move-bak-*` backup can be dropped. ### Related - k3s wedge fix (system-reserved/kube-reserved) that motivated the watched restart: landed in `scripts/k3s-start.sh`. - o2r FQT7 (apiserver Bad Gateway / power-cycle recovery) is the incident this hardening closes out. ### Side-finding (separate, low priority) During the k3s restart, journald logged `Failed to allocate directory watch: Too many open files`. Current limits: `fs.inotify.max_user_watches=65536`, `max_user_instances=128`. k3s + containerd can exhaust instances; worth a sysctl bump (e.g. instances 512, watches 524288) if it recurs.

coilysiren added the

label

2026-06-17 09:23:33 +00:00

coilysiren added

interactive

and removed

labels

2026-06-18 06:22:13 +00:00