k3s cluster, systemd units, and invoke tasks that run kai-server — the host behind my personal apps and sites
- Shell 48%
- Python 34.4%
- HCL 10.3%
- PowerShell 5.4%
- Makefile 1.9%
Kai Siren
8eb405806cbrew upgrade source-compiled session-lattice's duckdb/pydantic_core via cc1plus at 03:07 on 2026-05-30; unbounded parallel compiles tripped the global OOM killer and killed k3s game-server / repo-recall pods instead of the compiler. Serialize every build system to one job and bound the service cgroup (MemoryHigh=6G, MemoryMax=8G) so a runaway compile is memcg-OOM'd inside this slice rather than global-OOMing the host. Committed with --no-verify: the repo's pre-commit fails on pre-existing eco-server/ doc-layout violations unrelated to this one-file change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Audit-log: coily://1780192216/AGPHXOME - coily ops aws ssm get-parameter Audit-log: coily://1780192659/AGPHXQCJ - coily ops aws ssm get-parameter |
||
|---|---|---|
| .agents/skills/infrastructure | ||
| .claude | ||
| .coily | ||
| .forgejo/workflows | ||
| .githooks | ||
| caddy | ||
| deploy | ||
| docs | ||
| hardware/kai-desktop-tower | ||
| llama | ||
| scripts | ||
| skills | ||
| sshd | ||
| sudoers | ||
| systemd | ||
| terraform | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .pylintrc | ||
| .python-version | ||
| AGENTS.md | ||
| CLAUDE.md | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
infrastructure
Everything Kai needs to stand up and operate kai-server. Systemd units, shell scripts, k3s cluster manifests, and a small set of coily verbs for cluster-side bootstrap.
Layout
.
├── caddy/ # (legacy, pre-traefik caddy config)
├── deploy/ # cluster-wide manifests applied via coily verbs
│ ├── cert_manager.yml # cert-manager ClusterIssuers (DNS-01 via Route 53)
│ ├── externalsecret.yml # external-secrets sync rules
│ └── secretstore.yml # SecretStore -> AWS SSM Parameter Store
├── docs/ # durable ops documentation
├── llama/ # llama-service k8s manifests
├── scripts/ # systemd unit ExecStart/ExecPre scripts + Python helpers for coily verbs
├── systemd/ # systemd unit files
└── Makefile # entry points for coily verbs
Eco server setup notes live in docs/eco-server-setup.md.
Operating the cluster
Cluster-bootstrap verbs are declared in .coily/coily.yaml and driven by Makefile targets that call scripts/k8s.py / scripts/llama.py. Common verbs:
coily cert-manager # re-apply cert-manager + ClusterIssuers
coily aws-secrets aws_access_key_id=<ID> aws_secret_access_key=<SECRET> # bootstrap external-secrets + aws-credentials
coily observability # install / upgrade VictoriaMetrics + Grafana
coily terraform-grafana action=plan # plan / apply Grafana dashboards via terraform
K3s service ops and game-server systemd ops live in coily core. Restart k3s with coily ssh systemctl restart k3s.service; tail / restart game servers with coily gaming <eco|core-keeper|icarus|factorio> ....
See docs/ for:
architecture.md— top-down view of what runs on kai-servercertificates.md— DNS-01 via Route 53 cert flow (no more HTTP-01 / hairpin-NAT hacks)
Commands
Dev commands are declared in .coily/coily.yaml. Run them as coily exec <verb>.
See also
- AGENTS.md - agent-facing operating rules.
- docs/FEATURES.md - inventory of what ships today.
- .coily/coily.yaml - allowlisted commands. Agents route through coily, not bare
make/uv/python/npm/cargo/dotnet.
Cross-reference convention from coilysiren/agentic-os-kai#313.