CD: apply deploy/ cluster manifests on push to main (kill the manual kubectl apply) #346

New issue

Open

opened 2026-06-15 19:02:07 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-15 19:02:07 +00:00

Owner

Goal

A commit to infra main that changes a deploy/ cluster manifest should apply itself on the in-cluster deploy runner, not wait on a hand-run sudo k3s kubectl apply on kai-server. Today infra CI is config-validation only (AGENTS.md: "a commit to main is not a deploy"), so every manifest change carries a manual apply step. This was the last loose end of infra#345 / deploy#12: the eco-app deployer RBAC landed in deploy/forgejo-runner-deploy.yml but only reached the cluster when applied by hand.

What to add

A Forgejo workflow (.forgejo/workflows/deploy-infra.yml or similar), runs-on: deploy, push on deploy/** + workflow_dispatch, that kubectl applys the changed manifests. Mirrors the per-service deploy-<name>.yml pattern proven for atlas + eco-app, but for the cluster-infra layer.
Decide the apply surface: a curated allowlist of deploy/*.yml that are safe to auto-apply, vs. apply-everything-changed. Some deploy/ manifests (observability values, secretstore) may want to stay manual.

Nuances / blockers to design around

Bootstrapping / chicken-and-egg: the deploy runner is itself defined in deploy/forgejo-runner-deploy.yml. A workflow on that runner re-applying its own StatefulSet is workable (the apply is idempotent and the pod survives its own re-apply), but a breaking change to the runner manifest could strand the runner. The runner's own bootstrap apply stays manual.
Blast radius / RBAC: the deployer SA is deliberately narrow - namespaces get/create cluster-wide, plus RW scoped to the atlas and coilysiren-eco-app namespaces only. Applying forgejo-runner-deploy.yml (or any forgejo-namespace / ClusterRole resource) needs perms the deployer SA does not have today and arguably should never have (it would let CD rewrite its own RBAC - a privilege-escalation surface). Likely answer: infra CD applies app/namespace-scoped manifests but the cluster-RBAC + runner manifests stay an operator apply. Pin this down before wiring.
Validation gate: keep the existing config-validation CI as a required precondition so a malformed manifest never reaches kubectl apply.
Safety: per AGENTS.md, kubectl writes are confirm-gated for an operator; auto-applying removes that gate, so the allowlist above is load-bearing for blast control.

Relationship

Generalizes the per-service CD path (atlas, eco-app) to the infra repo's own deploy/ layer.
Predecessor context: infra#345, deploy#12, docs/cd-runner.md.

## Goal A commit to infra `main` that changes a `deploy/` cluster manifest should apply itself on the in-cluster `deploy` runner, not wait on a hand-run `sudo k3s kubectl apply` on kai-server. Today infra CI is config-validation only ([AGENTS.md](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/src/branch/main/AGENTS.md): "a commit to main is not a deploy"), so every manifest change carries a manual apply step. This was the last loose end of [infra#345](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/345) / [deploy#12](https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/12): the eco-app deployer RBAC landed in `deploy/forgejo-runner-deploy.yml` but only reached the cluster when applied by hand. ## What to add - A Forgejo workflow (`.forgejo/workflows/deploy-infra.yml` or similar), `runs-on: deploy`, `push` on `deploy/**` + `workflow_dispatch`, that `kubectl apply`s the changed manifests. Mirrors the per-service `deploy-<name>.yml` pattern proven for atlas + eco-app, but for the cluster-infra layer. - Decide the apply surface: a curated allowlist of `deploy/*.yml` that are safe to auto-apply, vs. apply-everything-changed. Some `deploy/` manifests (observability values, secretstore) may want to stay manual. ## Nuances / blockers to design around - **Bootstrapping / chicken-and-egg**: the `deploy` runner is *itself* defined in `deploy/forgejo-runner-deploy.yml`. A workflow on that runner re-applying its own StatefulSet is workable (the apply is idempotent and the pod survives its own re-apply), but a breaking change to the runner manifest could strand the runner. The runner's own bootstrap apply stays manual. - **Blast radius / RBAC**: the `deployer` SA is deliberately narrow - namespaces get/create cluster-wide, plus RW scoped to the `atlas` and `coilysiren-eco-app` namespaces only. Applying `forgejo-runner-deploy.yml` (or any `forgejo`-namespace / ClusterRole resource) needs perms the deployer SA does **not** have today and arguably should never have (it would let CD rewrite its own RBAC - a privilege-escalation surface). Likely answer: infra CD applies app/namespace-scoped manifests but the cluster-RBAC + runner manifests stay an operator apply. Pin this down before wiring. - **Validation gate**: keep the existing config-validation CI as a required precondition so a malformed manifest never reaches `kubectl apply`. - **Safety**: per AGENTS.md, kubectl writes are confirm-gated for an operator; auto-applying removes that gate, so the allowlist above is load-bearing for blast control. ## Relationship - Generalizes the per-service CD path (atlas, eco-app) to the infra repo's own `deploy/` layer. - Predecessor context: [infra#345](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/345), [deploy#12](https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/12), [docs/cd-runner.md](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/src/branch/main/docs/cd-runner.md).

coilysiren added the

label

2026-06-17 09:23:35 +00:00

coilysiren added

consult

and removed

labels

2026-06-18 06:21:18 +00:00