Deploy-safety harness: get infra out of babysitted deploys (pre-apply gate + classification + auto-rollback + canary) #403

New issue

Open

opened 2026-06-25 21:58:14 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-25 21:58:14 +00:00

Owner

Design goals for getting infra changes out of babysitted deploys. Stated as a slice for an infra explorer to expand into concrete work - not a finished spec.

The problem

Every infra issue is currently a babysat deploy, because the infra repo Flux-auto-deploys to the live k3s fleet and the human is the safety net in two automatable places:

Pre-apply validation: infra CI today only pylints Python (.forgejo/workflows/lint.yml) - it never kustomize builds, kubeconforms, or server-side dry-runs the actual manifests. A broken manifest reaches Flux with no gate.
Rollback: Flux does post-apply detection well (wait: true, timeout: 5m, healthChecks, prune: false, drift-correction) but does not auto-revert. A bad reconcile stays live until a human git-reverts.

So a human watches every apply to be the dry-run and the rollback. That does not scale to ~100 open infra issues.

Goal

Build the infra equivalent of ward's pre-commit/CI gate + reaper, so infra changes are safe to fire-and-forget and the backlog can drain autonomously (eventually under warded director). Replace human-as-safety-net with an automated gate + self-heal.

The harness (in leverage order)

Pre-apply validation gate. CI runs kustomize build + kubeconform (schema) + kubectl apply --dry-run=server (live admission/schema - the in-cluster runner can reach the API) + flux build/validate on changed overlays, before the push lands. Catches the majority of cluster-breaking errors with no human, and benefits all ~100 issues at once. Notably this is CI-only, not a cluster apply, so it is itself low blast radius.
Blast-radius classification. A label taxonomy separating fleet-singleton changes (runner, ingress, flux-system, external-secrets, networking) from isolated-service changes (a new service deploy, a dashboard, a values tweak). The dispatcher / pre-flight auto-fires the isolated-service majority and surfaces only singleton-touching changes for supervision. Likely turns most of the 100 fire-and-forget once the gate exists.
Auto-rollback (the reaper for deploys). On a failed Flux reconcile (health/timeout), auto-revert the offending commit so the cluster self-heals instead of waiting on a human. Mechanism open: a Flux-aware watcher that git-reverts on Kustomization Ready=False, or alert->revert.
Canary for the singletons. Stage runner/ingress/flux-system changes through a canary before the fleet adopts them (the runner already runs 2 replicas - make one a canary that adopts config first; or a staging namespace), so even the high-risk minority do not need a human.

The bootstrapping insight

Do NOT babysit 100 deploys - babysit the few that build this harness, then the rest drain autonomously. The first piece (the validation gate) is CI-only, so building it is itself fire-and-forget-safe: the thing that ends babysitting can be built without babysitting. Items 3-4 touch the live deploy loop / Flux / fleet singletons, so they want supervision while they roll out; after that, infra issues are gate-checked + self-healing and flow without a human.

Grounding (current state)

.forgejo/workflows/lint.yml - the Python-only CI this extends.
clusters/kai-server/{apps,infrastructure}.yaml - the Flux Kustomizations with wait/timeout/healthChecks/prune: false this builds on (post-apply detection already present; pre-apply gate + auto-rollback are the gaps).
deploy/forgejo-runner.yml - a fleet singleton with 2 replicas a canary could use.
infrastructure#401 - a current singleton change that has to be babysat under today's pattern (the motivating example).

Design goals for getting infra changes **out of babysitted deploys**. Stated as a slice for an infra explorer to expand into concrete work - not a finished spec. ## The problem Every infra issue is currently a babysat deploy, because the infra repo Flux-auto-deploys to the live k3s fleet and the **human is the safety net** in two automatable places: - **Pre-apply validation**: infra CI today only `pylint`s Python (`.forgejo/workflows/lint.yml`) - it never `kustomize build`s, `kubeconform`s, or server-side dry-runs the actual manifests. A broken manifest reaches Flux with no gate. - **Rollback**: Flux does post-apply *detection* well (`wait: true`, `timeout: 5m`, `healthChecks`, `prune: false`, drift-correction) but does **not** auto-revert. A bad reconcile stays live until a human git-reverts. So a human watches every apply to be the dry-run and the rollback. That does not scale to ~100 open infra issues. ## Goal Build the infra equivalent of ward's pre-commit/CI gate + reaper, so infra changes are **safe to fire-and-forget** and the backlog can drain autonomously (eventually under `warded director`). Replace human-as-safety-net with an automated gate + self-heal. ## The harness (in leverage order) 1. **Pre-apply validation gate.** CI runs `kustomize build` + `kubeconform` (schema) + `kubectl apply --dry-run=server` (live admission/schema - the in-cluster runner can reach the API) + `flux build`/validate on changed overlays, before the push lands. Catches the majority of cluster-breaking errors with no human, and benefits all ~100 issues at once. Notably this is **CI-only, not a cluster apply**, so it is itself low blast radius. 2. **Blast-radius classification.** A label taxonomy separating *fleet-singleton* changes (runner, ingress, flux-system, external-secrets, networking) from *isolated-service* changes (a new service deploy, a dashboard, a values tweak). The dispatcher / pre-flight auto-fires the isolated-service majority and surfaces only singleton-touching changes for supervision. Likely turns most of the 100 fire-and-forget once the gate exists. 3. **Auto-rollback** (the reaper for deploys). On a failed Flux reconcile (health/timeout), auto-revert the offending commit so the cluster self-heals instead of waiting on a human. Mechanism open: a Flux-aware watcher that git-reverts on `Kustomization` Ready=False, or alert->revert. 4. **Canary for the singletons.** Stage runner/ingress/flux-system changes through a canary before the fleet adopts them (the runner already runs 2 replicas - make one a canary that adopts config first; or a staging namespace), so even the high-risk minority do not need a human. ## The bootstrapping insight Do NOT babysit 100 deploys - babysit the **few that build this harness**, then the rest drain autonomously. The first piece (the validation gate) is CI-only, so building it is itself fire-and-forget-safe: the thing that ends babysitting can be built without babysitting. Items 3-4 touch the live deploy loop / Flux / fleet singletons, so they want supervision while they roll out; after that, infra issues are gate-checked + self-healing and flow without a human. ## Grounding (current state) - `.forgejo/workflows/lint.yml` - the Python-only CI this extends. - `clusters/kai-server/{apps,infrastructure}.yaml` - the Flux Kustomizations with `wait`/`timeout`/`healthChecks`/`prune: false` this builds on (post-apply detection already present; pre-apply gate + auto-rollback are the gaps). - `deploy/forgejo-runner.yml` - a fleet singleton with 2 replicas a canary could use. - infrastructure#401 - a current singleton change that has to be babysat under today's pattern (the motivating example).

coilyco-ops added the

consult

labels

2026-07-01 08:07:50 +00:00

coilyco-ops added

interactive

and removed

consult

labels

2026-07-09 21:46:23 +00:00

coilyco-ops added

consult

and removed

interactive

labels

2026-07-10 09:00:02 +00:00

coilyco-ops commented

2026-07-10 12:36:18 +00:00

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Additional framing from the dispatcher:
infrastructure#403

pre-commit installed at .git/hooks/pre-commit
pre-commit installed at .git/hooks/commit-msg

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. > > Additional framing from the dispatcher: > infrastructure#403 --- pre-commit installed at .git/hooks/pre-commit pre-commit installed at .git/hooks/commit-msg --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`