coilyco-flight-deck/infrastructure

Watch

Deploy-safety harness: get infra out of babysitted deploys (pre-apply gate + classification + auto-rollback + canary) #403

New issue

Closed

opened 2026-06-25 21:58:14 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-25 21:58:14 +00:00

Owner

Design goals for getting infra changes out of babysitted deploys. Stated as a slice for an infra explorer to expand into concrete work - not a finished spec.

The problem

Every infra issue is currently a babysat deploy, because the infra repo Flux-auto-deploys to the live k3s fleet and the human is the safety net in two automatable places:

Pre-apply validation: infra CI today only pylints Python (.forgejo/workflows/lint.yml) - it never kustomize builds, kubeconforms, or server-side dry-runs the actual manifests. A broken manifest reaches Flux with no gate.
Rollback: Flux does post-apply detection well (wait: true, timeout: 5m, healthChecks, prune: false, drift-correction) but does not auto-revert. A bad reconcile stays live until a human git-reverts.

So a human watches every apply to be the dry-run and the rollback. That does not scale to ~100 open infra issues.

Goal

Build the infra equivalent of ward's pre-commit/CI gate + reaper, so infra changes are safe to fire-and-forget and the backlog can drain autonomously (eventually under warded director). Replace human-as-safety-net with an automated gate + self-heal.

The harness (in leverage order)

Pre-apply validation gate. CI runs kustomize build + kubeconform (schema) + kubectl apply --dry-run=server (live admission/schema - the in-cluster runner can reach the API) + flux build/validate on changed overlays, before the push lands. Catches the majority of cluster-breaking errors with no human, and benefits all ~100 issues at once. Notably this is CI-only, not a cluster apply, so it is itself low blast radius.
Blast-radius classification. A label taxonomy separating fleet-singleton changes (runner, ingress, flux-system, external-secrets, networking) from isolated-service changes (a new service deploy, a dashboard, a values tweak). The dispatcher / pre-flight auto-fires the isolated-service majority and surfaces only singleton-touching changes for supervision. Likely turns most of the 100 fire-and-forget once the gate exists.
Auto-rollback (the reaper for deploys). On a failed Flux reconcile (health/timeout), auto-revert the offending commit so the cluster self-heals instead of waiting on a human. Mechanism open: a Flux-aware watcher that git-reverts on Kustomization Ready=False, or alert->revert.
Canary for the singletons. Stage runner/ingress/flux-system changes through a canary before the fleet adopts them (the runner already runs 2 replicas - make one a canary that adopts config first; or a staging namespace), so even the high-risk minority do not need a human.

The bootstrapping insight

Do NOT babysit 100 deploys - babysit the few that build this harness, then the rest drain autonomously. The first piece (the validation gate) is CI-only, so building it is itself fire-and-forget-safe: the thing that ends babysitting can be built without babysitting. Items 3-4 touch the live deploy loop / Flux / fleet singletons, so they want supervision while they roll out; after that, infra issues are gate-checked + self-healing and flow without a human.

Grounding (current state)

.forgejo/workflows/lint.yml - the Python-only CI this extends.
clusters/kai-server/{apps,infrastructure}.yaml - the Flux Kustomizations with wait/timeout/healthChecks/prune: false this builds on (post-apply detection already present; pre-apply gate + auto-rollback are the gaps).
deploy/forgejo-runner.yml - a fleet singleton with 2 replicas a canary could use.
infrastructure#401 - a current singleton change that has to be babysat under today's pattern (the motivating example).

Design goals for getting infra changes **out of babysitted deploys**. Stated as a slice for an infra explorer to expand into concrete work - not a finished spec. ## The problem Every infra issue is currently a babysat deploy, because the infra repo Flux-auto-deploys to the live k3s fleet and the **human is the safety net** in two automatable places: - **Pre-apply validation**: infra CI today only `pylint`s Python (`.forgejo/workflows/lint.yml`) - it never `kustomize build`s, `kubeconform`s, or server-side dry-runs the actual manifests. A broken manifest reaches Flux with no gate. - **Rollback**: Flux does post-apply *detection* well (`wait: true`, `timeout: 5m`, `healthChecks`, `prune: false`, drift-correction) but does **not** auto-revert. A bad reconcile stays live until a human git-reverts. So a human watches every apply to be the dry-run and the rollback. That does not scale to ~100 open infra issues. ## Goal Build the infra equivalent of ward's pre-commit/CI gate + reaper, so infra changes are **safe to fire-and-forget** and the backlog can drain autonomously (eventually under `warded director`). Replace human-as-safety-net with an automated gate + self-heal. ## The harness (in leverage order) 1. **Pre-apply validation gate.** CI runs `kustomize build` + `kubeconform` (schema) + `kubectl apply --dry-run=server` (live admission/schema - the in-cluster runner can reach the API) + `flux build`/validate on changed overlays, before the push lands. Catches the majority of cluster-breaking errors with no human, and benefits all ~100 issues at once. Notably this is **CI-only, not a cluster apply**, so it is itself low blast radius. 2. **Blast-radius classification.** A label taxonomy separating *fleet-singleton* changes (runner, ingress, flux-system, external-secrets, networking) from *isolated-service* changes (a new service deploy, a dashboard, a values tweak). The dispatcher / pre-flight auto-fires the isolated-service majority and surfaces only singleton-touching changes for supervision. Likely turns most of the 100 fire-and-forget once the gate exists. 3. **Auto-rollback** (the reaper for deploys). On a failed Flux reconcile (health/timeout), auto-revert the offending commit so the cluster self-heals instead of waiting on a human. Mechanism open: a Flux-aware watcher that git-reverts on `Kustomization` Ready=False, or alert->revert. 4. **Canary for the singletons.** Stage runner/ingress/flux-system changes through a canary before the fleet adopts them (the runner already runs 2 replicas - make one a canary that adopts config first; or a staging namespace), so even the high-risk minority do not need a human. ## The bootstrapping insight Do NOT babysit 100 deploys - babysit the **few that build this harness**, then the rest drain autonomously. The first piece (the validation gate) is CI-only, so building it is itself fire-and-forget-safe: the thing that ends babysitting can be built without babysitting. Items 3-4 touch the live deploy loop / Flux / fleet singletons, so they want supervision while they roll out; after that, infra issues are gate-checked + self-healing and flow without a human. ## Grounding (current state) - `.forgejo/workflows/lint.yml` - the Python-only CI this extends. - `clusters/kai-server/{apps,infrastructure}.yaml` - the Flux Kustomizations with `wait`/`timeout`/`healthChecks`/`prune: false` this builds on (post-apply detection already present; pre-apply gate + auto-rollback are the gaps). - `deploy/forgejo-runner.yml` - a fleet singleton with 2 replicas a canary could use. - infrastructure#401 - a current singleton change that has to be babysat under today's pattern (the motivating example).

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-01 08:07:50 +00:00

coilyco-ops commented

2026-07-10 12:36:18 +00:00

Owner

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Additional framing from the dispatcher:
infrastructure#403

pre-commit installed at .git/hooks/pre-commit
pre-commit installed at .git/hooks/commit-msg

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. > > Additional framing from the dispatcher: > infrastructure#403 --- pre-commit installed at .git/hooks/pre-commit pre-commit installed at .git/hooks/commit-msg --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilyco-ops removed the

autonomy

async-consult

label

2026-07-16 01:07:57 +00:00

coilyco-ops added

and removed

labels

2026-07-29 03:45:55 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-08-05 10:26:03 +00:00

coilyco-ops added the

role/platform

role/sysadmin

labels

2026-08-25 08:02:32 +00:00

coilyco-ops commented

2026-08-27 03:21:57 +00:00

Owner

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901).

Closing as too broad to schedule. The pre-apply validation gate is the one piece with clear leverage and it deserves its own concrete issue; the classification, auto-rollback, and canary legs are a design sketch nobody has picked up in two months.

This was priority/P3, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless.

If this is live work, reopen it. The whole set is recoverable with state:closed label:burndown-2026-08.

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901). Closing as too broad to schedule. The pre-apply validation gate is the one piece with clear leverage and it deserves its own concrete issue; the classification, auto-rollback, and canary legs are a design sketch nobody has picked up in two months. This was `priority/P3`, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless. If this is live work, reopen it. The whole set is recoverable with `state:closed label:burndown-2026-08`.

coilyco-ops

2026-08-27 03:21:58 +00:00

closed this issue
added the
burndown-2026-08
label

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#403

No description provided.

Rows
Columns