Expand Flux reconciler scope to the Sirens lanes, so a hand-rolled config shows as drift instead of an invisible divergence #876

New issue

Open

opened 2026-08-19 19:56:53 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-19 19:56:53 +00:00

Owner

Continues the staged adoption in docs/flux-gitops.md and #366. Kai's call, prompted by a concrete near-miss on stream day.

What prompted it

Telemetry from the 2026-08-19 Temporal stream shows Sirens Dowel calling five MCP servers that do not appear in sirens-dowel-mcp-roster.yml on coilyco-bridge/deploy@main: fetch, scratchpad, moxn, calculator, skills.

That was not drift. Kai was rolling changes in realtime during demo prep, which is legitimate and worked. The problem is that the divergence was invisible: nothing distinguished "deliberately rolled ahead of main" from "quietly stale," and there was no surface where either showed up.

It cost something the same morning. An assessment of the lane's dependencies read the git roster, concluded Dowel had no moxn dependency, and reported that as fact. The deployed lane was calling owl-glass.moxn.dev and failing against it ten times during the stream. The reasoning was sound and the source was wrong.

Where adoption currently stops

Reading clusters/kai-server/ as it stands:

apps reconciles ./deploy/_flux/apps from this repo, owning the scoped Forgejo runner pools. prune: false.
bridge-deploy GitRepository exists and points at coilyco-bridge/deploy, branch main.
bridge-deploy Kustomization is suspend: true, path: ./services, prune: false.

The doc explains the suspension precisely:

Its bridge-deploy Kustomization resolves the deploy-side ./services aggregation root, which currently exposes only the literal Tangled public bridge. It remains suspended with prune: false until...

So the mechanism is already built and deliberately parked. This issue is about finishing Stage 3 for the Sirens lanes specifically, not about inventing anything.

Prerequisites, which are real and ordered

Stage 2 is not done. The doc gates Stage 3 behind flipping prune: true on apps once drift correction is trusted, unsuspending the infrastructure Kustomization, and adding dependsOn: [{name: infrastructure}]. apps.yaml still carries prune: false and no dependsOn. Jumping to the Sirens lanes without that ordering skips the layer that proves drift correction works on something cheap.

The imperative-writer check is the hard one here. Stage 3 requires confirming "no imperative writer overlaps before more services/<name>/ slices are enrolled." The Sirens lane has at least two:

scripts/rollout-sirens-dowel.sh - the normal rollout path.
scripts/refresh-moxn-token.sh - mints a token, writes SSM and a local store, waits for the ExternalSecret to sync, then restarts the deployment. Its own comments record that ordering is load bearing and that getting it wrong crashloops the pod on a 2m connect timeout.

Neither is a plain kubectl apply of a manifest Flux would re-apply identically, which is the case the doc's break-glass section says does not fight. rollout-sirens-dowel.sh folds the lane's manifests into a config hash so that editing a ConfigMap rolls the pod. Flux applying the same ConfigMap would not reproduce that roll, so a naive adoption silently breaks the edit-rolls-the-pod property the lane depends on.

The tension worth deciding deliberately

Reconciliation and realtime rolling are in direct conflict, and the conflict peaks exactly when it hurts most.

With bridge-deploy active on a 10m interval, a hand-rolled change ahead of main gets reverted within ten minutes. On 2026-08-19 that would have undone Kai's prep mid-window. The behaviour that made today's divergence invisible is the same behaviour that made today's prep possible.

Options, and none is obviously right:

Enroll the lane fully. Git becomes the truth, hand rolls get reverted, and prep means committing. Strongest guarantee, highest friction on exactly the days that need speed.
Enroll with prune: false and treat Flux as a drift detector first. Reconcile, but pair it with an alert on divergence rather than trusting silent correction. Makes the invisible visible without taking away the realtime path. Probably the right first step given Stage 2 is unproven.
Suspend deliberately during demo windows. Add suspending bridge-deploy to demo-lockdown.sh alongside the controller sweep, so realtime rolling is an explicit, logged mode rather than a permanent hole.
Leave the Sirens lanes imperative and fix the reporting instead. If the only real cost was an agent reading git and reporting on the cluster, a cheaper fix is making the deployed roster readable. Worth pricing before spending the adoption effort.

Scope note

The .ward / AOS boundary in aos#778 is not touched here. This is about which manifests Flux reconciles, not about who owns policy.

Related: #366 (origin and staged-rollout checklist), #367 (Tangled knot, separate workstream), coilyco-bridge/deploy#712 (a dead third-party MCP wedging this lane's rollout, which is the failure mode adoption has to not make worse).

Continues the staged adoption in [`docs/flux-gitops.md`](../src/branch/main/docs/flux-gitops.md) and `#366`. Kai's call, prompted by a concrete near-miss on stream day. ## What prompted it Telemetry from the 2026-08-19 Temporal stream shows Sirens Dowel calling five MCP servers that do not appear in `sirens-dowel-mcp-roster.yml` on `coilyco-bridge/deploy@main`: `fetch`, `scratchpad`, `moxn`, `calculator`, `skills`. **That was not drift.** Kai was rolling changes in realtime during demo prep, which is legitimate and worked. The problem is that the divergence was **invisible**: nothing distinguished "deliberately rolled ahead of main" from "quietly stale," and there was no surface where either showed up. It cost something the same morning. An assessment of the lane's dependencies read the git roster, concluded Dowel had no moxn dependency, and reported that as fact. The deployed lane was calling `owl-glass.moxn.dev` and failing against it ten times during the stream. The reasoning was sound and the source was wrong. ## Where adoption currently stops Reading `clusters/kai-server/` as it stands: * `apps` reconciles `./deploy/_flux/apps` from this repo, owning the scoped Forgejo runner pools. `prune: false`. * `bridge-deploy` **GitRepository exists** and points at `coilyco-bridge/deploy`, branch `main`. * `bridge-deploy` **Kustomization is `suspend: true`**, `path: ./services`, `prune: false`. The doc explains the suspension precisely: > Its `bridge-deploy` Kustomization resolves the deploy-side `./services` aggregation root, which currently exposes only the literal Tangled public bridge. It remains suspended with `prune: false` until... So the mechanism is already built and deliberately parked. This issue is about finishing Stage 3 for the Sirens lanes specifically, not about inventing anything. ## Prerequisites, which are real and ordered **Stage 2 is not done.** The doc gates Stage 3 behind flipping `prune: true` on `apps` once drift correction is trusted, unsuspending the `infrastructure` Kustomization, and adding `dependsOn: [{name: infrastructure}]`. `apps.yaml` still carries `prune: false` and no `dependsOn`. Jumping to the Sirens lanes without that ordering skips the layer that proves drift correction works on something cheap. **The imperative-writer check is the hard one here.** Stage 3 requires confirming "no imperative writer overlaps before more `services/<name>/` slices are enrolled." The Sirens lane has at least two: * `scripts/rollout-sirens-dowel.sh` - the normal rollout path. * `scripts/refresh-moxn-token.sh` - mints a token, writes SSM **and** a local store, waits for the ExternalSecret to sync, then restarts the deployment. Its own comments record that ordering is load bearing and that getting it wrong crashloops the pod on a 2m connect timeout. Neither is a plain `kubectl apply` of a manifest Flux would re-apply identically, which is the case the doc's break-glass section says does not fight. `rollout-sirens-dowel.sh` folds the lane's manifests into a config hash so that editing a ConfigMap rolls the pod. Flux applying the same ConfigMap would **not** reproduce that roll, so a naive adoption silently breaks the edit-rolls-the-pod property the lane depends on. ## The tension worth deciding deliberately Reconciliation and realtime rolling are in direct conflict, and the conflict peaks exactly when it hurts most. With `bridge-deploy` active on a 10m interval, a hand-rolled change ahead of `main` gets **reverted within ten minutes**. On 2026-08-19 that would have undone Kai's prep mid-window. The behaviour that made today's divergence invisible is the same behaviour that made today's prep possible. Options, and none is obviously right: * **Enroll the lane fully.** Git becomes the truth, hand rolls get reverted, and prep means committing. Strongest guarantee, highest friction on exactly the days that need speed. * **Enroll with `prune: false` and treat Flux as a drift *detector* first.** Reconcile, but pair it with an alert on divergence rather than trusting silent correction. Makes the invisible visible without taking away the realtime path. Probably the right first step given Stage 2 is unproven. * **Suspend deliberately during demo windows.** Add suspending `bridge-deploy` to `demo-lockdown.sh` alongside the controller sweep, so realtime rolling is an explicit, logged mode rather than a permanent hole. * **Leave the Sirens lanes imperative and fix the reporting instead.** If the only real cost was an agent reading git and reporting on the cluster, a cheaper fix is making the deployed roster readable. Worth pricing before spending the adoption effort. ## Scope note The `.ward` / AOS boundary in `aos#778` is not touched here. This is about which manifests Flux reconciles, not about who owns policy. Related: `#366` (origin and staged-rollout checklist), `#367` (Tangled knot, separate workstream), `coilyco-bridge/deploy#712` (a dead third-party MCP wedging this lane's rollout, which is the failure mode adoption has to not make worse).

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:07 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:57 +00:00

coilyco-ops added this to the CD and rollout milestone

2026-08-29 06:10:07 +00:00