coilyco-flight-deck/infrastructure

Watch

Flux infrastructure and bridge-deploy Kustomizations are suspended, so those paths are not reconciling #837

New issue

Open

opened 2026-08-15 16:52:06 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 16:52:06 +00:00

Owner

Summary

Two of the four Flux Kustomizations are suspended. Anything committed to those paths is not being applied to the cluster, and nothing surfaces that fact except reading the object.

Evidence

NAMESPACE     NAME             AGE   READY   STATUS
flux-system   apps             58d   False   health check failed ...
flux-system   bridge-deploy    36d
flux-system   flux-system      58d   True    Applied revision: main@sha1:d0ff97a...
flux-system   infrastructure   58d

The blank READY column on bridge-deploy and infrastructure is because both carry:

spec:
  interval: 10m
  prune: false
  suspend: true

infrastructure has been in place 58 days and bridge-deploy 36 days. I did not establish when suspension was applied or whether it was deliberate.

Why it matters

Changes landed on main for those paths do not reach the cluster, so git stops being a truthful description of running state
Combined with prune: false on every Kustomization, removing a declaration deletes nothing, so drift accumulates in one direction only
A suspended Kustomization reports no Ready condition at all rather than an explicit warning, so it reads as blank rather than as broken

Ask

Confirm whether each suspension is intentional. If it is deliberate, record why in the repo next to the manifest so the next reader does not have to guess. If it is leftover from an incident, resume it and reconcile the accumulated drift deliberately rather than all at once.

Acceptance

Every Flux Kustomization is either actively reconciling or carries a committed note explaining why it is suspended.

## Summary Two of the four Flux Kustomizations are suspended. Anything committed to those paths is not being applied to the cluster, and nothing surfaces that fact except reading the object. ## Evidence ``` NAMESPACE NAME AGE READY STATUS flux-system apps 58d False health check failed ... flux-system bridge-deploy 36d flux-system flux-system 58d True Applied revision: main@sha1:d0ff97a... flux-system infrastructure 58d ``` The blank `READY` column on `bridge-deploy` and `infrastructure` is because both carry: ```yaml spec: interval: 10m prune: false suspend: true ``` `infrastructure` has been in place 58 days and `bridge-deploy` 36 days. I did not establish when suspension was applied or whether it was deliberate. ## Why it matters - Changes landed on `main` for those paths do not reach the cluster, so git stops being a truthful description of running state - Combined with `prune: false` on every Kustomization, removing a declaration deletes nothing, so drift accumulates in one direction only - A suspended Kustomization reports no Ready condition at all rather than an explicit warning, so it reads as blank rather than as broken ## Ask Confirm whether each suspension is intentional. If it is deliberate, record why in the repo next to the manifest so the next reader does not have to guess. If it is leftover from an incident, resume it and reconcile the accumulated drift deliberately rather than all at once. ## Acceptance Every Flux Kustomization is either actively reconciling or carries a committed note explaining why it is suspended.

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:10 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:00 +00:00

coilyco-ops referenced this issue

2026-08-28 21:12:13 +00:00

Flux apps Kustomization: one dead disposable runner fails the health gate for the entire app layer #836

coilyco-ops referenced this issue

2026-08-28 21:12:35 +00:00

forgejo-runner-deploy-atlas has crash-looped for 14h against a repository that does not exist #834

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops commented

2026-08-29 05:14:20 +00:00

Author

Owner

Confirmed suspended, and the status field says something worse than "suspended". Verified 2026-08-29 ~05:15Z.

#981 recorded these two as still suspended by reading kubectl get kustomization -A. That holds, and reading the objects rather than the table adds a fact the table hides.

NAME             SUSPEND   READY   LAST APPLIED REVISION
apps             false     True    main@sha1:837cb0c2
coredns          false     True    main@sha1:837cb0c2
flux-system      false     True    main@sha1:837cb0c2
bridge-deploy    TRUE      -       none
infrastructure   TRUE      -       none

lastAppliedRevision: none on both. Not a stale revision, not a revision from before the suspension. None. These two Kustomizations have never successfully applied anything.

Ages are 72 days for infrastructure and 49 days for bridge-deploy.

Why that changes the issue

"Suspended" implies a working reconciliation that someone paused, where resuming returns you to a known state. That is not what these are. They were created, never reconciled once, and have sat inert for 72 and 49 days respectively.

So the risk on resuming is not "Flux catches up on a backlog of drift". It is that the first successful apply these objects ever perform will happen against a live cluster whose entire current state was established by other means. Nobody has seen what that apply does, because it has never run.

Everything in the infrastructure and bridge-deploy trees is therefore reaching the cluster by hand or by CD scripts rather than by GitOps. That is the standing explanation for the hand-applied drift this repository keeps finding, #910's dead service.loadBalancerIP key and hand-applied Service drift being the example that already closed.

What this issue should now ask for

Not "unsuspend". The safe sequence is:

Diff before apply. flux diff kustomization or an equivalent dry run against the live cluster, for each of the two, so the first-ever apply is read before it is taken.
Reconcile the diff deliberately, deciding per-resource whether the tracked definition or the live object is correct. On a 72-day-old tree with no reconciliation history, expect the tracked side to be wrong in places, not just the cluster.
Only then unsuspend, and watch the first real apply.

aosguard ops kubectl exposes diff, so step 1 is reachable from this seat. I did not run it, because a meaningful diff wants an attended operator watching the output against a cluster serving live traffic, and this is a backlog pass rather than a change window.

Also worth noting on the other cluster

ser8 has no infrastructure or bridge-deploy Kustomization at all. Its four, coredns, flux-system, forgejo-runners-canary and forgejo-runners-general, are all suspend=false and Ready=True on the same revision. ser8's GitOps is healthy and kai-server's is half-inert, which is a per-cluster split this issue does not currently mention.

## Confirmed suspended, and the status field says something worse than "suspended". Verified 2026-08-29 ~05:15Z. `#981` recorded these two as still suspended by reading `kubectl get kustomization -A`. That holds, and reading the objects rather than the table adds a fact the table hides. ``` NAME SUSPEND READY LAST APPLIED REVISION apps false True main@sha1:837cb0c2 coredns false True main@sha1:837cb0c2 flux-system false True main@sha1:837cb0c2 bridge-deploy TRUE - none infrastructure TRUE - none ``` **`lastAppliedRevision: none` on both.** Not a stale revision, not a revision from before the suspension. **None.** These two Kustomizations have never successfully applied anything. Ages are 72 days for `infrastructure` and 49 days for `bridge-deploy`. ### Why that changes the issue "Suspended" implies a working reconciliation that someone paused, where resuming returns you to a known state. **That is not what these are.** They were created, never reconciled once, and have sat inert for 72 and 49 days respectively. So the risk on resuming is not "Flux catches up on a backlog of drift". It is that **the first successful apply these objects ever perform will happen against a live cluster whose entire current state was established by other means.** Nobody has seen what that apply does, because it has never run. Everything in the `infrastructure` and `bridge-deploy` trees is therefore reaching the cluster by hand or by CD scripts rather than by GitOps. That is the standing explanation for the hand-applied drift this repository keeps finding, `#910`'s dead `service.loadBalancerIP` key and hand-applied Service drift being the example that already closed. ### What this issue should now ask for Not "unsuspend". The safe sequence is: 1. **Diff before apply.** `flux diff kustomization` or an equivalent dry run against the live cluster, for each of the two, so the first-ever apply is read before it is taken. 2. Reconcile the diff deliberately, deciding per-resource whether the tracked definition or the live object is correct. **On a 72-day-old tree with no reconciliation history, expect the tracked side to be wrong in places**, not just the cluster. 3. Only then unsuspend, and watch the first real apply. `aosguard ops kubectl` exposes `diff`, so step 1 is reachable from this seat. **I did not run it**, because a meaningful diff wants an attended operator watching the output against a cluster serving live traffic, and this is a backlog pass rather than a change window. ### Also worth noting on the other cluster ser8 has no `infrastructure` or `bridge-deploy` Kustomization at all. Its four, `coredns`, `flux-system`, `forgejo-runners-canary` and `forgejo-runners-general`, are all `suspend=false` and `Ready=True` on the same revision. **ser8's GitOps is healthy and kai-server's is half-inert**, which is a per-cluster split this issue does not currently mention.

coilyco-ops added this to the CD and rollout milestone

2026-08-29 06:10:08 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#837

No description provided.

Rows
Columns