Flux apps Kustomization: one dead disposable runner fails the health gate for the entire app layer #836

New issue

Closed

opened 2026-08-15 16:51:58 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 16:51:58 +00:00

Owner

Summary

A single crashlooping runner StatefulSet holds the whole apps Flux Kustomization at Ready=False. Disposable CI runners are the most failure-prone objects in that inventory, and gating the entire app layer on their health means any one of them going bad silently degrades GitOps reporting for everything else.

Evidence

flux-system  apps  58d  False  health check failed after 11.84506868s: failed early due to
stalled resources: [StatefulSet/forgejo/forgejo-runner-deploy-atlas status: 'Failed']

Conditions on kustomization/apps in flux-system:

Ready=False, reason HealthCheckFailed
Healthy=False, reason HealthCheckFailed
Reconciling=True, reason ProgressingWithRetry
observedGeneration: 8

spec.healthChecks names six runner StatefulSets explicitly (forgejo-runner-flight-deck, forgejo-runner-bridge, forgejo-runner-gaming, forgejo-runner-build-flight-deck, forgejo-runner-deploy-scoped, forgejo-runner-tap-writer-scoped). forgejo-runner-deploy-atlas is not in that list, so the failure is coming from whole-inventory health checking rather than the named set.

Not a resource problem

Worth recording so nobody chases it: the retry loop is cheap. interval: 10m, and the controllers are idle.

helm-controller           1m CPU   16Mi
kustomize-controller      1m CPU   35Mi
notification-controller   1m CPU   20Mi
source-controller         2m CPU   31Mi

The cost is lost signal, not lost capacity. A permanently red apps Kustomization means a genuinely broken app deploy would not stand out.

Options to consider

Narrow health checking to the explicitly named spec.healthChecks set instead of the full inventory
Move disposable runners into their own Kustomization so their health is reported separately from real applications
Keep runner pools out of health gating entirely and alert on runner readiness through a dedicated check

Acceptance

A single crashlooping disposable runner no longer flips the app-layer Kustomization to Ready=False, and runner health is still visible somewhere.

## Summary A single crashlooping runner StatefulSet holds the whole `apps` Flux Kustomization at `Ready=False`. Disposable CI runners are the most failure-prone objects in that inventory, and gating the entire app layer on their health means any one of them going bad silently degrades GitOps reporting for everything else. ## Evidence ``` flux-system apps 58d False health check failed after 11.84506868s: failed early due to stalled resources: [StatefulSet/forgejo/forgejo-runner-deploy-atlas status: 'Failed'] ``` Conditions on `kustomization/apps` in `flux-system`: - `Ready=False`, reason `HealthCheckFailed` - `Healthy=False`, reason `HealthCheckFailed` - `Reconciling=True`, reason `ProgressingWithRetry` - `observedGeneration: 8` `spec.healthChecks` names six runner StatefulSets explicitly (`forgejo-runner-flight-deck`, `forgejo-runner-bridge`, `forgejo-runner-gaming`, `forgejo-runner-build-flight-deck`, `forgejo-runner-deploy-scoped`, `forgejo-runner-tap-writer-scoped`). `forgejo-runner-deploy-atlas` is not in that list, so the failure is coming from whole-inventory health checking rather than the named set. ## Not a resource problem Worth recording so nobody chases it: the retry loop is cheap. `interval: 10m`, and the controllers are idle. ``` helm-controller 1m CPU 16Mi kustomize-controller 1m CPU 35Mi notification-controller 1m CPU 20Mi source-controller 2m CPU 31Mi ``` The cost is lost signal, not lost capacity. A permanently red `apps` Kustomization means a genuinely broken app deploy would not stand out. ## Options to consider - Narrow health checking to the explicitly named `spec.healthChecks` set instead of the full inventory - Move disposable runners into their own Kustomization so their health is reported separately from real applications - Keep runner pools out of health gating entirely and alert on runner readiness through a dedicated check ## Acceptance A single crashlooping disposable runner no longer flips the app-layer Kustomization to `Ready=False`, and runner health is still visible somewhere.

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:10 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:00 +00:00

coilyco-ops commented

2026-08-28 21:12:13 +00:00

Author

Owner

Resolved. Closing.

Verified against live state 2026-08-28:

$ kubectl get kustomization -A --context kai-server
NAMESPACE     NAME     AGE   READY   STATUS
flux-system   apps     71d   True    Applied revision: main@sha1:837cb0c23ac1f7faf4f30174403fee3f9dcef180

The apps Kustomization is READY=True and applying cleanly, so the dead disposable runner no longer fails the health gate for the app layer.

Not verified: which specific change cleared it. The runner in question is gone rather than fixed, so this may have resolved as a side effect of runner churn rather than deliberately. If the same shape recurs, the underlying complaint stands, that one disposable pod can gate an entire layer's health, and that is worth its own issue rather than reopening this one.

Related and still open: #837, where infrastructure and bridge-deploy remain suspended.

## Resolved. Closing. Verified against live state 2026-08-28: ``` $ kubectl get kustomization -A --context kai-server NAMESPACE NAME AGE READY STATUS flux-system apps 71d True Applied revision: main@sha1:837cb0c23ac1f7faf4f30174403fee3f9dcef180 ``` The `apps` Kustomization is `READY=True` and applying cleanly, so the dead disposable runner no longer fails the health gate for the app layer. Not verified: which specific change cleared it. The runner in question is gone rather than fixed, so this may have resolved as a side effect of runner churn rather than deliberately. If the same shape recurs, the underlying complaint stands, that one disposable pod can gate an entire layer's health, and that is worth its own issue rather than reopening this one. Related and still open: #837, where `infrastructure` and `bridge-deploy` remain suspended.

coilyco-ops

2026-08-28 21:12:18 +00:00