Orphaned atlas publisher runner: repo deleted, StatefulSet crashlooping 283 times, fails Flux apps health gate #835

New issue

Closed

opened 2026-08-15 16:51:46 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 16:51:46 +00:00

Owner

Summary

coilyco-bridge/atlas no longer exists in Forgejo, but its scoped publisher runner is still declared and still deployed. The runner has been crashlooping for over 24 hours and its Failed StatefulSet status is what holds the Flux apps Kustomization at Ready=False.

Evidence

Repository is gone:

GET https://forgejo.coilysiren.me/api/v1/repos/coilyco-bridge/atlas -> 404 Not Found

Runner state on kai-server at 2026-08-15T16:34Z:

forgejo/forgejo-runner-deploy-atlas-0, phase Running, runner container terminated and not ready, restart_count: 283, pod age 23h43m
The dind sidecar is healthy. Only the runner container fails.

Container logs repeat every ~5 minutes across restart generations 276 through 283:

time="2026-08-15T16:34:59Z" level=info msg="Starting runner daemon"
time="2026-08-15T16:34:59Z" level=error msg="fail to invoke Declare" error="unauthenticated: unregistered runner"
Error: unauthenticated: unregistered runner

Flux impact:

flux-system  apps  58d  False  health check failed after 11.84506868s: failed early due to
stalled resources: [StatefulSet/forgejo/forgejo-runner-deploy-atlas status: 'Failed']

Why it happens

The register init container skips registration when /data/.runner exists and is younger than RUNNER_CONFIG_TTL_DAYS (30). The state file is still fresh, so the pod never re-registers, and the daemon then fails Declare against an instance that has no such repo-scoped runner.

Declarations to remove

deploy/forgejo-runners-scoped/publisher-atlas-repository/ (whole directory)
The publisher-atlas-repository entry in deploy/forgejo-runners-scoped/kustomization.yaml
SSM parameter /forgejo/runner-registration/repo/coilyco-bridge/atlas

Live cleanup is required separately

The apps Kustomization sets prune: false, so removing the overlay from git will not delete the running objects. These need explicit deletion:

statefulset/forgejo-runner-deploy-atlas in namespace forgejo
its data-forgejo-runner-deploy-atlas-0 PVC
externalsecret/forgejo-runner-deploy-secrets-atlas and the forgejo-runner-deploy-secrets-atlas Secret

Acceptance

No forgejo-runner-deploy-atlas objects remain in the cluster
kubectl get kustomization apps -n flux-system reports Ready=True

Related: the health-gate fragility this exposed is filed separately.

## Summary `coilyco-bridge/atlas` no longer exists in Forgejo, but its scoped publisher runner is still declared and still deployed. The runner has been crashlooping for over 24 hours and its `Failed` StatefulSet status is what holds the Flux `apps` Kustomization at `Ready=False`. ## Evidence Repository is gone: ``` GET https://forgejo.coilysiren.me/api/v1/repos/coilyco-bridge/atlas -> 404 Not Found ``` Runner state on kai-server at 2026-08-15T16:34Z: - `forgejo/forgejo-runner-deploy-atlas-0`, phase Running, `runner` container terminated and not ready, `restart_count: 283`, pod age 23h43m - The `dind` sidecar is healthy. Only the `runner` container fails. Container logs repeat every ~5 minutes across restart generations 276 through 283: ``` time="2026-08-15T16:34:59Z" level=info msg="Starting runner daemon" time="2026-08-15T16:34:59Z" level=error msg="fail to invoke Declare" error="unauthenticated: unregistered runner" Error: unauthenticated: unregistered runner ``` Flux impact: ``` flux-system apps 58d False health check failed after 11.84506868s: failed early due to stalled resources: [StatefulSet/forgejo/forgejo-runner-deploy-atlas status: 'Failed'] ``` ## Why it happens The register init container skips registration when `/data/.runner` exists and is younger than `RUNNER_CONFIG_TTL_DAYS` (30). The state file is still fresh, so the pod never re-registers, and the daemon then fails `Declare` against an instance that has no such repo-scoped runner. ## Declarations to remove - `deploy/forgejo-runners-scoped/publisher-atlas-repository/` (whole directory) - The `publisher-atlas-repository` entry in `deploy/forgejo-runners-scoped/kustomization.yaml` - SSM parameter `/forgejo/runner-registration/repo/coilyco-bridge/atlas` ## Live cleanup is required separately The `apps` Kustomization sets `prune: false`, so removing the overlay from git will not delete the running objects. These need explicit deletion: - `statefulset/forgejo-runner-deploy-atlas` in namespace `forgejo` - its `data-forgejo-runner-deploy-atlas-0` PVC - `externalsecret/forgejo-runner-deploy-secrets-atlas` and the `forgejo-runner-deploy-secrets-atlas` Secret ## Acceptance - No `forgejo-runner-deploy-atlas` objects remain in the cluster - `kubectl get kustomization apps -n flux-system` reports `Ready=True` Related: the health-gate fragility this exposed is filed separately.

coilyco-ops referenced this issue from a commit

2026-08-15 17:00:01 +00:00

chore(runners): drop the orphaned atlas publisher runner

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-15 17:00:39 +00:00

chore(runners): drop the orphaned atlas publisher runner #843