coilyco-flight-deck/infrastructure

Watch

forgejo-runner-deploy-atlas has crash-looped for 14h against a repository that does not exist #834

New issue

Closed

opened 2026-08-15 07:46:05 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 07:46:05 +00:00

Owner

Filed by Olaf (ops, claude seat). Found during a fleet-wide health sweep while working in deploy. Not caused by that work, and it predates it by hours.

Symptom

forgejo/forgejo-runner-deploy-atlas-0 on kai-server:

1/2   CrashLoopBackOff   179 restarts   14h

It is the only failing runner of sixteen. Every sibling is 2/2 Running with 0 restarts at the same 15h age, so they all came up together on one convergence and atlas alone did not.

Cause

The runner container never registers:

time="2026-08-15T07:43:34Z" level=info  msg="Starting runner daemon"
time="2026-08-15T07:43:34Z" level=error msg="fail to invoke Declare" error="unauthenticated: unregistered runner"
Error: unauthenticated: unregistered runner

deploy/forgejo-runners-scoped/publisher-atlas-repository/kustomization.yaml scopes it to one repository and pulls its registration token from a matching SSM path:

coilyco.io/runner-scope: repo/coilyco-bridge/atlas
/forgejo/runner-registration/repo/coilyco-bridge/atlas

That repository does not exist. GET /repos/{owner}/atlas returns 404 for coilyco-bridge, coilyco-flight-deck, coilyco-gaming, coilysiren and coilyco-ops, and a repo search for atlas returns an empty set.

So this is not an expired token to rotate. It is a runner scoped to a repository that is gone, and a repo-scoped registration cannot succeed without the repo.

Worth noting a repo-atlas skill still exists describing atlas as a static architecture site, so the project is not obviously abandoned. The repository is just not on this Forgejo under any owner I can see.

Which is it

Three possibilities and they need different fixes, which is why this is filed rather than guessed at:

The repository was deleted or never created. Remove the atlas overlay from forgejo-runners-scoped, and delete the orphaned StatefulSet and its ExternalSecret. The SSM parameter goes too.
It was renamed. Update the scope annotation and the SSM path to the new name, and re-register.
It is private and invisible to the read token I hold. Then the registration token is simply stale and wants rotating. I cannot distinguish this from case 1 with the access I have, which is worth someone checking before anything is deleted.

Impact

Low but not zero. Any atlas deploy job queues with no runner to claim it, silently. The churn itself is minor: one container backing off, pulling kai-registry.local/forgejo-runner-deploy:latest roughly every five minutes for fourteen hours.

The part worth reacting to is that nothing noticed. A runner died at the same convergence that brought fifteen others up cleanly, and it took a manual sweep to find it. Whatever answers case 1 through 3, a crash-looping runner is probably worth an alert.

What I did not do

Nothing. I did not scale it to zero, because the StatefulSet is kustomize-managed from this repo and scaling would drift from declared state and be undone at the next convergence. aosguard ops kubectl has no delete verb either, so the cleanup in case 1 needs an operator regardless.

**Filed by Olaf (ops, claude seat).** Found during a fleet-wide health sweep while working in deploy. Not caused by that work, and it predates it by hours. ## Symptom `forgejo/forgejo-runner-deploy-atlas-0` on kai-server: ``` 1/2 CrashLoopBackOff 179 restarts 14h ``` It is the **only** failing runner of sixteen. Every sibling is `2/2 Running` with `0` restarts at the same 15h age, so they all came up together on one convergence and atlas alone did not. ## Cause The runner container never registers: ``` time="2026-08-15T07:43:34Z" level=info msg="Starting runner daemon" time="2026-08-15T07:43:34Z" level=error msg="fail to invoke Declare" error="unauthenticated: unregistered runner" Error: unauthenticated: unregistered runner ``` `deploy/forgejo-runners-scoped/publisher-atlas-repository/kustomization.yaml` scopes it to one repository and pulls its registration token from a matching SSM path: ```yaml coilyco.io/runner-scope: repo/coilyco-bridge/atlas /forgejo/runner-registration/repo/coilyco-bridge/atlas ``` **That repository does not exist.** `GET /repos/{owner}/atlas` returns 404 for `coilyco-bridge`, `coilyco-flight-deck`, `coilyco-gaming`, `coilysiren` and `coilyco-ops`, and a repo search for `atlas` returns an empty set. So this is not an expired token to rotate. It is a runner scoped to a repository that is gone, and a repo-scoped registration cannot succeed without the repo. Worth noting a `repo-atlas` skill still exists describing atlas as a static architecture site, so the project is not obviously abandoned. The repository is just not on this Forgejo under any owner I can see. ## Which is it Three possibilities and they need different fixes, which is why this is filed rather than guessed at: 1. **The repository was deleted or never created.** Remove the atlas overlay from `forgejo-runners-scoped`, and delete the orphaned StatefulSet and its ExternalSecret. The SSM parameter goes too. 2. **It was renamed.** Update the scope annotation and the SSM path to the new name, and re-register. 3. **It is private and invisible to the read token I hold.** Then the registration token is simply stale and wants rotating. I cannot distinguish this from case 1 with the access I have, which is worth someone checking before anything is deleted. ## Impact Low but not zero. Any atlas deploy job queues with no runner to claim it, silently. The churn itself is minor: one container backing off, pulling `kai-registry.local/forgejo-runner-deploy:latest` roughly every five minutes for fourteen hours. The part worth reacting to is that **nothing noticed**. A runner died at the same convergence that brought fifteen others up cleanly, and it took a manual sweep to find it. Whatever answers case 1 through 3, a crash-looping runner is probably worth an alert. ## What I did not do Nothing. I did not scale it to zero, because the StatefulSet is kustomize-managed from this repo and scaling would drift from declared state and be undone at the next convergence. `aosguard ops kubectl` has no `delete` verb either, so the cleanup in case 1 needs an operator regardless.

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:11 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:01 +00:00

coilyco-ops commented

2026-08-28 21:12:35 +00:00

Author

Owner

Resolved. Closing.

The StatefulSet no longer exists. Verified 2026-08-28:

$ kubectl get statefulset,deployment -n forgejo --context kai-server
(no forgejo-runner-deploy-atlas)

Nothing named atlas remains in the forgejo namespace on kai-server.

This was also the cause of #837's sibling

Worth recording because the two issues were filed separately without the link. #836 reported that one dead disposable runner was holding the entire apps Flux Kustomization at Ready=False, and its evidence block names the culprit:

health check failed: failed early due to stalled resources:
[StatefulSet/forgejo/forgejo-runner-deploy-atlas status: 'Failed']

So this issue and #836 were one fault seen from two angles. apps is now READY=True, which is a direct consequence of this runner being gone. Both closed together.

Not established

How it was removed, or by whom. It may have been deleted deliberately or swept by runner churn. If runners can be created pointing at repositories that do not exist, the filing condition can recur, and #955 (reconcile runner scopes for active deploy workflows) is the closest open issue to that gap.

## Resolved. Closing. The StatefulSet no longer exists. Verified 2026-08-28: ``` $ kubectl get statefulset,deployment -n forgejo --context kai-server (no forgejo-runner-deploy-atlas) ``` Nothing named `atlas` remains in the `forgejo` namespace on kai-server. ### This was also the cause of #837's sibling Worth recording because the two issues were filed separately without the link. #836 reported that one dead disposable runner was holding the entire `apps` Flux Kustomization at `Ready=False`, and its evidence block names the culprit: ``` health check failed: failed early due to stalled resources: [StatefulSet/forgejo/forgejo-runner-deploy-atlas status: 'Failed'] ``` So this issue and #836 were one fault seen from two angles. `apps` is now `READY=True`, which is a direct consequence of this runner being gone. Both closed together. ### Not established How it was removed, or by whom. It may have been deleted deliberately or swept by runner churn. If runners can be created pointing at repositories that do not exist, the filing condition can recur, and #955 (reconcile runner scopes for active deploy workflows) is the closest open issue to that gap.

coilyco-ops

2026-08-28 21:12:45 +00:00