Retire the untracked forgejo-runner-deploy-sirens-discord-ops runner #771

New issue

Open

opened 2026-08-09 06:26:11 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-09 06:26:11 +00:00

Owner

Summary

forgejo-runner-deploy-sirens-discord-ops is a live StatefulSet in namespace forgejo on kai-server with no manifest in any repository. It was applied once and its source was later removed without a Flux prune, so it now runs with no tracked definition, no rollback artifact, and no owner.

It is currently the only remaining scratch holder on the node.

Evidence

After the nightly-recycle fix landed (#770, merge 23d887fe) and the attended rollout restart, 13 of the 14 deploy-family runners are at zero scratch. This one is untouched because a rollout restart over tracked manifests cannot reach it:

21.73Gi  since=2026-07-31T08:42:59Z  forgejo-runner-deploy-sirens-discord-ops-0
 0.00Gi  since=2026-08-09T06:06:04Z  forgejo-runner-deploy-scoped-0
 0.00Gi  since=2026-08-09T06:06:01Z  forgejo-runner-deploy-ward-mcp-0
 0.00Gi  since=2026-08-09T06:05:59Z  forgejo-runner-deploy-sirens-echo-0
 ... 10 more at 0.00Gi

Node rootfs went from 89.65 percent critical to 74.05 percent ok on the back of that rollout. The remaining 21.73 GiB sits here.

It also will not be recycled going forward. #770 selects on coilyco.io/runner-scratch: recycle, which is declared on the tracked bases. This object predates the label and has no manifest to carry it, so the nightly job skips it by construction. That is the correct behaviour for an untracked workload, and it means the drift stays visible rather than being silently papered over.

Why it needs a decision, not a fix

Three distinct outcomes are defensible and they are not the agent's call:

Retire it. Delete the StatefulSet and its PVC. Correct if the Sirens Discord ops deploy lane is dead or has been folded into forgejo-runner-deploy-sirens-echo.
Adopt it. Add a deploy/forgejo-runners-scoped/deploy-sirens-discord-ops-repository/ overlay so it becomes tracked, labelled, and recycled like its siblings. Correct if the lane is still wanted.
Retire the runner, keep the registration. Relevant only if a Forgejo-side registration or repository secret still points at it.

Restarting or deleting a workload with no tracked manifest has no rollback artifact, so none of these should happen unattended.

Suggested checks before deciding

Whether coilyco-gaming/sirens-echo or a sibling repository still selects a sirens-discord-ops runner label in any workflow.
Whether the runner is registered and idle in the Forgejo admin runner list, or registered and still picking up jobs.
Whether a registration token or SSM parameter exists solely for it.

#770 - nightly recycle by label, which established the current coverage boundary
#769 - superseded approach, which independently reached the same exclusion conclusion

Also noted, separate work

The aggregate emptyDir ceiling is oversized: 14 deploy runners times a 24Gi sizeLimit permits 336 GiB on a 479.55 GiB filesystem. Being reduced to 10Gi separately.

## Summary `forgejo-runner-deploy-sirens-discord-ops` is a live StatefulSet in namespace `forgejo` on kai-server with **no manifest in any repository**. It was applied once and its source was later removed without a Flux prune, so it now runs with no tracked definition, no rollback artifact, and no owner. It is currently the **only** remaining scratch holder on the node. ## Evidence After the nightly-recycle fix landed (#770, merge `23d887fe`) and the attended rollout restart, 13 of the 14 deploy-family runners are at zero scratch. This one is untouched because a rollout restart over tracked manifests cannot reach it: ``` 21.73Gi since=2026-07-31T08:42:59Z forgejo-runner-deploy-sirens-discord-ops-0 0.00Gi since=2026-08-09T06:06:04Z forgejo-runner-deploy-scoped-0 0.00Gi since=2026-08-09T06:06:01Z forgejo-runner-deploy-ward-mcp-0 0.00Gi since=2026-08-09T06:05:59Z forgejo-runner-deploy-sirens-echo-0 ... 10 more at 0.00Gi ``` Node rootfs went from 89.65 percent critical to **74.05 percent ok** on the back of that rollout. The remaining 21.73 GiB sits here. It also will not be recycled going forward. #770 selects on `coilyco.io/runner-scratch: recycle`, which is declared on the tracked bases. This object predates the label and has no manifest to carry it, so the nightly job skips it by construction. That is the correct behaviour for an untracked workload, and it means the drift stays visible rather than being silently papered over. ## Why it needs a decision, not a fix Three distinct outcomes are defensible and they are not the agent's call: 1. **Retire it.** Delete the StatefulSet and its PVC. Correct if the Sirens Discord ops deploy lane is dead or has been folded into `forgejo-runner-deploy-sirens-echo`. 2. **Adopt it.** Add a `deploy/forgejo-runners-scoped/deploy-sirens-discord-ops-repository/` overlay so it becomes tracked, labelled, and recycled like its siblings. Correct if the lane is still wanted. 3. **Retire the runner, keep the registration.** Relevant only if a Forgejo-side registration or repository secret still points at it. Restarting or deleting a workload with no tracked manifest has no rollback artifact, so none of these should happen unattended. ## Suggested checks before deciding - Whether `coilyco-gaming/sirens-echo` or a sibling repository still selects a `sirens-discord-ops` runner label in any workflow. - Whether the runner is registered and idle in the Forgejo admin runner list, or registered and still picking up jobs. - Whether a registration token or SSM parameter exists solely for it. ## Related - #770 - nightly recycle by label, which established the current coverage boundary - #769 - superseded approach, which independently reached the same exclusion conclusion ## Also noted, separate work The aggregate `emptyDir` ceiling is oversized: 14 deploy runners times a 24Gi `sizeLimit` permits 336 GiB on a 479.55 GiB filesystem. Being reduced to 10Gi separately.

coilyco-ops referenced this issue from a commit

2026-08-09 06:31:50 +00:00

fix(forgejo): cap deploy and publisher scratch at 10Gi

coilyco-ops referenced this issue

2026-08-09 06:32:12 +00:00

fix(forgejo): cap deploy and publisher scratch at 10Gi #772

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:15 +00:00

coilyco-ops added the