Make Forgejo runner recycling job-aware #706

New issue

Open

opened 2026-07-31 09:22:46 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-31 09:22:46 +00:00

Member

Outcome

Prevent runner recovery or recycle operations from terminating active Forgejo Actions jobs.

Evidence

During agentic-os dev-base publication run 2595, Node, Go, and .NET publication completed and Python plus Rust began on the dedicated image-build runner.

Around 09:10Z on 2026-07-31, Kubernetes recorded a burst of normal Killing events across every flight-deck runner pod, including both general runners and the dedicated image-build runner. The StatefulSet revision did not change. There was no eviction, OOM, rollout, renewed disk pressure, or scheduled forgejo-runner-recycle job at that time.

The build-runner replacement interrupted Python and Rust. Forgejo held the orphaned tasks until run 2595 failed at 09:19:06Z.

Required live action

Identify the actor or recovery path that deleted the runner pods around 09:10Z.
Make that path check for active Forgejo jobs before deletion, drain runners first, or defer until idle.
Verify a recycle during an idle window and a no-op or defer result while a runner has an active job.
Keep the change in the infrastructure layer that owns the runner workload and fleet behavior.

Acceptance

A recovery or scheduled recycle cannot kill an active Actions job, and the operator path reports which runners were recycled versus deferred.

## Outcome Prevent runner recovery or recycle operations from terminating active Forgejo Actions jobs. ## Evidence During agentic-os dev-base publication [run 2595](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2595), Node, Go, and .NET publication completed and Python plus Rust began on the dedicated image-build runner. Around 09:10Z on 2026-07-31, Kubernetes recorded a burst of normal `Killing` events across every flight-deck runner pod, including both general runners and the dedicated image-build runner. The StatefulSet revision did not change. There was no eviction, OOM, rollout, renewed disk pressure, or scheduled `forgejo-runner-recycle` job at that time. The build-runner replacement interrupted Python and Rust. Forgejo held the orphaned tasks until run 2595 failed at 09:19:06Z. ## Required live action 1. Identify the actor or recovery path that deleted the runner pods around 09:10Z. 2. Make that path check for active Forgejo jobs before deletion, drain runners first, or defer until idle. 3. Verify a recycle during an idle window and a no-op or defer result while a runner has an active job. 4. Keep the change in the infrastructure layer that owns the runner workload and fleet behavior. ## Acceptance A recovery or scheduled recycle cannot kill an active Actions job, and the operator path reports which runners were recycled versus deferred.

coilyco-ops added the

autonomy

live-collab

label

2026-07-31 09:22:46 +00:00

coilyco-ops commented

2026-07-31 09:30:56 +00:00

Author

Member

Cause identified after filing

The runner termination was caused by the deliberate infrastructure change in #705, landed as 317bfbe86da591bf2880b5e50447978b1be17707. That change sets the scoped image-build runner to zero replicas because resuming dev-base publication raised root filesystem use back toward the disk-pressure threshold. Flux is correctly enforcing 0/0.

The immediate safety choice in #705 was correct. Keep this issue scoped to future job-awareness for non-emergency maintenance and migration work. A planned disable or recycle should drain or defer when jobs are active. An emergency disk-pressure action may terminate work, but it should report that explicitly. Capacity root cause remains #624 and runner migration remains #693.

Cause identified after filing The runner termination was caused by the deliberate infrastructure change in [#705](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/705), landed as `317bfbe86da591bf2880b5e50447978b1be17707`. That change sets the scoped image-build runner to zero replicas because resuming dev-base publication raised root filesystem use back toward the disk-pressure threshold. Flux is correctly enforcing `0/0`. The immediate safety choice in #705 was correct. Keep this issue scoped to future job-awareness for non-emergency maintenance and migration work. A planned disable or recycle should drain or defer when jobs are active. An emergency disk-pressure action may terminate work, but it should report that explicitly. Capacity root cause remains #624 and runner migration remains #693.

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-31 09:31:18 +00:00

Inspect failed dev-base gates and authorize PRs #818 and #821 #823

coilyco-ops added the

priority

label

2026-08-05 10:25:57 +00:00