Roll out scoped Forgejo runner autoscaling #956

New issue

Open

opened 2026-08-27 03:24:59 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-27 03:24:59 +00:00

Owner

Configuration dependency

#682

Outcome

Perform the attended migration from permanently warm scoped Forgejo runner pods to the approved autoscaled topology, verify every security scope and workload class live, and preserve a tested rollback until the soak completes.

Target live envelope

Normal: three to six capacity-one runner pods, one to two for each approved organization scope.
Image builder: one larger capacity-four pod with its persistent BuildKit cache.
Deploy and writer: zero to two capacity-one ephemeral pods shared across service deploys, exact-repository publishers, tap writes, and deploy-owned SigNoz Terraform while preserving exact repository registration and credential scopes.
Controller: one always-on pod outside the runner count.
Runner total: four idle and nine maximum. Physical total including the controller: approximately five idle and ten maximum.

Preconditions

Infrastructure issue #682 is landed and repository validation passes.
Infrastructure issue #675 has restored and verified every exact-repository publisher scope.
Required registration tokens and workload credentials exist in SSM-backed projections without exposing values.
The operator has rendered rollback manifests that restore the current scoped StatefulSets and replica counts.
The operator has captured current runner registrations, pod inventory, queued work, resource usage, and successful workflow examples.

Attended rollout

Deploy the controller and ephemeral runner templates while the current scoped runners remain available.
Prove normal execution independently in coilyco-flight-deck, coilyco-bridge, and coilyco-gaming, including scale-up and return to one warm pod per organization.
Prove the single capacity-four builder with the AOS four-way image fan-out and confirm cache reuse remains healthy.
Prove exact-scope deploy and writer execution for the deploy repo, one existing publisher, the Homebrew tap, and the SigNoz Terraform preflight and plan path. Do not broaden credentials between those jobs.
Prove a third simultaneous deploy or writer job queues while the global maximum remains two.
Scale the replaced persistent scoped runners to zero in bounded groups. Observe queued-job pickup, completion, unregister behavior, resource telemetry, and orphan cleanup after each group.
Keep rollback resources and registrations available through the soak. Remove them only after the acceptance evidence is attached.

Rollback triggers

A queued job does not receive an eligible runner within the agreed startup window.
A runner appears outside its intended organization or repository scope.
A job receives another scope credentials, ServiceAccount, network authority, or persistent workspace.
A failed or canceled job leaves an active registration or pod beyond the cleanup window.
The builder loses four-way fan-out, cache integrity, or its bounded resource behavior.
The controller cannot recover cleanly after restart or node disruption.

Acceptance evidence

Idle observation shows three normal pods, one builder pod, zero deploy or writer pods, and one controller.
Burst observation never exceeds six normal pods, one builder pod, or two deploy and writer pods.
Workflow links demonstrate all three organization scopes, the builder fan-out, deploy, publisher, tap, and SigNoz Terraform paths.
Negative visibility checks show an unrelated repository cannot select protected runner scopes.
Kubernetes events, runner status, and telemetry show clean scale-up, scale-down, cancellation, and restart recovery.
The operator records the final resource delta from the pre-rollout eighteen-pod state.
Rollback is either exercised successfully or retained until the soak closes with an explicit removal checkpoint.

## Configuration dependency * https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/682 ## Outcome Perform the attended migration from permanently warm scoped Forgejo runner pods to the approved autoscaled topology, verify every security scope and workload class live, and preserve a tested rollback until the soak completes. ## Target live envelope * Normal: three to six capacity-one runner pods, one to two for each approved organization scope. * Image builder: one larger capacity-four pod with its persistent BuildKit cache. * Deploy and writer: zero to two capacity-one ephemeral pods shared across service deploys, exact-repository publishers, tap writes, and deploy-owned SigNoz Terraform while preserving exact repository registration and credential scopes. * Controller: one always-on pod outside the runner count. * Runner total: four idle and nine maximum. Physical total including the controller: approximately five idle and ten maximum. ## Preconditions * Infrastructure issue #682 is landed and repository validation passes. * Infrastructure issue #675 has restored and verified every exact-repository publisher scope. * Required registration tokens and workload credentials exist in SSM-backed projections without exposing values. * The operator has rendered rollback manifests that restore the current scoped StatefulSets and replica counts. * The operator has captured current runner registrations, pod inventory, queued work, resource usage, and successful workflow examples. ## Attended rollout 1. Deploy the controller and ephemeral runner templates while the current scoped runners remain available. 2. Prove normal execution independently in coilyco-flight-deck, coilyco-bridge, and coilyco-gaming, including scale-up and return to one warm pod per organization. 3. Prove the single capacity-four builder with the AOS four-way image fan-out and confirm cache reuse remains healthy. 4. Prove exact-scope deploy and writer execution for the deploy repo, one existing publisher, the Homebrew tap, and the SigNoz Terraform preflight and plan path. Do not broaden credentials between those jobs. 5. Prove a third simultaneous deploy or writer job queues while the global maximum remains two. 6. Scale the replaced persistent scoped runners to zero in bounded groups. Observe queued-job pickup, completion, unregister behavior, resource telemetry, and orphan cleanup after each group. 7. Keep rollback resources and registrations available through the soak. Remove them only after the acceptance evidence is attached. ## Rollback triggers * A queued job does not receive an eligible runner within the agreed startup window. * A runner appears outside its intended organization or repository scope. * A job receives another scope credentials, ServiceAccount, network authority, or persistent workspace. * A failed or canceled job leaves an active registration or pod beyond the cleanup window. * The builder loses four-way fan-out, cache integrity, or its bounded resource behavior. * The controller cannot recover cleanly after restart or node disruption. ## Acceptance evidence * Idle observation shows three normal pods, one builder pod, zero deploy or writer pods, and one controller. * Burst observation never exceeds six normal pods, one builder pod, or two deploy and writer pods. * Workflow links demonstrate all three organization scopes, the builder fan-out, deploy, publisher, tap, and SigNoz Terraform paths. * Negative visibility checks show an unrelated repository cannot select protected runner scopes. * Kubernetes events, runner status, and telemetry show clean scale-up, scale-down, cancellation, and restart recovery. * The operator records the final resource delta from the pre-rollout eighteen-pod state. * Rollback is either exercised successfully or retained until the soak closes with an explicit removal checkpoint.

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops referenced this issue

2026-08-29 04:22:59 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops added this to the Forgejo runners milestone

2026-08-29 05:40:46 +00:00