Autoscale scoped Forgejo runners to a four-pod idle floor #682

New issue

Open

opened 2026-07-29 00:24:49 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-29 00:24:49 +00:00

Member

Outcome

Replace the post-lockdown fleet of permanently idle scoped runner pods with scope-aware ephemeral execution while preserving every organization and repository registration boundary.

Target topology

Normal runners: three warm capacity-one pods, one per approved organization scope. Each scope scales from one to two, for three to six normal pods total.
Image builder: exactly one larger pod with capacity four and the existing persistent BuildKit cache. The builder remains warm and organization-scoped to coilyco-flight-deck.
Deploy and writer runners: zero to two capacity-one pods total. The shared execution template serves service deploys, exact-repository image publishers, the Homebrew tap writer, and deploy-owned SigNoz Terraform. Each job still receives its exact registration scope and only its own credentials.
Controller: one small always-on autoscaling controller outside the runner count.

The resulting runner-pod envelope is four at idle and nine at maximum. Including the controller, the physical pod envelope is approximately five to ten.

Design constraints

Use Forgejo 15 ephemeral one-job registrations with an external scope-aware Kubernetes controller. Ephemeral mode alone is not an autoscaler.
Keep all organization and exact-repository registration scopes introduced by the external-account lockdown. No global runner may return.
Keep normal pods at capacity one.
Keep the single builder at capacity four. Preserve its cache ownership and larger CPU, memory, storage, and timeout envelope.
Apply a global maximum of two deploy and writer pods, with at most one active pod for each exact repository scope. Additional work queues.
Select the registration token, ServiceAccount, network access, and credential projection from the queued job scope. Do not place the union of deploy, tap, registry, SigNoz, or S3 credentials in a shared pod environment.
Add Terraform and jq to the deploy execution image. SigNoz Terraform reads its service-account token from SSM-backed runtime projection and uses an AWS identity limited to the exact remote state object and lockfile.
Keep rollout and rollback observable through runner status, Kubernetes state, workflow evidence, and resource telemetry.

Acceptance

Repository validation renders every scope and proves the minimum and maximum policy without duplicating configuration in tests.
Idle steady state contains three normal runner pods, one builder pod, zero deploy or writer pods, and one controller pod.
A normal queue scales only its owning organization from one to two and returns to one after the idle window.
The builder runs the AOS four-way image fan-out on its single capacity-four pod and retains its persistent cache.
Deploy, publisher, tap, and SigNoz Terraform jobs each create an exact-scope ephemeral pod, complete once, unregister, and return the deploy pool toward zero.
More than two simultaneous deploy or writer jobs queue rather than exceeding the global maximum.
An unrelated repository cannot see or use another scope, label, ServiceAccount, or credential.
Controller restart, node restart, failed jobs, and abandoned jobs converge without orphaned registrations or pods.
Documentation records scaling triggers, idle windows, resource envelopes, credentials, failure behavior, rollback, and live verification.
docs/FEATURES.md reflects the new runner architecture.

Dependencies and history

#675 restores the exact-repository publisher roster after the scoped-runner migration and must be healthy before this consolidation rolls out.
#658 and #665 established the registration-scope security boundary that this work must preserve.

## Outcome Replace the post-lockdown fleet of permanently idle scoped runner pods with scope-aware ephemeral execution while preserving every organization and repository registration boundary. ## Target topology * Normal runners: three warm capacity-one pods, one per approved organization scope. Each scope scales from one to two, for three to six normal pods total. * Image builder: exactly one larger pod with capacity four and the existing persistent BuildKit cache. The builder remains warm and organization-scoped to coilyco-flight-deck. * Deploy and writer runners: zero to two capacity-one pods total. The shared execution template serves service deploys, exact-repository image publishers, the Homebrew tap writer, and deploy-owned SigNoz Terraform. Each job still receives its exact registration scope and only its own credentials. * Controller: one small always-on autoscaling controller outside the runner count. The resulting runner-pod envelope is four at idle and nine at maximum. Including the controller, the physical pod envelope is approximately five to ten. ## Design constraints * Use Forgejo 15 ephemeral one-job registrations with an external scope-aware Kubernetes controller. Ephemeral mode alone is not an autoscaler. * Keep all organization and exact-repository registration scopes introduced by the external-account lockdown. No global runner may return. * Keep normal pods at capacity one. * Keep the single builder at capacity four. Preserve its cache ownership and larger CPU, memory, storage, and timeout envelope. * Apply a global maximum of two deploy and writer pods, with at most one active pod for each exact repository scope. Additional work queues. * Select the registration token, ServiceAccount, network access, and credential projection from the queued job scope. Do not place the union of deploy, tap, registry, SigNoz, or S3 credentials in a shared pod environment. * Add Terraform and jq to the deploy execution image. SigNoz Terraform reads its service-account token from SSM-backed runtime projection and uses an AWS identity limited to the exact remote state object and lockfile. * Keep rollout and rollback observable through runner status, Kubernetes state, workflow evidence, and resource telemetry. ## Acceptance * Repository validation renders every scope and proves the minimum and maximum policy without duplicating configuration in tests. * Idle steady state contains three normal runner pods, one builder pod, zero deploy or writer pods, and one controller pod. * A normal queue scales only its owning organization from one to two and returns to one after the idle window. * The builder runs the AOS four-way image fan-out on its single capacity-four pod and retains its persistent cache. * Deploy, publisher, tap, and SigNoz Terraform jobs each create an exact-scope ephemeral pod, complete once, unregister, and return the deploy pool toward zero. * More than two simultaneous deploy or writer jobs queue rather than exceeding the global maximum. * An unrelated repository cannot see or use another scope, label, ServiceAccount, or credential. * Controller restart, node restart, failed jobs, and abandoned jobs converge without orphaned registrations or pods. * Documentation records scaling triggers, idle windows, resource envelopes, credentials, failure behavior, rollback, and live verification. * docs/FEATURES.md reflects the new runner architecture. ## Dependencies and history * #675 restores the exact-repository publisher roster after the scoped-runner migration and must be healthy before this consolidation rolls out. * #658 and #665 established the registration-scope security boundary that this work must preserve.

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-29 03:46:38 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#682

No description provided.

Rows
Columns