Make disposable Forgejo Actions runners lower priority #615

New issue

Closed

opened 2026-07-24 04:18:58 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-24 04:18:58 +00:00

Member

Context

The 2026-07-23 and 2026-07-24 Forgejo DiskPressure incidents showed that the Forgejo app, database, and Actions runners all run at the default pod priority of zero. The general runners own disposable DinD emptyDir scratch capped at 24 GiB per pod, while Forgejo and PostgreSQL are stateful services.

Kubelet ranks node-pressure eviction candidates by whether usage exceeds requests, pod priority, and usage relative to requests. Kai does not want a critical-service PriorityClass for Forgejo. This issue owns the inverse policy: explicitly classify disposable batch runners below ordinary services.

Scope

Define one custom, non-global PriorityClass for disposable batch workloads with a value below the default priority.
Keep the reserved system-cluster-critical and system-node-critical classes limited to Kubernetes system workloads.
Inventory the general, build, deploy, and tap-writer runner StatefulSets and document which are safe to classify as disposable.
Apply the lower class only where eviction or preemption may cancel a job without risking durable Forgejo data.
Keep Forgejo and PostgreSQL at ordinary default priority.
Document how priority interacts with ephemeral-storage requests and the node.kubernetes.io/disk-pressure scheduling taint.

Acceptance

The selected runner pod templates resolve to a priority below zero.
Forgejo and PostgreSQL remain at default priority.
No global default or application-critical PriorityClass is introduced.
Manifest validation covers the class value, preemption policy, and every intended consumer.
The disk-pressure runbook states that priority changes victim ordering but does not bypass the DiskPressure taint or replace scratch cleanup.

Related: #587 and #596.

## Context The 2026-07-23 and 2026-07-24 Forgejo DiskPressure incidents showed that the Forgejo app, database, and Actions runners all run at the default pod priority of zero. The general runners own disposable DinD `emptyDir` scratch capped at 24 GiB per pod, while Forgejo and PostgreSQL are stateful services. Kubelet ranks node-pressure eviction candidates by whether usage exceeds requests, pod priority, and usage relative to requests. Kai does not want a critical-service PriorityClass for Forgejo. This issue owns the inverse policy: explicitly classify disposable batch runners below ordinary services. ## Scope * Define one custom, non-global PriorityClass for disposable batch workloads with a value below the default priority. * Keep the reserved `system-cluster-critical` and `system-node-critical` classes limited to Kubernetes system workloads. * Inventory the general, build, deploy, and tap-writer runner StatefulSets and document which are safe to classify as disposable. * Apply the lower class only where eviction or preemption may cancel a job without risking durable Forgejo data. * Keep Forgejo and PostgreSQL at ordinary default priority. * Document how priority interacts with ephemeral-storage requests and the `node.kubernetes.io/disk-pressure` scheduling taint. ## Acceptance * The selected runner pod templates resolve to a priority below zero. * Forgejo and PostgreSQL remain at default priority. * No global default or application-critical PriorityClass is introduced. * Manifest validation covers the class value, preemption policy, and every intended consumer. * The disk-pressure runbook states that priority changes victim ordering but does not bypass the DiskPressure taint or replace scratch cleanup. Related: #587 and #596.

coilyco-ops referenced this issue from a commit

2026-07-24 04:29:24 +00:00

fix: lower disposable runner priority

coilyco-ops closed this issue

2026-07-24 04:29:24 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#615

No description provided.

Rows
Columns