Set measured ephemeral-storage budgets for Forgejo and PostgreSQL #616

New issue

Closed

opened 2026-07-24 04:18:58 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-24 04:18:58 +00:00

Member

Context

During the 2026-07-24 DiskPressure incident, kubelet evicted the Forgejo app after reporting roughly 15 MiB of ephemeral usage against a zero request. PostgreSQL was also evicted with no ephemeral-storage request. Their durable local-path PVCs contribute to node filesystem pressure, but PVC capacity is separate from pod writable-layer, log, and emptyDir accounting.

Kubelet considers whether a pod exceeds its resource request before pod priority when ranking node-pressure eviction candidates. Forgejo and PostgreSQL therefore need measured ephemeral-storage requests and limits rather than implicit zero requests.

Scope

Measure steady-state and realistic peak writable-layer and container-log usage for the Forgejo app and PostgreSQL.
Separate pod ephemeral usage from the Forgejo and database PVC ownership domains.
Add evidence-based ephemeral-storage requests and limits to both containers with enough operational headroom for startup, indexing, maintenance, and ordinary log bursts.
Confirm the limits cannot turn normal behavior into self-eviction.
Add validation that prevents either stateful service from returning to a zero ephemeral-storage request.
Document the measurement and adjustment procedure in the disk-pressure runbook.

Acceptance

deploy/forgejo.yml carries measured ephemeral-storage requests and limits for both Forgejo and PostgreSQL.
The chosen budgets cite live measurements and explain their headroom.
Offline manifest tests reject missing or zero requests.
A live rollout returns both workloads Ready without probe regressions.
Node-pressure documentation clearly distinguishes PVC growth from pod ephemeral usage.

Related: #596 and #600.

## Context During the 2026-07-24 DiskPressure incident, kubelet evicted the Forgejo app after reporting roughly 15 MiB of ephemeral usage against a zero request. PostgreSQL was also evicted with no ephemeral-storage request. Their durable local-path PVCs contribute to node filesystem pressure, but PVC capacity is separate from pod writable-layer, log, and `emptyDir` accounting. Kubelet considers whether a pod exceeds its resource request before pod priority when ranking node-pressure eviction candidates. Forgejo and PostgreSQL therefore need measured ephemeral-storage requests and limits rather than implicit zero requests. ## Scope * Measure steady-state and realistic peak writable-layer and container-log usage for the Forgejo app and PostgreSQL. * Separate pod ephemeral usage from the Forgejo and database PVC ownership domains. * Add evidence-based `ephemeral-storage` requests and limits to both containers with enough operational headroom for startup, indexing, maintenance, and ordinary log bursts. * Confirm the limits cannot turn normal behavior into self-eviction. * Add validation that prevents either stateful service from returning to a zero ephemeral-storage request. * Document the measurement and adjustment procedure in the disk-pressure runbook. ## Acceptance * `deploy/forgejo.yml` carries measured ephemeral-storage requests and limits for both Forgejo and PostgreSQL. * The chosen budgets cite live measurements and explain their headroom. * Offline manifest tests reject missing or zero requests. * A live rollout returns both workloads Ready without probe regressions. * Node-pressure documentation clearly distinguishes PVC growth from pod ephemeral usage. Related: #596 and #600.

coilyco-ops referenced this issue from a commit

2026-07-24 04:33:19 +00:00

fix: budget Forgejo ephemeral storage

coilyco-ops closed this issue

2026-07-24 04:33:19 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#616

No description provided.

Rows
Columns