Alert when the Forgejo runner recycle stops completing #617

New issue

Closed

opened 2026-07-24 04:18:58 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-24 04:18:58 +00:00

Member

Context

The Forgejo runner recycle CronJob failed with StartError for three consecutive nights because its kubectl image had no shell. No alert reported the missed recycle. General-runner DinD scratch continued accumulating until kubelet entered DiskPressure and evicted Forgejo.

Issue #587 owns making the recycle executable. A shell-free fix is now applied, and a manual verification Job completed successfully on 2026-07-24. This issue owns detecting future missed or failed recycle runs before scratch growth reaches the node-pressure threshold.

Scope

Observe the CronJob's last successful completion, current failed Jobs, and expected nightly schedule.
Alert when the scheduled recycle fails, remains incomplete past a bounded deadline, or has no recent successful run.
Send one deduplicated recovery notification after a later successful run.
Reuse the existing operations-plane alert path and credential handling.
Keep the watcher read-only. It must not restart runners, delete Jobs, prune storage, or mutate Forgejo.
Record the exact general and build runner targets that a healthy recycle rolls.
Add the check to the disk-pressure and Forgejo runner runbooks.

Acceptance

Offline tests cover success, failure, stale-success, active-within-deadline, deduplication, and recovery.
A failed synthetic fixture produces one actionable alert with the Job and CronJob names.
A later success produces one recovery and clears the alert state.
The next real scheduled recycle completes and the watcher records it as healthy.
The watcher warns before the root filesystem reaches the 85% pressure threshold.

Related: #587, #596, and #600.

## Context The Forgejo runner recycle CronJob failed with `StartError` for three consecutive nights because its kubectl image had no shell. No alert reported the missed recycle. General-runner DinD scratch continued accumulating until kubelet entered DiskPressure and evicted Forgejo. Issue #587 owns making the recycle executable. A shell-free fix is now applied, and a manual verification Job completed successfully on 2026-07-24. This issue owns detecting future missed or failed recycle runs before scratch growth reaches the node-pressure threshold. ## Scope * Observe the CronJob's last successful completion, current failed Jobs, and expected nightly schedule. * Alert when the scheduled recycle fails, remains incomplete past a bounded deadline, or has no recent successful run. * Send one deduplicated recovery notification after a later successful run. * Reuse the existing operations-plane alert path and credential handling. * Keep the watcher read-only. It must not restart runners, delete Jobs, prune storage, or mutate Forgejo. * Record the exact general and build runner targets that a healthy recycle rolls. * Add the check to the disk-pressure and Forgejo runner runbooks. ## Acceptance * Offline tests cover success, failure, stale-success, active-within-deadline, deduplication, and recovery. * A failed synthetic fixture produces one actionable alert with the Job and CronJob names. * A later success produces one recovery and clears the alert state. * The next real scheduled recycle completes and the watcher records it as healthy. * The watcher warns before the root filesystem reaches the 85% pressure threshold. Related: #587, #596, and #600.

coilyco-ops referenced this issue from a commit

2026-07-24 04:43:50 +00:00

fix: monitor Forgejo runner recycling

coilyco-ops closed this issue

2026-07-24 04:43:50 +00:00

coilyco-ops referenced this issue

2026-07-26 05:55:54 +00:00

Make Forgejo runner scratch classification and recycle busy-aware #650