coilyco-flight-deck/infrastructure

Watch

fix: make the Forgejo runner recycle CronJob executable #587

New issue

Closed

opened 2026-07-23 04:35:41 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-23 04:35:41 +00:00

Member

Problem

The daily Forgejo runner recycle CronJob uses rancher/kubectl:v1.31.4 with a /bin/sh -c command, but that image does not contain /bin/sh. Every observed Job terminates with StartError before the Job restarts any runner StatefulSet.

The failed recycle path leaves general-runner scratch on the node until eviction or another restart removes it. During the 2026-07-22 DiskPressure incident, free rootfs fell from about 75 GiB to 62 GiB after the four general runners recovered.

Scope

Select an image and entrypoint pairing that can execute the restart logic.
Verify that the Job targets the intended general and image-build runner StatefulSets.
Add validation that proves the container command can start.
Confirm the recycle actually clears the intended ephemeral scratch.

Acceptance

A manually created Job and the next scheduled Job complete successfully.
The intended runner StatefulSets roll and return Ready.
The Job leaves no recurring StartError pods.
The runbook documents what state the recycle clears.

Related to #560.

## Problem The daily Forgejo runner recycle CronJob uses `rancher/kubectl:v1.31.4` with a `/bin/sh -c` command, but that image does not contain `/bin/sh`. Every observed Job terminates with `StartError` before the Job restarts any runner StatefulSet. The failed recycle path leaves general-runner scratch on the node until eviction or another restart removes it. During the 2026-07-22 DiskPressure incident, free rootfs fell from about 75 GiB to 62 GiB after the four general runners recovered. ## Scope * Select an image and entrypoint pairing that can execute the restart logic. * Verify that the Job targets the intended general and image-build runner StatefulSets. * Add validation that proves the container command can start. * Confirm the recycle actually clears the intended ephemeral scratch. ## Acceptance * A manually created Job and the next scheduled Job complete successfully. * The intended runner StatefulSets roll and return Ready. * The Job leaves no recurring `StartError` pods. * The runbook documents what state the recycle clears. Related to #560.

coilyco-ops referenced this issue

2026-07-23 04:36:22 +00:00

kai-server unhealthy: forgejo bounce, runner crashloop, missing /github/pat, signoz/metrics 500s #308

coilyco-ops added the

priority

label

2026-07-23 07:16:48 +00:00

coilyco-ops referenced this issue

2026-07-24 04:18:58 +00:00

Make disposable Forgejo Actions runners lower priority #615

coilyco-ops referenced this issue

2026-07-24 04:18:58 +00:00

Alert when the Forgejo runner recycle stops completing #617

coilyco-ops added the

autonomy

headless

label

2026-07-29 03:46:21 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-07-29 06:27:52 +00:00

coilyco-ops referenced this issue from a commit

2026-07-29 12:48:55 +00:00

fix(runners): make scoped recycle executable

coilyco-ops closed this issue

2026-07-29 12:50:08 +00:00

coilyco-ops referenced this issue

2026-07-29 12:56:47 +00:00

interactive: verify scoped Forgejo runner recycle #689

coilyco-ops commented

2026-07-29 12:57:57 +00:00

Author

Member

WARD-WORKFLOW: done ✅

details

workflow: merge-remote-main; review summary: skipped

The in-container review gate was intentionally skipped because implementation and QA are role-separated. QA is a separate, opt-in exact-commit verification role bound to 59acdd65ee8cb22e9b801ed4a0f6b4d99069b8f2.

Landed d1216d5 through main; lint and trufflehog passed. Felt: crisp once current scoped topology replaced stale global assumptions. Confidence: high. Surprise: the earlier shell-free patch was neither Flux-owned nor current-targeted. Follow-up: interactive #689 owns manual and next-schedule live evidence.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> workflow: merge-remote-main; review summary: skipped The in-container review gate was intentionally skipped because implementation and QA are role-separated. QA is a separate, opt-in exact-commit verification role bound to `59acdd65ee8cb22e9b801ed4a0f6b4d99069b8f2`. Landed `d1216d5` through `main`; lint and trufflehog passed. Felt: crisp once current scoped topology replaced stale global assumptions. Confidence: high. Surprise: the earlier shell-free patch was neither Flux-owned nor current-targeted. Follow-up: interactive #689 owns manual and next-schedule live evidence. </details>

coilyco-ops commented

2026-07-29 12:58:10 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-infrastructure-587 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-infrastructure-587` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#587

No description provided.

Rows
Columns