Adopt Forgejo 16 run cancellation for stale-job reaping #621

New issue

Open

opened 2026-07-24 06:20:47 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-24 06:20:47 +00:00

Member

Blocked by coilyco-flight-deck/infrastructure#620.

Goal

Infrastructure adopts Forgejo 16's workflow run cancellation API and adds an out-of-band stale-job reaper for Forgejo Actions jobs that stop making progress.

Runner topology

The general pool runs one job per Docker-in-Docker runner.
The image-build pool runs up to four jobs against shared Docker-in-Docker and BuildKit services.
The deploy runner uses host mode and needs an explicitly coarser fallback.

Work

Infrastructure defines an activity lease from CPU, block I/O, network I/O, container events, and log progress. CPU idleness alone must not classify a job as stuck.
Infrastructure maps each Forgejo task and run to the correct job container.
The reaper requests cancellation through the Forgejo API first, waits through a bounded grace period, and kills only the mapped job container as a fallback.
The reaper never kills a shared Docker-in-Docker daemon, BuildKit service, or runner daemon.
Infrastructure defines separate thresholds for general jobs, image builds, and deploy jobs.
Infrastructure ships dry-run mode, structured audit output, and an alert before enforcement.
Tests exercise active low-CPU work, genuinely stalled work, shared-builder contention, cancellation failure, and fallback cleanup.
The operator performs live verification against each runner topology before enabling enforcement.

Done

A stalled job is cancelled without taking unrelated jobs or shared services down.
Legitimate low-CPU or long export work renews the activity lease.
Every cancellation records the run, job, reason, evidence, and cleanup result.
The operator can disable enforcement without redeploying the runner fleet.

Blocked by coilyco-flight-deck/infrastructure#620. ## Goal Infrastructure adopts Forgejo 16's workflow run cancellation API and adds an out-of-band stale-job reaper for Forgejo Actions jobs that stop making progress. ## Runner topology * The general pool runs one job per Docker-in-Docker runner. * The image-build pool runs up to four jobs against shared Docker-in-Docker and BuildKit services. * The deploy runner uses host mode and needs an explicitly coarser fallback. ## Work * Infrastructure defines an activity lease from CPU, block I/O, network I/O, container events, and log progress. CPU idleness alone must not classify a job as stuck. * Infrastructure maps each Forgejo task and run to the correct job container. * The reaper requests cancellation through the Forgejo API first, waits through a bounded grace period, and kills only the mapped job container as a fallback. * The reaper never kills a shared Docker-in-Docker daemon, BuildKit service, or runner daemon. * Infrastructure defines separate thresholds for general jobs, image builds, and deploy jobs. * Infrastructure ships dry-run mode, structured audit output, and an alert before enforcement. * Tests exercise active low-CPU work, genuinely stalled work, shared-builder contention, cancellation failure, and fallback cleanup. * The operator performs live verification against each runner topology before enabling enforcement. ## Done * A stalled job is cancelled without taking unrelated jobs or shared services down. * Legitimate low-CPU or long export work renews the activity lease. * Every cancellation records the run, job, reason, evidence, and cleanup result. * The operator can disable enforcement without redeploying the runner fleet.

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-29 03:46:26 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-08-05 10:25:59 +00:00