Configure Forgejo runner default job timeouts without breaking long jobs #486
Labels
No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
coilyco-flight-deck/infrastructure#486
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context from the 2026-07-09 warded-agent recovery incident:
Forgejo workflow syntax supports
jobs.<job_id>.timeout-minutesandjobs.<job_id>.steps[*].timeout-minutes. Forgejo Runner also has a runner config value underrunner.timeoutdescribed as the timeout for a job to finish, defaulting to 3h, with the Forgejo instance also having a 3h job timeout. The current kai-server runner manifests (deploy/forgejo-runner.yml,deploy/forgejo-runner-deploy.yml) runforgejo-runner daemonwithout a mounted config file, so they inherit runner defaults.Goal:
timeout-minutesfor narrow known-risk steps like lint, since runner config is a hard fleet guard and not a per-job default injector.Acceptance:
Director handoff note, 2026-07-09T06:15Z:
The issue is captured from the warded-agent recovery incident, but this read-only director surface could not successfully dispatch it. Two attempts were made:
warded advisor coilyco-flight-deck/infrastructure#486 --harness codex --forcewarded engineer coilyco-flight-deck/infrastructure#486 --harness codex --force --skip-reviewBoth printed only the issue resolution line and then hung before the dispatch broker acknowledgement or the standard
ward-agent-reservationcomment. The launches were interrupted before any reservation appeared, so no sibling run is currently carrying this issue.Next director/host pass should retry dispatch from a healthy surface or hand-run
warded engineer coilyco-flight-deck/infrastructure#486 --harness codex --skip-review --force.🔎 ward agent advisor
ward agent advisor --harness codexran a one-shot standard research pass on this question:Current state in this repo:
deploy/forgejo-runner.ymlstarts the shared build runner withexec forgejo-runner daemonand no mounted config file.deploy/forgejo-runner-deploy.ymlstarts the dedicated deploy runner withcommand: ["forgejo-runner", "daemon"]and no mounted config file.deploy/forgejo-runner-tap-writer.ymlis another dedicated runner class already using its own label (tap-writer:host), which is a useful pattern reference.docs/FEATURES.mddescribes it as the runner that serves every repo's Forgejo Actions job, while the deploy runner is already isolated behindruns-on: deploy/ labeldeploy:host.What Forgejo itself documents:
jobs.<job_id>.timeout-minutesandjobs.<job_id>.steps[*].timeout-minutesas per-job / per-step walls.runner.timeout, documented as "the timeout for a job to be finished", default3h, with the note that the Forgejo instance also has its own job timeout (also3hby default).forgejo-runner daemon --config runner-config.yml.Advisory recommendation:
Use runner config as a fleet guardrail, not as a substitute for workflow-level intent. That means:
timeout-minuteson known-risk jobs or steps like lint, flaky network probes, and long shell loops.dockerrunner to5myet. That is too aggressive for a runner class that still appears to handle legitimate image-build and publish work.Recommended timeout values:
deploy/forgejo-runner.yml): start withrunner.timeout: 15m.20m+hangs.15mdoes that directly.5m: this runner still serves broadruns-on: dockertraffic, including at least this repo's own image-build workflow (.forgejo/workflows/build-runner-deploy.yml). A hard5mcap is likely to create false failures before the fleet has been split by label.3hfleet default is too weak for the incident class this issue is trying to contain.deploy/forgejo-runner-deploy.yml): setrunner.timeout: 60m.kubectl/helm, and rollout wait paths. The deploy runner is not the right place to enforce a short hang wall.45mwould also be defensible, but60mis the safer first cap.Phased option set:
5mto10m)runs-on: docker.15mto20m)5mto10mruns-on: dockerjobs.Important implementation boundary the issue should capture:
runner.timeoutis a hard max for everything that lands on that runner.timeout-minutesin workflow YAML is where job-specific intent belongs.Open questions that need one quick live check before anyone lands the change:
runs-on: docker, but the preciserunner.labelsvalue is not checked into git today because the build runner relies on persisted.runnerstate. The engineer should capture the current live label from the existing runner before pinningrunner.labels, otherwise a config rollout could strand jobs.runs-on: dockerjobs in sibling repos regularly exceed 15 minutes? If yes, those jobs should either get explicit workflowtimeout-minutesunder a longer dedicated runner class, or the first shared cap should be20minstead of15m.Risks to call out:
.runner. If checked-in config and registration flags diverge, a later PVC recreation or re-registration can silently change behavior. The engineer should keep checked-inrunner.labelsaligned with registration-time labels instead of relying on.runnerdefaults.timeout-minutesloses the ability to make narrow, intentional walls around especially risky steps.Validation recommendation:
--config. A small script or grep-based assertion is enough.workflow_dispatchtimeout probe, ideally temporary, with a job that intentionally sleeps longer than the configured runner cap. That proves cancellation is surfaced clearly by Forgejo Runner and the Forgejo UI. Do not make this probe push-triggered.Bottom line:
infrastructure.15mshared build cap,60mdeploy cap, and keep workflowtimeout-minutesfor job/step-specific control.5mto10mcap for small CI jobs later.Sources:
Researched and posted automatically by
ward agent advisor --harness codex(ward#179). This is one-shot research, not a carried change - verify before acting on it.— Codex, via
ward agent🔒 Reserved by
ward agent --harness codex— containerengineer-codex-infrastructure-486on hostkais-macbook-pro-2.localis carrying this issue (reserved 2026-07-09T06:16:08Z). Concurrentward agentruns are blocked until it finishes or the reservation goes stale (1h TTL);--forceoverrides.Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).
run seed context — what this run is carrying (ward#609)
coilyco-flight-deck/infrastructure#486· branchissue-486· harnesscodex· workflowprengineer-codex-infrastructure-486· wardv0.474.0· dispatched2026-07-09T06:16:08ZStatic container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.474.0).
— Codex, via
ward agentWARD-OUTCOME: done - explicit runner timeout caps landed for the shared build runner and deploy runner
workflow: pr; review summary: review gate skipped by --skip-review / --no-review-gate
The change went through cleanly once the runner config shape was pinned down. The only real fight was the docs size gate, which pushed the timeout boundary note into its own doc. Confidence is high in the guard and timeout split, and the main follow-up is watching for any future long-running build family that should get its own label instead of a bigger shared cap.
WARD-RESERVATION: held 🔒
reservation details
Holder: container
engineer-codex-infrastructure-486on hostkais-macbook-pro-2.local.Reserved by
ward agent --harness codex(reserved 2026-07-10T10:46:28Z). Concurrentward agentruns are blocked until it finishes or the reservation goes stale (1h TTL).--forceoverrides.Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).
run seed context — what this run is carrying (ward#609)
coilyco-flight-deck/infrastructure#486· branchissue-486· harnesscodex· workflowdirect-mainengineer-codex-infrastructure-486· wardv0.580.0· dispatched2026-07-10T10:46:28ZStatic container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).
— Codex, via
ward agentWARD-OUTCOME: done
details
workflow: direct-to-main; review summary: skipped (review gate skipped by ~/.ward/config.yaml default)
retrospective: this was a merge repair, not a fresh feature pass.
confidence: high
surprises: origin/main advanced once during the run, so the branch needed a second merge before landing.
follow-ups: none.