Configure Forgejo runner default job timeouts without breaking long jobs #486

Closed
opened 2026-07-09 06:13:29 +00:00 by coilyco-ops · 6 comments
Member

Context from the 2026-07-09 warded-agent recovery incident:

Forgejo workflow syntax supports jobs.<job_id>.timeout-minutes and jobs.<job_id>.steps[*].timeout-minutes. Forgejo Runner also has a runner config value under runner.timeout described as the timeout for a job to finish, defaulting to 3h, with the Forgejo instance also having a 3h job timeout. The current kai-server runner manifests (deploy/forgejo-runner.yml, deploy/forgejo-runner-deploy.yml) run forgejo-runner daemon without a mounted config file, so they inherit runner defaults.

Goal:

  • add explicit runner config for the build runners and deploy runner instead of relying on generated defaults.
  • choose safe timeout caps per runner class. A 5m cap may be right for small test/build runners but is probably too low for release, image build, and deploy jobs unless those use separate labels/runners.
  • keep workflow-level timeout-minutes for narrow known-risk steps like lint, since runner config is a hard fleet guard and not a per-job default injector.
  • document the boundary so future agents know when to use runner config vs workflow YAML.

Acceptance:

  • the runner manifests mount and pass an explicit config file with timeout values.
  • build/test jobs have a bounded default or cap that prevents 20m+ hangs.
  • longer release/deploy jobs either keep a longer runner cap or move to a separate runner label.
  • existing labels/capacity/cache behavior are preserved.
  • validation includes a manifest check and, if practical, one controlled workflow that proves timeout cancellation is reported clearly.
Context from the 2026-07-09 warded-agent recovery incident: Forgejo workflow syntax supports `jobs.<job_id>.timeout-minutes` and `jobs.<job_id>.steps[*].timeout-minutes`. Forgejo Runner also has a runner config value under `runner.timeout` described as the timeout for a job to finish, defaulting to 3h, with the Forgejo instance also having a 3h job timeout. The current kai-server runner manifests (`deploy/forgejo-runner.yml`, `deploy/forgejo-runner-deploy.yml`) run `forgejo-runner daemon` without a mounted config file, so they inherit runner defaults. Goal: * add explicit runner config for the build runners and deploy runner instead of relying on generated defaults. * choose safe timeout caps per runner class. A 5m cap may be right for small test/build runners but is probably too low for release, image build, and deploy jobs unless those use separate labels/runners. * keep workflow-level `timeout-minutes` for narrow known-risk steps like lint, since runner config is a hard fleet guard and not a per-job default injector. * document the boundary so future agents know when to use runner config vs workflow YAML. Acceptance: * the runner manifests mount and pass an explicit config file with timeout values. * build/test jobs have a bounded default or cap that prevents 20m+ hangs. * longer release/deploy jobs either keep a longer runner cap or move to a separate runner label. * existing labels/capacity/cache behavior are preserved. * validation includes a manifest check and, if practical, one controlled workflow that proves timeout cancellation is reported clearly.
Author
Member

Director handoff note, 2026-07-09T06:15Z:

The issue is captured from the warded-agent recovery incident, but this read-only director surface could not successfully dispatch it. Two attempts were made:

  • warded advisor coilyco-flight-deck/infrastructure#486 --harness codex --force
  • warded engineer coilyco-flight-deck/infrastructure#486 --harness codex --force --skip-review

Both printed only the issue resolution line and then hung before the dispatch broker acknowledgement or the standard ward-agent-reservation comment. The launches were interrupted before any reservation appeared, so no sibling run is currently carrying this issue.

Next director/host pass should retry dispatch from a healthy surface or hand-run warded engineer coilyco-flight-deck/infrastructure#486 --harness codex --skip-review --force.

Director handoff note, 2026-07-09T06:15Z: The issue is captured from the warded-agent recovery incident, but this read-only director surface could not successfully dispatch it. Two attempts were made: * `warded advisor coilyco-flight-deck/infrastructure#486 --harness codex --force` * `warded engineer coilyco-flight-deck/infrastructure#486 --harness codex --force --skip-review` Both printed only the issue resolution line and then hung before the dispatch broker acknowledgement or the standard `ward-agent-reservation` comment. The launches were interrupted before any reservation appeared, so no sibling run is currently carrying this issue. Next director/host pass should retry dispatch from a healthy surface or hand-run `warded engineer coilyco-flight-deck/infrastructure#486 --harness codex --skip-review --force`.
Author
Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.


Current state in this repo:

  • deploy/forgejo-runner.yml starts the shared build runner with exec forgejo-runner daemon and no mounted config file.
  • deploy/forgejo-runner-deploy.yml starts the dedicated deploy runner with command: ["forgejo-runner", "daemon"] and no mounted config file.
  • deploy/forgejo-runner-tap-writer.yml is another dedicated runner class already using its own label (tap-writer:host), which is a useful pattern reference.
  • The shared build runner is especially sensitive because docs/FEATURES.md describes it as the runner that serves every repo's Forgejo Actions job, while the deploy runner is already isolated behind runs-on: deploy / label deploy:host.

What Forgejo itself documents:

  • Forgejo workflow syntax supports jobs.<job_id>.timeout-minutes and jobs.<job_id>.steps[*].timeout-minutes as per-job / per-step walls.
  • Forgejo Runner also supports runner.timeout, documented as "the timeout for a job to be finished", default 3h, with the note that the Forgejo instance also has its own job timeout (also 3h by default).
  • Runner config is not auto-discovered. Official docs say the daemon should be started with an explicit config path such as forgejo-runner daemon --config runner-config.yml.

Advisory recommendation:

Use runner config as a fleet guardrail, not as a substitute for workflow-level intent. That means:

  1. Add checked-in runner config files and mount them explicitly for the shared build runner and the deploy runner.
  2. Keep targeted workflow timeout-minutes on known-risk jobs or steps like lint, flaky network probes, and long shell loops.
  3. Do not set the shared docker runner to 5m yet. That is too aggressive for a runner class that still appears to handle legitimate image-build and publish work.
  4. Set a moderate cap on the shared build runner first, then split out true long-running jobs if tighter caps are still wanted.

Recommended timeout values:

  • Shared build runner (deploy/forgejo-runner.yml): start with runner.timeout: 15m.
    • Why: the issue goal is to stop 20m+ hangs. 15m does that directly.
    • Why not 5m: this runner still serves broad runs-on: docker traffic, including at least this repo's own image-build workflow (.forgejo/workflows/build-runner-deploy.yml). A hard 5m cap is likely to create false failures before the fleet has been split by label.
    • Why not leave it high: a 3h fleet default is too weak for the incident class this issue is trying to contain.
  • Deploy runner (deploy/forgejo-runner-deploy.yml): set runner.timeout: 60m.
    • Why: deploy jobs are rarer, already isolated by label, and can legitimately spend time in checkout, source resolution, image pull, kubectl / helm, and rollout wait paths. The deploy runner is not the right place to enforce a short hang wall.
    • If observed deploys are consistently much shorter, 45m would also be defensible, but 60m is the safer first cap.

Phased option set:

  • Option A: Tight shared cap now (5m to 10m)
    • Pro: strongest immediate hang protection.
    • Con: high regression risk for any legitimate build/publish job still on runs-on: docker.
    • Recommendation: do not choose this as the first move.
  • Option B: Moderate shared cap now (15m to 20m)
    • Pro: meets the stated goal of preventing 20m+ hangs with low migration cost.
    • Con: not as strict as a small-job-only runner class.
    • Recommendation: best immediate move.
  • Option C: Split long jobs onto a new runner label, then shrink shared build runner to 5m to 10m
    • Pro: best long-term design.
    • Con: needs cross-repo workflow edits because the shared build runner currently serves fleet-wide runs-on: docker jobs.
    • Recommendation: best end state, but phase 2, not phase 1.

Important implementation boundary the issue should capture:

  • runner.timeout is a hard max for everything that lands on that runner.
  • timeout-minutes in workflow YAML is where job-specific intent belongs.
  • So the repo docs should explicitly say: use runner config to set the outer safety envelope for a runner class, and use workflow YAML to narrow specific risky jobs or steps inside that envelope.

Open questions that need one quick live check before anyone lands the change:

  • What exact label string does the current shared build runner advertise? This repo's workflows use runs-on: docker, but the precise runner.labels value is not checked into git today because the build runner relies on persisted .runner state. The engineer should capture the current live label from the existing runner before pinning runner.labels, otherwise a config rollout could strand jobs.
  • Do any real runs-on: docker jobs in sibling repos regularly exceed 15 minutes? If yes, those jobs should either get explicit workflow timeout-minutes under a longer dedicated runner class, or the first shared cap should be 20m instead of 15m.
  • Should tap-writer get the same explicit-config treatment in the same sweep? It is out of the issue acceptance as written, but from a design-consistency standpoint it probably should.

Risks to call out:

  • Label drift risk: the runner PVCs persist .runner. If checked-in config and registration flags diverge, a later PVC recreation or re-registration can silently change behavior. The engineer should keep checked-in runner.labels aligned with registration-time labels instead of relying on .runner defaults.
  • Fleet regression risk: the shared build runner serves many repos, so a too-low cap will create cross-repo failures that do not show up from this repo alone.
  • False sense of safety risk: adding only runner caps without preserving workflow-level timeout-minutes loses the ability to make narrow, intentional walls around especially risky steps.

Validation recommendation:

  • Static: add a repo-level manifest check that fails if runner manifests do not mount a checked-in config file and do not start the daemon with --config. A small script or grep-based assertion is enough.
  • Behavioral: add one manual-only workflow_dispatch timeout probe, ideally temporary, with a job that intentionally sleeps longer than the configured runner cap. That proves cancellation is surfaced clearly by Forgejo Runner and the Forgejo UI. Do not make this probe push-triggered.

Bottom line:

  • Land this as single-repo work in infrastructure.
  • Recommend explicit runner config files, 15m shared build cap, 60m deploy cap, and keep workflow timeout-minutes for job/step-specific control.
  • Treat a dedicated long-job runner label as the likely follow-on if Kai still wants a 5m to 10m cap for small CI jobs later.

Sources:


Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- Current state in this repo: - `deploy/forgejo-runner.yml` starts the shared build runner with `exec forgejo-runner daemon` and no mounted config file. - `deploy/forgejo-runner-deploy.yml` starts the dedicated deploy runner with `command: ["forgejo-runner", "daemon"]` and no mounted config file. - `deploy/forgejo-runner-tap-writer.yml` is another dedicated runner class already using its own label (`tap-writer:host`), which is a useful pattern reference. - The shared build runner is especially sensitive because `docs/FEATURES.md` describes it as the runner that serves every repo's Forgejo Actions job, while the deploy runner is already isolated behind `runs-on: deploy` / label `deploy:host`. What Forgejo itself documents: - Forgejo workflow syntax supports `jobs.<job_id>.timeout-minutes` and `jobs.<job_id>.steps[*].timeout-minutes` as per-job / per-step walls. - Forgejo Runner also supports `runner.timeout`, documented as "the timeout for a job to be finished", default `3h`, with the note that the Forgejo instance also has its own job timeout (also `3h` by default). - Runner config is not auto-discovered. Official docs say the daemon should be started with an explicit config path such as `forgejo-runner daemon --config runner-config.yml`. Advisory recommendation: **Use runner config as a fleet guardrail, not as a substitute for workflow-level intent.** That means: 1. Add checked-in runner config files and mount them explicitly for the shared build runner and the deploy runner. 2. Keep targeted workflow `timeout-minutes` on known-risk jobs or steps like lint, flaky network probes, and long shell loops. 3. Do not set the shared `docker` runner to `5m` yet. That is too aggressive for a runner class that still appears to handle legitimate image-build and publish work. 4. Set a moderate cap on the shared build runner first, then split out true long-running jobs if tighter caps are still wanted. Recommended timeout values: - **Shared build runner (`deploy/forgejo-runner.yml`)**: start with **`runner.timeout: 15m`**. - Why: the issue goal is to stop `20m+` hangs. `15m` does that directly. - Why not `5m`: this runner still serves broad `runs-on: docker` traffic, including at least this repo's own image-build workflow (`.forgejo/workflows/build-runner-deploy.yml`). A hard `5m` cap is likely to create false failures before the fleet has been split by label. - Why not leave it high: a `3h` fleet default is too weak for the incident class this issue is trying to contain. - **Deploy runner (`deploy/forgejo-runner-deploy.yml`)**: set **`runner.timeout: 60m`**. - Why: deploy jobs are rarer, already isolated by label, and can legitimately spend time in checkout, source resolution, image pull, `kubectl` / `helm`, and rollout wait paths. The deploy runner is not the right place to enforce a short hang wall. - If observed deploys are consistently much shorter, `45m` would also be defensible, but `60m` is the safer first cap. Phased option set: - **Option A: Tight shared cap now (`5m` to `10m`)** - Pro: strongest immediate hang protection. - Con: high regression risk for any legitimate build/publish job still on `runs-on: docker`. - Recommendation: **do not choose this as the first move**. - **Option B: Moderate shared cap now (`15m` to `20m`)** - Pro: meets the stated goal of preventing 20m+ hangs with low migration cost. - Con: not as strict as a small-job-only runner class. - Recommendation: **best immediate move**. - **Option C: Split long jobs onto a new runner label, then shrink shared build runner to `5m` to `10m`** - Pro: best long-term design. - Con: needs cross-repo workflow edits because the shared build runner currently serves fleet-wide `runs-on: docker` jobs. - Recommendation: **best end state, but phase 2, not phase 1**. Important implementation boundary the issue should capture: - `runner.timeout` is a **hard max** for everything that lands on that runner. - `timeout-minutes` in workflow YAML is where job-specific intent belongs. - So the repo docs should explicitly say: use runner config to set the outer safety envelope for a runner class, and use workflow YAML to narrow specific risky jobs or steps inside that envelope. Open questions that need one quick live check before anyone lands the change: - **What exact label string does the current shared build runner advertise?** This repo's workflows use `runs-on: docker`, but the precise `runner.labels` value is not checked into git today because the build runner relies on persisted `.runner` state. The engineer should capture the current live label from the existing runner before pinning `runner.labels`, otherwise a config rollout could strand jobs. - **Do any real `runs-on: docker` jobs in sibling repos regularly exceed 15 minutes?** If yes, those jobs should either get explicit workflow `timeout-minutes` under a longer dedicated runner class, or the first shared cap should be `20m` instead of `15m`. - **Should tap-writer get the same explicit-config treatment in the same sweep?** It is out of the issue acceptance as written, but from a design-consistency standpoint it probably should. Risks to call out: - **Label drift risk**: the runner PVCs persist `.runner`. If checked-in config and registration flags diverge, a later PVC recreation or re-registration can silently change behavior. The engineer should keep checked-in `runner.labels` aligned with registration-time labels instead of relying on `.runner` defaults. - **Fleet regression risk**: the shared build runner serves many repos, so a too-low cap will create cross-repo failures that do not show up from this repo alone. - **False sense of safety risk**: adding only runner caps without preserving workflow-level `timeout-minutes` loses the ability to make narrow, intentional walls around especially risky steps. Validation recommendation: - **Static**: add a repo-level manifest check that fails if runner manifests do not mount a checked-in config file and do not start the daemon with `--config`. A small script or grep-based assertion is enough. - **Behavioral**: add one **manual-only** `workflow_dispatch` timeout probe, ideally temporary, with a job that intentionally sleeps longer than the configured runner cap. That proves cancellation is surfaced clearly by Forgejo Runner and the Forgejo UI. Do not make this probe push-triggered. Bottom line: - Land this as **single-repo work in `infrastructure`**. - Recommend **explicit runner config files**, **`15m` shared build cap**, **`60m` deploy cap**, and **keep workflow `timeout-minutes` for job/step-specific control**. - Treat a dedicated long-job runner label as the likely follow-on if Kai still wants a `5m` to `10m` cap for small CI jobs later. Sources: - Forgejo Runner installation/config docs: https://forgejo.org/docs/v15.0/admin/actions/installation/docker/ and https://forgejo.org/docs/v15.0/admin/actions/installation/binary/ - Forgejo Runner config reference: https://forgejo.org/docs/v15.0/admin/actions/configuration/ - Forgejo Actions workflow reference: https://forgejo.org/docs/v15.0/user/actions/reference/ --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it. <!-- ward-agent-reply --> <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

🔒 Reserved by ward agent --harness codex — container engineer-codex-infrastructure-486 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-09T06:16:08Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/infrastructure#486 · branch issue-486 · harness codex · workflow pr
  • Run: engineer-codex-infrastructure-486 · ward v0.474.0 · dispatched 2026-07-09T06:16:08Z
  • Comment thread: 2 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.474.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --harness codex` — container `engineer-codex-infrastructure-486` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-09T06:16:08Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#486` · branch `issue-486` · harness `codex` · workflow `pr` - **Run:** `engineer-codex-infrastructure-486` · ward `v0.474.0` · dispatched `2026-07-09T06:16:08Z` - **Comment thread:** 2 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-09T06:15:46Z), @coilyco-ops (2026-07-09T06:16:07Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.474.0). </details> <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

WARD-OUTCOME: done - explicit runner timeout caps landed for the shared build runner and deploy runner
workflow: pr; review summary: review gate skipped by --skip-review / --no-review-gate
The change went through cleanly once the runner config shape was pinned down. The only real fight was the docs size gate, which pushed the timeout boundary note into its own doc. Confidence is high in the guard and timeout split, and the main follow-up is watching for any future long-running build family that should get its own label instead of a bigger shared cap.

WARD-OUTCOME: done - explicit runner timeout caps landed for the shared build runner and deploy runner workflow: pr; review summary: review gate skipped by --skip-review / --no-review-gate The change went through cleanly once the runner config shape was pinned down. The only real fight was the docs size gate, which pushed the timeout boundary note into its own doc. Confidence is high in the guard and timeout split, and the main follow-up is watching for any future long-running build family that should get its own label instead of a bigger shared cap.
Author
Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-486 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:46:28Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/infrastructure#486 · branch issue-486 · harness codex · workflow direct-main
  • Run: engineer-codex-infrastructure-486 · ward v0.580.0 · dispatched 2026-07-10T10:46:28Z
  • Comment thread: 3 included in the pre-flight read, 1 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-486` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:46:28Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#486` · branch `issue-486` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-486` · ward `v0.580.0` · dispatched `2026-07-10T10:46:28Z` - **Comment thread:** 3 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-09T06:15:46Z), @coilyco-ops (2026-07-09T06:16:07Z), @coilyco-ops (2026-07-09T06:25:40Z) - stripped: @coilyco-ops (2026-07-09T06:16:10Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details> <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

WARD-OUTCOME: done

details

workflow: direct-to-main; review summary: skipped (review gate skipped by ~/.ward/config.yaml default)
retrospective: this was a merge repair, not a fresh feature pass.
confidence: high
surprises: origin/main advanced once during the run, so the branch needed a second merge before landing.
follow-ups: none.

WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-to-main; review summary: skipped (review gate skipped by ~/.ward/config.yaml default) retrospective: this was a merge repair, not a fresh feature pass. confidence: high surprises: origin/main advanced once during the run, so the branch needed a second merge before landing. follow-ups: none. </details>
Sign in to join this conversation.
No description provided.