Forgejo CI runner fails all pull_request gate jobs before checkout (0s) - blocks every aos/ward PR merge #536

New issue

Closed

opened 2026-07-10 11:40:04 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-10 11:40:04 +00:00

Member

Problem

The Forgejo CI runner is failing every pull_request gate job before checkout, after 0s, which blocks all merges in the PR-gated repos (aos, ward, cli-guard). Repos that land via direct-main (infrastructure) are unaffected and merging normally, which is why this hid.

Observed:

ci / gate (pull_request) => failure after ~0s, "failed before checkout", on aos PRs #441, #436, #437 (and earlier #435/#415/#393). forgejo actions tasks list shows the recent pull_request gate tasks all failure.
Local ward test and ward pre-commit-all pass on those branches, so this is the CI runner environment, not the code.
The job-log route 404s (GET .../actions/runs/{run}/jobs/{job}/attempt/{attempt}/logs), so the failing step output is not visible from the run UI - see the sibling Forgejo Actions log plumbing issues (aos#415/#434).

Leading hypothesis (verify before acting)

infrastructure#486 ("Configure Forgejo runner default job timeouts") landed a runner config change to main at 2026-07-10T10:56Z - explicit runner config files now mounted into the shared build runner and deploy runner (deploy/forgejo-runner.yml, deploy/forgejo-runner-deploy.yml), plus a manifest guard. The gate failures began right after. A malformed mounted runner-config.yml, a bad --config path, or a mount that breaks the runner's job-container startup would produce exactly this "die before checkout after 0s" signature.

infrastructure#401 (seccomp opt-in for one job) also landed this session and is a secondary suspect, though it was scoped to a single opt-in job.

Diagnosis + fix

Inspect the shared build runner pod (forgejo-runner) logs on the cluster to see why the job container dies before checkout (config parse error, mount path, image pull, seccomp/cli-guard jail).
If the mounted runner config from #486 is the cause, repair or revert that config in deploy/forgejo-runner.yml / deploy/forgejo-runner-deploy.yml. Land the fix via infrastructure's direct-main workflow - it does not go through the broken PR gate, so the repair can land even while the gate is down.
Confirm a real pull_request gate job reaches checkout and passes after the runner reconverges.

Escalation

If the runner needs a live pod restart / manual convergence that a headless container cannot perform (no cluster reach - cf. #503, #531), escalate to an ops-role run or a human with live cluster access, and note it on this issue.

Impact / why P1

Blocks all aos + ward PR-workflow merges fleet-wide, including implemented-and-waiting PRs (aos#441 restores dropped ward-specs override capability + removes brittle validation; aos#436/#437 Forgejo Actions plumbing). Every director-lane engineer targeting a PR-gated repo will WARD-OUTCOME: blocked until this is fixed.

Suspected cause: infrastructure#486 (runner config, landed 10:56Z).
infrastructure#531 - deploy-factory-game workflow queue stuck (possibly same runner health).
aos#434/#415 - Forgejo Actions rerun + log plumbing (why we can't retrigger or read the failing logs).

## Problem The Forgejo CI runner is failing **every** `pull_request` gate job **before checkout, after 0s**, which blocks all merges in the PR-gated repos (aos, ward, cli-guard). Repos that land via `direct-main` (infrastructure) are unaffected and merging normally, which is why this hid. Observed: * `ci / gate (pull_request)` => `failure` after ~0s, "failed before checkout", on aos PRs **#441, #436, #437** (and earlier #435/#415/#393). `forgejo actions tasks list` shows the recent `pull_request` gate tasks all `failure`. * Local `ward test` and `ward pre-commit-all` pass on those branches, so this is the **CI runner environment, not the code**. * The job-log route 404s (`GET .../actions/runs/{run}/jobs/{job}/attempt/{attempt}/logs`), so the failing step output is not visible from the run UI - see the sibling Forgejo Actions log plumbing issues (aos#415/#434). ## Leading hypothesis (verify before acting) `infrastructure#486` ("Configure Forgejo runner default job timeouts") landed a **runner config change** to main at **2026-07-10T10:56Z** - explicit runner config files now mounted into the shared build runner and deploy runner (`deploy/forgejo-runner.yml`, `deploy/forgejo-runner-deploy.yml`), plus a manifest guard. The gate failures began right after. A malformed mounted `runner-config.yml`, a bad `--config` path, or a mount that breaks the runner's job-container startup would produce exactly this "die before checkout after 0s" signature. `infrastructure#401` (seccomp opt-in for one job) also landed this session and is a secondary suspect, though it was scoped to a single opt-in job. ## Diagnosis + fix 1. Inspect the shared build runner pod (`forgejo-runner`) logs on the cluster to see why the job container dies before checkout (config parse error, mount path, image pull, seccomp/cli-guard jail). 2. If the mounted runner config from `#486` is the cause, **repair or revert** that config in `deploy/forgejo-runner.yml` / `deploy/forgejo-runner-deploy.yml`. Land the fix via infrastructure's `direct-main` workflow - it does **not** go through the broken PR gate, so the repair can land even while the gate is down. 3. Confirm a real `pull_request` gate job reaches checkout and passes after the runner reconverges. ## Escalation If the runner needs a live pod restart / manual convergence that a headless container cannot perform (no cluster reach - cf. `#503`, `#531`), escalate to an ops-role run or a human with live cluster access, and note it on this issue. ## Impact / why P1 Blocks all aos + ward PR-workflow merges fleet-wide, including implemented-and-waiting PRs (aos#441 restores dropped ward-specs override capability + removes brittle validation; aos#436/#437 Forgejo Actions plumbing). Every director-lane engineer targeting a PR-gated repo will `WARD-OUTCOME: blocked` until this is fixed. ## Related * Suspected cause: `infrastructure#486` (runner config, landed 10:56Z). * `infrastructure#531` - deploy-factory-game workflow queue stuck (possibly same runner health). * aos#434/#415 - Forgejo Actions rerun + log plumbing (why we can't retrigger or read the failing logs).

coilyco-ops commented

2026-07-10 11:40:19 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-536 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T11:40:18Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#536 · branch issue-536 · harness codex · workflow direct-main
Run: engineer-codex-infrastructure-536 · ward v0.580.0 · dispatched 2026-07-10T11:40:18Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-536` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T11:40:18Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#536` · branch `issue-536` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-536` · ward `v0.580.0` · dispatched `2026-07-10T11:40:18Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 11:56:57 +00:00

fix(dev-base): refuse unpublished pins

coilyco-ops closed this issue

2026-07-10 11:56:57 +00:00

coilyco-ops commented

2026-07-10 12:00:43 +00:00

Author

Member

WARD-OUTCOME: done

details

workflow: direct-main; review summary: review gate skipped by ~/.ward/config.yaml default

felt: mechanical. confidence: high. surprise: the live failure was a missing dev-base image tag, not the runner config. follow-ups: added a rollout guard, pushed the workflow retag, and the probe PR reached checkout before failing in repo tests.

WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-main; review summary: review gate skipped by ~/.ward/config.yaml default felt: mechanical. confidence: high. surprise: the live failure was a missing dev-base image tag, not the runner config. follow-ups: added a rollout guard, pushed the workflow retag, and the probe PR reached checkout before failing in repo tests. </details>