Main CI has been red since the Rust workspace entered the gate - needs live runner diagnosis #24

New issue

Closed

opened 2026-07-21 03:15:43 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-21 03:15:43 +00:00

Member

Needs an interactive / live-ops lane. Filed from a director-surface session that deliberately stopped short of push-probing CI.

Timeline evidence from ward ops forgejo action-run list:

Runs 8 and 9 (2026-07-09 ~18:10, commits 0988875 / 7fc2b39) are the last successes. At that point scripts/test-gate.sh ran only the pre-commit baseline.
Run 10 (commit 15ef500, the first push after the Rust workspace bootstrap added cargo test --workspace to the gate) failed, and every run since has failed - 11 days of red main across runs 10-20.
#14 bumped the CI image to forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:v0.230.0 specifically so cargo exists in CI, and was closed on the tag edit alone - run 11 (2ef497e, the bump commit itself) still failed, so the bump did not fix it and the failure mode was never confirmed fixed.
Runs 14-16 are three identical chore(ci): add Telegram main-failure alerts commits, which reads like an earlier agent iterating against the red workflow.
Runs 17-20 (2026-07-21, the #20-#23 simulation slices) fail the same way. bash scripts/test-gate.sh - the exact command CI runs - passes locally on every one of those commits, and cargo test --workspace is green, so the tree itself is healthy.

What could not be verified from here:

Job logs. The Forgejo API GET .../actions/runs/{run}/jobs/{job}/attempt/1/logs returns 404 for every run/job id combination tried (run index, run id, task id), so the actual failing step is unconfirmed.
action-run get shows duration: 1e+09 on the failed runs, which looks like a never-completed sentinel and hints the job may be dying early (image pull, runner capacity, or checkout) rather than failing in the test step.

Hypotheses to check with live access, in order:

Image pull failure or missing Rust toolchain at runtime in agentic-os:v0.230.0 on the runner (was run 11's failure ever read?).
cargo test --workspace cold-start cost in a fresh container (crates.io index + first build) against the 12-minute job timeout or restricted runner egress.
pre-commit run --all-files hook-environment installs downloading in-container.
LFS checkout (lfs: true) behavior on the runner.

Please read the run 20 job log from the Forgejo UI or runner host, identify the failing step, and either fix the runner/image side or file the resulting repo-side change. The Telegram alert step (if: failure() on main) presumably fires on every push right now, which is worth confirming too.

Needs an interactive / live-ops lane. Filed from a director-surface session that deliberately stopped short of push-probing CI. Timeline evidence from `ward ops forgejo action-run list`: * Runs 8 and 9 (2026-07-09 ~18:10, commits `0988875` / `7fc2b39`) are the last successes. At that point `scripts/test-gate.sh` ran only the pre-commit baseline. * Run 10 (commit `15ef500`, the first push after the Rust workspace bootstrap added `cargo test --workspace` to the gate) failed, and every run since has failed - 11 days of red main across runs 10-20. * #14 bumped the CI image to `forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:v0.230.0` specifically so `cargo` exists in CI, and was closed on the tag edit alone - run 11 (`2ef497e`, the bump commit itself) still failed, so the bump did not fix it and the failure mode was never confirmed fixed. * Runs 14-16 are three identical `chore(ci): add Telegram main-failure alerts` commits, which reads like an earlier agent iterating against the red workflow. * Runs 17-20 (2026-07-21, the #20-#23 simulation slices) fail the same way. `bash scripts/test-gate.sh` - the exact command CI runs - passes locally on every one of those commits, and `cargo test --workspace` is green, so the tree itself is healthy. What could not be verified from here: * Job logs. The Forgejo API `GET .../actions/runs/{run}/jobs/{job}/attempt/1/logs` returns 404 for every run/job id combination tried (run index, run id, task id), so the actual failing step is unconfirmed. * `action-run get` shows `duration: 1e+09` on the failed runs, which looks like a never-completed sentinel and hints the job may be dying early (image pull, runner capacity, or checkout) rather than failing in the test step. Hypotheses to check with live access, in order: 1. Image pull failure or missing Rust toolchain at runtime in `agentic-os:v0.230.0` on the runner (was run 11's failure ever read?). 2. `cargo test --workspace` cold-start cost in a fresh container (crates.io index + first build) against the 12-minute job timeout or restricted runner egress. 3. `pre-commit run --all-files` hook-environment installs downloading in-container. 4. LFS checkout (`lfs: true`) behavior on the runner. Please read the run 20 job log from the Forgejo UI or runner host, identify the failing step, and either fix the runner/image side or file the resulting repo-side change. The Telegram alert step (`if: failure()` on main) presumably fires on every push right now, which is worth confirming too.

coilyco-ops referenced this issue from a commit

2026-07-21 05:11:08 +00:00

fix(ci): bump dev-base image to the Rust+wasm v0.255.0 (closes #24)

coilyco-ops closed this issue

2026-07-21 05:11:08 +00:00

coilyco-ops commented

2026-07-21 05:14:43 +00:00

Author

Member

Resolved on main in 41fbc51 - CI run 22 is green, the first green main since run 9 on Jul 9.

Root cause confirmed: v0.230.0 never contained Rust. The aos#382 outcome pointed downstream at a tag cut from e69540e, minutes before the rustup commit (3badc26) landed - the first Rust-bearing aos release was actually v0.243.0. Every factory-game run since the gate added cargo test --workspace died on the missing toolchain.

The fix rode the aos#610 dev-base work: the repo now pins v0.255.0, which carries the new lang-rust tier (cargo/rustc 1.97, wasm32-unknown-unknown, trunk) plus the Bevy-class native libs, so the same bump also unblocks the #18 app shell. Job timeout raised to 30 minutes ahead of Bevy compiles entering the workspace.

Not pursued from the original hypothesis list: the job-logs API 404s turned out to be a Forgejo version gap (the web UI serves them fine), and the duration: 1e+09 sentinel was cosmetic. One real runner-side observation from the aos side worth keeping an eye on: two concurrent buildx builds died simultaneously mid-publish (builder/daemon death, likely memory pressure), and the docker-CLI bootstrap in the publish jobs hit curl (35) network resets twice tonight. If those recur, that is a separate infra issue.

WARD-OUTCOME: done

Resolved on `main` in `41fbc51` - CI run 22 is green, the first green main since run 9 on Jul 9. Root cause confirmed: `v0.230.0` never contained Rust. The aos#382 outcome pointed downstream at a tag cut from `e69540e`, minutes before the rustup commit (`3badc26`) landed - the first Rust-bearing aos release was actually `v0.243.0`. Every factory-game run since the gate added `cargo test --workspace` died on the missing toolchain. The fix rode the aos#610 dev-base work: the repo now pins `v0.255.0`, which carries the new `lang-rust` tier (cargo/rustc 1.97, `wasm32-unknown-unknown`, trunk) plus the Bevy-class native libs, so the same bump also unblocks the #18 app shell. Job timeout raised to 30 minutes ahead of Bevy compiles entering the workspace. Not pursued from the original hypothesis list: the job-logs API 404s turned out to be a Forgejo version gap (the web UI serves them fine), and the `duration: 1e+09` sentinel was cosmetic. One real runner-side observation from the aos side worth keeping an eye on: two concurrent buildx builds died simultaneously mid-publish (builder/daemon death, likely memory pressure), and the docker-CLI bootstrap in the publish jobs hit `curl (35)` network resets twice tonight. If those recur, that is a separate infra issue. `WARD-OUTCOME: done`

coilyco-ops referenced this issue

2026-07-21 05:15:51 +00:00

Shell follow-ups: CI cargo caching and wasm-opt for the bundle #26