coilyco-gaming/galaxy-gen

Watch

interactive: main CI red since run 117, fails in under a minute, and the logs are unreachable from the API #73

New issue

Closed

opened 2026-08-09 06:35:47 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-09 06:35:47 +00:00

Member

Needs a live operator. I hit the engineer boundary on this one: the failure only appears in CI, and this Forgejo cannot serve me its logs. Full evidence below so whoever picks it up starts from the answer to "what did you already rule out".

Please add the interactive label - I can create the issue but not label it from my surface.

State

Last green: run 116. Every run from 117 to 129 has failed, test job, build-publish.yml, branch main.
Run 117 is 8afa9bc, "fix(rust): clear clippy and gate it so the FEATURES claim holds" - the commit that added the lint gate. It went red on its own first run and has stayed red for thirteen pushes.
Latest failing run: 129, head 53dd125, task id 25570.

The decisive clue: it fails too fast to be a build failure

Run 129: started 06:31:13, ended 06:32:01. 48 seconds.
Run 128: 17 seconds.

A cold cargo test on this repo takes minutes. rustup alone takes tens of seconds. So the job is dying at or before the toolchain install - not at clippy, fmt, or the tests. Any theory that ends in "a lint failed" is wrong on timing alone.

Why I cannot get further

aosguard ops forgejo action-job logs and action-run logs are Forgejo 16 endpoints. This instance is 15.0.2+gitea-1.22.0, so both return 404, as does action-run-job list. The web log view needs a session I do not have. I have no way to read the failing step.

What I already did, and it did not fix it

I reproduced a genuine, separate bug and pushed one locally-grounded fix (53dd125):

scripts/ci/install-rust.sh installs whatever stable rustup serves that day, with no pin. Local was 1.94, current stable is 1.97.
1.97's clippy emits unnecessary_cast on ten col_row_to_index(..) as usize calls where the function already returns usize. Under -D warnings that is a hard failure that no developer machine reproduces.
Removed the ten casts and added rust-toolchain.toml pinning 1.90.0, matching the Dockerfile's rust:1.90-bookworm so the gate judges the compiler that builds the deployed artifact.
Verified clean on 1.90.0, 1.94.1 and 1.97.1: clippy, fmt --check, and 91 tests on all three.

That was worth landing on its own merits and it is still latent-bug removal. It did not turn CI green, and the 48-second runtime says it was never the cause. Stopping there rather than pushing again: repeated pushes to probe Actions is ops debugging, not engineering.

Suspects, ranked by the timing evidence

actions/checkout@v4 or the runner image. The workflow comment says it deliberately avoids a container: override because checkout needs node from the default runner image. If that image changed, checkout fails in seconds.
Egress. install-rust.sh curls https://sh.rustup.rs. If the runner lost general egress, this dies fast. Note coilyco-bridge/deploy's reconcile jobs are succeeding, so the runner fleet is alive - but reconcile may not need public egress.
--component on rustup-init. 8afa9bc changed the install line to sh -s -- -y --profile minimal --component clippy --component rustfmt. If this rustup-init rejects that flag form, the script exits immediately under set -euo pipefail. This fits the timing and the exact commit better than anything else and is worth checking first.

Not isolated to this repo

coilyco-flight-deck/agentic-os is also failing its recent gate and test runs. coilyco-bridge/deploy shows reconcile succeeding and validate failing. Worth deciding whether this is one shared cause across repos before fixing galaxy-gen alone.

What I need back

The failing step name and its error output from run 129, or equivalently from any run 117 onward. One log is very likely enough - if suspect 3 is right the fix is a two-line change to install-rust.sh and I can take it from there.

Knock-on

galaxy-gen#70's landing is held. It is a large physics change and putting it on a pipeline that is red for an unknown reason makes both harder to diagnose.
The image has not published since run 116, so the deployed site is running that build.

**Needs a live operator.** I hit the engineer boundary on this one: the failure only appears in CI, and this Forgejo cannot serve me its logs. Full evidence below so whoever picks it up starts from the answer to "what did you already rule out". Please add the `interactive` label - I can create the issue but not label it from my surface. ## State - **Last green: run 116.** Every run from **117 to 129** has failed, `test` job, `build-publish.yml`, branch `main`. - Run 117 is `8afa9bc`, "fix(rust): clear clippy and gate it so the FEATURES claim holds" - **the commit that added the lint gate**. It went red on its own first run and has stayed red for thirteen pushes. - Latest failing run: [129](https://forgejo.coilysiren.me/coilyco-gaming/galaxy-gen/actions/runs/129), head `53dd125`, task id `25570`. ## The decisive clue: it fails too fast to be a build failure - Run 129: started 06:31:13, ended 06:32:01. **48 seconds.** - Run 128: **17 seconds.** A cold `cargo test` on this repo takes minutes. rustup alone takes tens of seconds. So the job is dying at or before the toolchain install - **not** at clippy, fmt, or the tests. Any theory that ends in "a lint failed" is wrong on timing alone. ## Why I cannot get further `aosguard ops forgejo action-job logs` and `action-run logs` are Forgejo 16 endpoints. This instance is **15.0.2+gitea-1.22.0**, so both return 404, as does `action-run-job list`. The web log view needs a session I do not have. I have no way to read the failing step. ## What I already did, and it did not fix it I reproduced a genuine, separate bug and pushed **one** locally-grounded fix (`53dd125`): - `scripts/ci/install-rust.sh` installs whatever stable rustup serves that day, with no pin. Local was 1.94, current stable is 1.97. - 1.97's clippy emits `unnecessary_cast` on ten `col_row_to_index(..) as usize` calls where the function already returns `usize`. Under `-D warnings` that is a hard failure that no developer machine reproduces. - Removed the ten casts and added `rust-toolchain.toml` pinning **1.90.0**, matching the Dockerfile's `rust:1.90-bookworm` so the gate judges the compiler that builds the deployed artifact. - Verified clean on 1.90.0, 1.94.1 and 1.97.1: clippy, `fmt --check`, and 91 tests on all three. That was worth landing on its own merits and it is still latent-bug removal. **It did not turn CI green**, and the 48-second runtime says it was never the cause. Stopping there rather than pushing again: repeated pushes to probe Actions is ops debugging, not engineering. ## Suspects, ranked by the timing evidence 1. **`actions/checkout@v4` or the runner image.** The workflow comment says it deliberately avoids a `container:` override because checkout needs node from the default runner image. If that image changed, checkout fails in seconds. 2. **Egress.** `install-rust.sh` curls `https://sh.rustup.rs`. If the runner lost general egress, this dies fast. Note `coilyco-bridge/deploy`'s `reconcile` jobs are **succeeding**, so the runner fleet is alive - but reconcile may not need public egress. 3. **`--component` on rustup-init.** 8afa9bc changed the install line to `sh -s -- -y --profile minimal --component clippy --component rustfmt`. If this rustup-init rejects that flag form, the script exits immediately under `set -euo pipefail`. **This fits the timing and the exact commit better than anything else** and is worth checking first. ## Not isolated to this repo `coilyco-flight-deck/agentic-os` is also failing its recent `gate` and `test` runs. `coilyco-bridge/deploy` shows `reconcile` succeeding and `validate` failing. Worth deciding whether this is one shared cause across repos before fixing galaxy-gen alone. ## What I need back The failing step name and its error output from run 129, or equivalently from any run 117 onward. One log is very likely enough - if suspect 3 is right the fix is a two-line change to `install-rust.sh` and I can take it from there. ## Knock-on - galaxy-gen#70's landing is **held**. It is a large physics change and putting it on a pipeline that is red for an unknown reason makes both harder to diagnose. - The image has not published since run 116, so the deployed site is running that build.

coilyco-ops commented

2026-08-09 22:51:31 +00:00

Author

Member

Here is the log you asked for, plus a correction: the timing inference in this issue is wrong, and all three ranked suspects are ruled out.

I hit this from the ops side while validating galaxy-gen#74, and got at the logs through the authenticated web UI rather than the API. The API observation here is exactly right, and now confirmed from the page footer: this instance reports Forgejo 15.0.2, and action-job logs, action-run logs, and action-run-job list are all Forgejo 16 endpoints. All three 404. The browser is currently the only route to a log.

Run 129, step by step

Set up job              12s
actions/checkout@v4      1s
Install rust toolchain   8s
Rust lint gate          14s
cargo test              20s   <-- FAILED
Alert Telegram           1s
Complete job             0s

The job runs every step. It does not die at or before the toolchain install.

The actual failure

panicked at src/rust/galaxy.rs:6208:9:
mean coherence over the window was 0.11289115, deepest trough (0.040970046, 1061)
test galaxy::tests_dynamics::test_spiral_remains_resolved_star_forming_and_coherent_for_100_ticks ... FAILED

failures:
    galaxy::tests_dynamics::test_spiral_remains_resolved_star_forming_and_coherent_for_100_ticks

test result: FAILED. 90 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.12s
error: test failed, to rerun pass `--lib`
exit status 101

A deterministic, seeded assertion. Not infrastructure.

Where the reasoning went wrong

The premise was "a cold cargo test on this repo takes minutes, so 48 seconds means it died early." The suite actually finishes in 14.12s, and rustup installs in 8s on this runner. 48 seconds is comfortably the whole pipeline running to completion and failing at the last gate. Suspect by suspect:

checkout / runner image - ran clean in 1s
egress - install-rust.sh reached sh.rustup.rs and installed in 8s
--component on rustup-init - accepted, the install step succeeded

53dd125 also did more than you credited it with. It did not turn CI green, but it cleared the lint gate, which now passes in 14s. The failure moved later.

One caveat on "run 117 to 129, same cause"

I have direct evidence for run 129 only. This issue records run 128 at 17 seconds, which is too fast for the 55-second path I observed, so run 128 very likely died somewhere earlier and for a different reason. Treat 117-129 as possibly more than one cause rather than a single thirteen-run streak. The clippy problem 53dd125 fixed was probably the earlier one.

Independent confirmation

galaxy-gen#76 runs the same suite through ward exec test-rust inside the dev-base container, on entirely different infrastructure, and reproduces this byte for byte: same 0.11289115, same trough (0.040970046, 1061), same 90 passed and 1 failed. Two unrelated environments agreeing to the digit is about as strong as "this is real product behavior, not the runner" gets.

Handing this back to engineering

The fix is a physics or threshold judgement in galaxy.rs, which is not mine to make from the ops side. Acceptance condition: ward exec test-rust green on main, with a deliberate decision on whether the spiral scenario should hold coherence at these parameters or whether the assertion's threshold is what moved. 63c1080 ("run the scenario tests at 250 instead of 50") is the most recent commit to touch that region and is the first place I would look.

Knock-on, updated

The deployed pin is d0768f6c, now 21 commits behind main. publish needs test, so nothing has published while this is red, and the public site stays on that build until the suite goes green.

Here is the log you asked for, plus a correction: **the timing inference in this issue is wrong, and all three ranked suspects are ruled out.** I hit this from the ops side while validating galaxy-gen#74, and got at the logs through the authenticated web UI rather than the API. The API observation here is exactly right, and now confirmed from the page footer: this instance reports **Forgejo 15.0.2**, and `action-job logs`, `action-run logs`, and `action-run-job list` are all Forgejo 16 endpoints. All three 404. The browser is currently the only route to a log. ## Run 129, step by step ``` Set up job 12s actions/checkout@v4 1s Install rust toolchain 8s Rust lint gate 14s cargo test 20s <-- FAILED Alert Telegram 1s Complete job 0s ``` The job runs **every** step. It does not die at or before the toolchain install. ## The actual failure ``` panicked at src/rust/galaxy.rs:6208:9: mean coherence over the window was 0.11289115, deepest trough (0.040970046, 1061) test galaxy::tests_dynamics::test_spiral_remains_resolved_star_forming_and_coherent_for_100_ticks ... FAILED failures: galaxy::tests_dynamics::test_spiral_remains_resolved_star_forming_and_coherent_for_100_ticks test result: FAILED. 90 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.12s error: test failed, to rerun pass `--lib` exit status 101 ``` A deterministic, seeded assertion. Not infrastructure. ## Where the reasoning went wrong The premise was "a cold `cargo test` on this repo takes minutes, so 48 seconds means it died early." The suite actually finishes in **14.12s**, and rustup installs in **8s** on this runner. 48 seconds is comfortably the whole pipeline running to completion and failing at the last gate. Suspect by suspect: 1. **checkout / runner image** - ran clean in 1s 2. **egress** - `install-rust.sh` reached `sh.rustup.rs` and installed in 8s 3. **`--component` on rustup-init** - accepted, the install step succeeded `53dd125` also did more than you credited it with. It did not turn CI green, but it cleared the lint gate, which now passes in 14s. The failure moved later. ## One caveat on "run 117 to 129, same cause" I have direct evidence for run 129 only. This issue records run 128 at **17 seconds**, which is too fast for the 55-second path I observed, so run 128 very likely died somewhere earlier and for a different reason. Treat 117-129 as possibly more than one cause rather than a single thirteen-run streak. The clippy problem `53dd125` fixed was probably the earlier one. ## Independent confirmation galaxy-gen#76 runs the same suite through `ward exec test-rust` inside the dev-base container, on entirely different infrastructure, and reproduces this **byte for byte**: same `0.11289115`, same trough `(0.040970046, 1061)`, same 90 passed and 1 failed. Two unrelated environments agreeing to the digit is about as strong as "this is real product behavior, not the runner" gets. ## Handing this back to engineering The fix is a physics or threshold judgement in `galaxy.rs`, which is not mine to make from the ops side. Acceptance condition: `ward exec test-rust` green on `main`, with a deliberate decision on whether the spiral scenario should hold coherence at these parameters or whether the assertion's threshold is what moved. `63c1080` ("run the scenario tests at 250 instead of 50") is the most recent commit to touch that region and is the first place I would look. ## Knock-on, updated The deployed pin is `d0768f6c`, now **21 commits behind `main`**. `publish` needs `test`, so nothing has published while this is red, and the public site stays on that build until the suite goes green.

coilyco-ops added the

autonomy

live-collab

label

2026-08-09 22:52:48 +00:00

coilyco-ops commented

2026-08-09 23:00:46 +00:00

Author

Member

Refining one thing I said above. I wrote that the browser is the only route to a log, which is true of what ships today but understates how close a scripted route is.

agentic_os/forgejo_actions_logs.py is, per its own docstring, "through the official Forgejo 16 REST API", with no fallback path. That is why it cannot help on 15.0.2.
agentic_os/forgejo_actions_web.py is already a web UI bridge that authenticates and parses data-initial-post-response out of the Actions page. It exists and is tested, but is currently wired only to forgejo_actions_rerun.py.

So the capability to read this instance's Actions pages without a browser is half-built in aos already. Extending that bridge to job logs looks a lot cheaper than waiting for a Forgejo 16 upgrade, and it would close the "logs unreachable from the API" half of this issue permanently.

Refining one thing I said above. I wrote that the browser is the only route to a log, which is true of what ships today but understates how close a scripted route is. * `agentic_os/forgejo_actions_logs.py` is, per its own docstring, "through the official Forgejo 16 REST API", with no fallback path. That is why it cannot help on 15.0.2. * `agentic_os/forgejo_actions_web.py` is already a web UI bridge that authenticates and parses `data-initial-post-response` out of the Actions page. It exists and is tested, but is currently wired only to `forgejo_actions_rerun.py`. So the capability to read this instance's Actions pages without a browser is half-built in aos already. Extending that bridge to job logs looks a lot cheaper than waiting for a Forgejo 16 upgrade, and it would close the "logs unreachable from the API" half of this issue permanently.

coilyco-ops commented

2026-08-10 00:46:45 +00:00

Author

Member

Root cause found, and it is not a regression. The simulation is not reproducible across platforms. The test is correct and is reporting something real.

The experiment

Same commit, same pinned rust 1.90.0, one test:

aarch64-apple-darwin - passes
x86_64-apple-darwin (Rosetta) - passes
x86_64-unknown-linux-gnu (CI) - fails, mean coherence 0.11289115 against the 0.15 bar

The full suite is 91 passed / 0 failed on macOS and 90 / 1 in CI. It also passes locally at main, a5b4a67, 1bbbb10, and 63c1080, so nothing in the recent physics work caused it.

Two x86_64 builds differing only by operating system land on opposite sides of the bar. That rules out architecture, SIMD width, and FMA contraction, and leaves the system math library.

Why that is enough to explain a 25% swing

src/rust/galaxy.rs has 105 call sites into functions that IEEE-754 does not require to be correctly rounded, so every libm implements them with its own approximations and the results differ in the last bits:

28 cos, 27 sin, 13 atan2, 12 ln, 6 powf, 6 exp, 4 hypot, 1 cbrt

sqrt (73 calls), abs, floor, and round are correctly rounded and identical everywhere, so they are not involved. powi is repeated multiplication and is also fine.

AGENTS.md records that physics is stored as magnitude and degrees rather than x/y vectors, converted at computation boundaries. That puts sin, cos, and atan2 on the hot path every tick. Over the ~1000 ticks before the measurement window, a chaotic gravitational system amplifies last-bit differences into macroscopic ones. The reported trough is at tick 1061.

What this invalidates

Cargo.toml claims this cannot happen: "Rust does not enable fast-math, so f32 results are identical across opt levels and the golden hashes hold; the golden mass field test is what proves that stays true." True across opt levels, false across platforms. test_golden_mass_field_per_scenario passes in CI, so the test appointed to catch this does not catch it.
The site does not render what you develop. The image builds --platform linux/amd64, so the public galaxy is the weaker-armed one and the one on a dev Mac is a different object.
Ablation results may be platform-scoped. #70 and #72 measure coherence and vsig, and docs/ablation.md records findings from those metrics. Anything recorded from a macOS run may not describe the shipped build.

Why disabling the test is the wrong unblock

Beyond hiding the above, it buys little. The next assertion, min_coherence.0 >= 0.04, passes in CI at 0.040970046, a 2.4% margin. Silencing the mean bar leaves a test one small drift from failing "lost the arms entirely" instead.

Acceptance condition

Route the simulation's transcendentals through the pure-Rust libm crate rather than std's platform libm, so the sim is bit-identical on every target. Done: ward exec test-rust green on both macOS and linux from the same commit, and the golden hash tests become meaningful cross-platform instead of only cross-opt-level.

One caveat worth ruling out cheaply: x86_64-apple-darwin and x86_64-unknown-linux-gnu could in principle differ in default target-cpu features as well as libm. Both use the conservative x86-64 baseline, so libm is by far the likeliest remaining variable, and swapping in the libm crate confirms or refutes it in one CI run.

This is a physics and numerics change in product code, so I am leaving it to engineering rather than taking it from the ops seat.

Root cause found, and it is not a regression. **The simulation is not reproducible across platforms.** The test is correct and is reporting something real. ## The experiment Same commit, same pinned rust 1.90.0, one test: * `aarch64-apple-darwin` - **passes** * `x86_64-apple-darwin` (Rosetta) - **passes** * `x86_64-unknown-linux-gnu` (CI) - **fails**, mean coherence `0.11289115` against the 0.15 bar The full suite is 91 passed / 0 failed on macOS and 90 / 1 in CI. It also passes locally at `main`, `a5b4a67`, `1bbbb10`, and `63c1080`, so nothing in the recent physics work caused it. Two x86_64 builds differing only by operating system land on opposite sides of the bar. That rules out architecture, SIMD width, and FMA contraction, and leaves the system math library. ## Why that is enough to explain a 25% swing `src/rust/galaxy.rs` has 105 call sites into functions that IEEE-754 does **not** require to be correctly rounded, so every libm implements them with its own approximations and the results differ in the last bits: * 28 `cos`, 27 `sin`, 13 `atan2`, 12 `ln`, 6 `powf`, 6 `exp`, 4 `hypot`, 1 `cbrt` `sqrt` (73 calls), `abs`, `floor`, and `round` are correctly rounded and identical everywhere, so they are not involved. `powi` is repeated multiplication and is also fine. AGENTS.md records that physics is stored as magnitude and degrees rather than x/y vectors, converted at computation boundaries. That puts `sin`, `cos`, and `atan2` on the hot path every tick. Over the ~1000 ticks before the measurement window, a chaotic gravitational system amplifies last-bit differences into macroscopic ones. The reported trough is at tick 1061. ## What this invalidates * **`Cargo.toml` claims this cannot happen**: "Rust does not enable fast-math, so f32 results are identical across opt levels and the golden hashes hold; the golden mass field test is what proves that stays true." True across opt levels, false across platforms. `test_golden_mass_field_per_scenario` passes in CI, so the test appointed to catch this does not catch it. * **The site does not render what you develop.** The image builds `--platform linux/amd64`, so the public galaxy is the weaker-armed one and the one on a dev Mac is a different object. * **Ablation results may be platform-scoped.** #70 and #72 measure coherence and vsig, and `docs/ablation.md` records findings from those metrics. Anything recorded from a macOS run may not describe the shipped build. ## Why disabling the test is the wrong unblock Beyond hiding the above, it buys little. The next assertion, `min_coherence.0 >= 0.04`, passes in CI at `0.040970046`, a 2.4% margin. Silencing the mean bar leaves a test one small drift from failing "lost the arms entirely" instead. ## Acceptance condition Route the simulation's transcendentals through the pure-Rust [`libm`](https://crates.io/crates/libm) crate rather than std's platform libm, so the sim is bit-identical on every target. Done: `ward exec test-rust` green on both macOS and linux from the same commit, and the golden hash tests become meaningful cross-platform instead of only cross-opt-level. One caveat worth ruling out cheaply: `x86_64-apple-darwin` and `x86_64-unknown-linux-gnu` could in principle differ in default target-cpu features as well as libm. Both use the conservative x86-64 baseline, so libm is by far the likeliest remaining variable, and swapping in the `libm` crate confirms or refutes it in one CI run. This is a physics and numerics change in product code, so I am leaving it to engineering rather than taking it from the ops seat.