feat(dev-base): bake the Rust toolchains consumers pin #1058

Merged

coilyco-ops merged 1 commit from ops/dev-base-pin-rust-1.90.0 into main

2026-08-14 06:41:55 +00:00

coilyco-ops commented

2026-08-14 06:10:07 +00:00

Member

The problem

A repo pinning a channel in rust-toolchain.toml downloads it from static.rust-lang.org on the first cargo call of every CI run. That request has no cache, and when the runner's egress to that host degrades, every Rust repo goes red on a network fault rather than on its own code.

galaxy-gen#84 is the worked example. Five consecutive runs produced no usable signal, and publish has not run since, so that site is several commits behind its own main.

vantage point	result
outside the cluster	HTTP 200 in 0.157s
inside the runner, 3 attempts over 10 min	all timed out

Two fixes were tried there and rejected on evidence. Retrying was implemented, measured, and reverted - it turned a 3-minute failure into a 10-minute one and fixed nothing, because no retry repairs a path that will not complete a body transfer. Unpinning trades a loud failure for a silent one; that pin exists because a floating channel broke galaxy-gen's main on the same commit that added its clippy gate.

The change

RUST_PINNED_VERSIONS, a space-separated list declared once in docker-bake.hcl and threaded to both Dockerfiles. stable remains the default toolchain, so nothing changes for consumers that do not pin.

Each baked toolchain additionally gets clippy, rustfmt, and wasm32-unknown-unknown.

The components are the point. galaxy-gen pins components = ["clippy", "rustfmt"], and baking a bare toolchain would leave rustup fetching those at first use - the same download in a different place. This is the detail that decides whether the fix works at all.

Verified by running it, not reading it

Built the payload and executed the toolchain inside it:

default: cargo 1.97.1 (c980f4866 2026-06-30)
pinned:  cargo 1.90.0 (840b83a10 2025-07-30)
clippy:  clippy 0.1.90 (1159e78c47 2025-09-14)
fmt:     rustfmt 1.8.0-stable (1159e78c47 2025-09-14)
targets: aarch64-unknown-linux-gnu wasm32-unknown-unknown

Worth recording: the first attempt failed to build. rustup toolchain install X --profile minimal --component clippy rustfmt is invalid - the [TOOLCHAIN]... argument is variadic and swallows rustfmt, giving invalid toolchain name: 'rustfmt'. Each component needs its own flag. A static check would not have caught that; only the build did.

ward exec dev-base-check passes with no warnings.

Guard against a silent pass

The full image verifies each pin end to end - cargo +PIN --version, clippy --version, fmt --version, and the wasm32 target - and fails when the list arrives empty. The bake file passes args to full explicitly, so an unplumbed RUST_PINNED_VERSIONS would otherwise make the loop iterate zero times and pass. Confirmed both targets resolve it via --print.

Cost and scope

One extra toolchain in the Rust payload: ~150s on a cold build, cached thereafter. Adding a future pin is a one-line edit to the default. Consumers keep owning their own rust-toolchain.toml; this only decides what is already resident when they ask.

Refs coilyco-gaming/galaxy-gen#84

## The problem A repo pinning a channel in `rust-toolchain.toml` downloads it from `static.rust-lang.org` on the first cargo call of **every** CI run. That request has no cache, and when the runner's egress to that host degrades, every Rust repo goes red on a network fault rather than on its own code. [galaxy-gen#84](https://forgejo.coilysiren.me/coilyco-gaming/galaxy-gen/issues/84) is the worked example. Five consecutive runs produced no usable signal, and `publish` has not run since, so that site is several commits behind its own main. | vantage point | result | |---|---| | outside the cluster | **HTTP 200 in 0.157s** | | inside the runner, 3 attempts over 10 min | **all timed out** | Two fixes were tried there and rejected on evidence. **Retrying** was implemented, measured, and reverted - it turned a 3-minute failure into a 10-minute one and fixed nothing, because no retry repairs a path that will not complete a body transfer. **Unpinning** trades a loud failure for a silent one; that pin exists because a floating channel broke galaxy-gen's main on the same commit that added its clippy gate. ## The change `RUST_PINNED_VERSIONS`, a space-separated list declared once in `docker-bake.hcl` and threaded to both Dockerfiles. `stable` remains the default toolchain, so **nothing changes for consumers that do not pin**. Each baked toolchain additionally gets `clippy`, `rustfmt`, and `wasm32-unknown-unknown`. **The components are the point.** galaxy-gen pins `components = ["clippy", "rustfmt"]`, and baking a bare toolchain would leave rustup fetching those at first use - the same download in a different place. This is the detail that decides whether the fix works at all. ## Verified by running it, not reading it Built the payload and executed the toolchain inside it: ``` default: cargo 1.97.1 (c980f4866 2026-06-30) pinned: cargo 1.90.0 (840b83a10 2025-07-30) clippy: clippy 0.1.90 (1159e78c47 2025-09-14) fmt: rustfmt 1.8.0-stable (1159e78c47 2025-09-14) targets: aarch64-unknown-linux-gnu wasm32-unknown-unknown ``` Worth recording: the **first attempt failed to build**. `rustup toolchain install X --profile minimal --component clippy rustfmt` is invalid - the `[TOOLCHAIN]...` argument is variadic and swallows `rustfmt`, giving `invalid toolchain name: 'rustfmt'`. Each component needs its own flag. A static check would not have caught that; only the build did. `ward exec dev-base-check` passes with no warnings. ## Guard against a silent pass The full image verifies each pin end to end - `cargo +PIN --version`, `clippy --version`, `fmt --version`, and the wasm32 target - and **fails when the list arrives empty**. The bake file passes args to `full` explicitly, so an unplumbed `RUST_PINNED_VERSIONS` would otherwise make the loop iterate zero times and pass. Confirmed both targets resolve it via `--print`. ## Cost and scope One extra toolchain in the Rust payload: ~150s on a cold build, cached thereafter. Adding a future pin is a one-line edit to the default. Consumers keep owning their own `rust-toolchain.toml`; this only decides what is already resident when they ask. Refs coilyco-gaming/galaxy-gen#84

coilyco-ops added 1 commit

2026-08-14 06:10:07 +00:00

feat(dev-base): bake the Rust toolchains consumers pin

ci / aos-cli-tests (pull_request) Successful in 40s

Details

ci / ward-doctor (pull_request) Successful in 52s

Details

ci / gate (pull_request) Successful in 1m19s

Details

dev-base-pr / build (pull_request) Successful in 18m17s

Details

276fb51800

A repo pinning a channel in rust-toolchain.toml downloads it from
static.rust-lang.org on the first cargo call of every CI run. The request
has no cache, and when the runner's egress to that host degrades every
Rust repo goes red on a network fault rather than on its own code.

galaxy-gen#84 is the worked example: three retries over ten minutes all
timed out from inside the runner while the same URL answered in 0.157s
from outside. Retrying cannot fix a path that does not complete a body
transfer, and unpinning trades a loud failure for a silent one - the pin
exists because a floating channel broke main once already.

RUST_PINNED_VERSIONS is a space-separated list, declared once in
docker-bake.hcl and threaded to both Dockerfiles. stable stays the
default, so nothing changes for consumers that do not pin.

Each baked toolchain also gets clippy, rustfmt, and wasm32-unknown-unknown.
The components are the point: galaxy-gen pins components clippy and
rustfmt, and a bare toolchain leaves rustup fetching those at first use,
which is the same download in a different place.

Verified by building the payload and running it, not by reading it:
default cargo 1.97.1, pinned cargo 1.90.0, clippy 0.1.90, rustfmt 1.8.0,
and the wasm32 target present.

The full image now asserts the same per pin and fails when the list
arrives empty, so an unplumbed build arg cannot make that check a silent
pass. ward exec dev-base-check is clean.

Refs coilyco-gaming/galaxy-gen#84

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-14 06:10:07 +00:00

coilyco-ops merged commit 864e3a08ba into main

2026-08-14 06:41:55 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-14 06:41:57 +00:00

feat(dev-base): bake the Rust toolchains consumers pin (#1058)

coilyco-ops referenced this pull request

2026-08-14 07:06:53 +00:00

dev-base release blocked: publish-lang-rust builds but cannot push, and kai-server root is at 87% critical #1060

coilyco-ops referenced this pull request from a commit

2026-08-19 04:08:33 +00:00