coilyco-gaming/galaxy-gen

Watch

main is red on a rustup download timeout, not on anything in the commit - run 150 #84

New issue

Open

opened 2026-08-14 03:11:26 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-14 03:11:26 +00:00

Member

Run 150 (01bcd3c, a docs-and-comments-only commit) failed in ward exec ci-setup after 3m12s. The publish job was skipped, so main's tip is not the deployed image - prod is still 42ad68f from run 149.

The failure

Error: `cargo metadata` exited with an error: info: syncing channel updates for 1.90.0-x86_64-unknown-linux-gnu
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml'
  to '/usr/local/rustup/tmp/bh6swia7hfgnf00__file.toml':
  error decoding response body: request or response body error: operation timed out

Nothing to do with the commit, which touched dataviz.tsx comments, three new docs, and one pyproject.toml line. All four CI steps - lint-rust, test-rust (91 passed), check-js, and the full precommit gate - pass locally on that exact tree.

Why it is structural rather than pure bad luck

rust-toolchain.toml predicted this cost in its own comment:

CI and the image build now both run in dev-base, whose own default toolchain is stable. This pin is what keeps that from mattering [...] Bumping it costs one toolchain download per build until dev-base's default catches up.

So every CI run currently fetches the 1.90.0 channel from static.rust-lang.org inside the container. That is a per-build dependency on a third-party CDN with no cache and no retry, and run 150 is what it looks like when that request times out. Run 149 passed on the identical toolchain minutes earlier, so this is a flake, but it is a flake the pipeline is structurally exposed to on every single push.

Fixes, cheapest first

Re-run 150. It should go green unmodified. workflow_dispatch is already enabled (2f224a7), so it is one manual dispatch - I could not do it from here, the POST needs write auth.
Bake 1.90.0 into the dev-base image, or move dev-base's default to it. Removes the download from the hot path entirely and is what the toolchain comment anticipates.
Retry the rustup fetch in ci-setup if neither of the above is convenient - two or three attempts with a backoff would absorb a timeout like this one.

Related: #73 covered main being red and its logs being unreachable. The logs are reachable now with an authenticated read token (/api/v1/repos/{owner}/{repo}/actions/jobs/{job_id}/logs); anonymous requests still 404, which is worth knowing for the next triage.

Run 150 (`01bcd3c`, a docs-and-comments-only commit) failed in `ward exec ci-setup` after 3m12s. The publish job was skipped, so **main's tip is not the deployed image** - prod is still `42ad68f` from run 149. ## The failure ``` Error: `cargo metadata` exited with an error: info: syncing channel updates for 1.90.0-x86_64-unknown-linux-gnu error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml' to '/usr/local/rustup/tmp/bh6swia7hfgnf00__file.toml': error decoding response body: request or response body error: operation timed out ``` Nothing to do with the commit, which touched `dataviz.tsx` comments, three new docs, and one `pyproject.toml` line. All four CI steps - `lint-rust`, `test-rust` (91 passed), `check-js`, and the full `precommit` gate - pass locally on that exact tree. ## Why it is structural rather than pure bad luck `rust-toolchain.toml` predicted this cost in its own comment: > CI and the image build now both run in dev-base, whose own default toolchain is `stable`. This pin is what keeps that from mattering [...] Bumping it costs **one toolchain download per build** until dev-base's default catches up. So every CI run currently fetches the 1.90.0 channel from `static.rust-lang.org` inside the container. That is a per-build dependency on a third-party CDN with no cache and no retry, and run 150 is what it looks like when that request times out. Run 149 passed on the identical toolchain minutes earlier, so this is a flake, but it is a flake the pipeline is structurally exposed to on every single push. ## Fixes, cheapest first 1. **Re-run 150.** It should go green unmodified. `workflow_dispatch` is already enabled (`2f224a7`), so it is one manual dispatch - I could not do it from here, the POST needs write auth. 2. **Bake 1.90.0 into the dev-base image**, or move dev-base's default to it. Removes the download from the hot path entirely and is what the toolchain comment anticipates. 3. **Retry the rustup fetch** in `ci-setup` if neither of the above is convenient - two or three attempts with a backoff would absorb a timeout like this one. Related: #73 covered main being red and its logs being unreachable. The logs are reachable now with an authenticated read token (`/api/v1/repos/{owner}/{repo}/actions/jobs/{job_id}/logs`); anonymous requests still 404, which is worth knowing for the next triage.

coilyco-ops referenced this issue from a commit

2026-08-14 05:08:01 +00:00

fix(ci): retry the pinned-toolchain fetch instead of going red on it

coilyco-ops commented

2026-08-14 05:08:26 +00:00

Author

Member

Twice in a row now, so I stopped waiting it out and added the retry

Run 151 (5ab7302, the #70 physics change) failed identically to run 150 - same step, same URL, ~3 minutes in:

error: could not download file from
  'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml.sha256':
  operation timed out

Two consecutive failures on unrelated commits is not a flake to re-run past, it is the structural exposure rust-toolchain.toml predicted. Both runs took the deploy with them - publish skipped both times - so prod is still 42ad68f while main has moved three commits past it.

Landed: fix 3 from the list above

ensure_toolchain in scripts/ward-command.sh, called from ci-setup: three attempts with a 15/30s backoff before any cargo command runs. On the last failure it returns cleanly rather than erroring, so the real command still surfaces the real error instead of this wrapper masking it. Commit 190a1b3.

This is the cheap half. Fix 2 is still the better one - baking 1.90.0 into dev-base, or moving dev-base's default to it, removes the download from the hot path entirely rather than making it survivable. That is not this repo's to make, so it wants an issue against the dev-base image.

One process note

I committed the retry once with a gate failure in it - my own new comment block was over the two-line cap - because the command chain did not stop on the failed gate. Caught, fixed, amended before push.

Root cause worth recording for anyone else working from a temp clone: pre-commit install had never been run in it, so the git hook was absent and nothing enforced the gate at commit time. Every commit this session was gated by running ward exec precommit manually, which is fine until the one time the result is not checked. The hook is installed in the clone now, and AGENTS.md's checkout-residency section is the natural place to say so if this keeps recurring.

## Twice in a row now, so I stopped waiting it out and added the retry Run 151 (`5ab7302`, the #70 physics change) failed identically to run 150 - same step, same URL, ~3 minutes in: ``` error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml.sha256': operation timed out ``` Two consecutive failures on unrelated commits is not a flake to re-run past, it is the structural exposure `rust-toolchain.toml` predicted. Both runs took the deploy with them - `publish` skipped both times - so prod is still `42ad68f` while main has moved three commits past it. ### Landed: fix 3 from the list above `ensure_toolchain` in `scripts/ward-command.sh`, called from `ci-setup`: three attempts with a 15/30s backoff before any cargo command runs. On the last failure it returns cleanly rather than erroring, so the real command still surfaces the real error instead of this wrapper masking it. Commit `190a1b3`. This is the cheap half. **Fix 2 is still the better one** - baking 1.90.0 into dev-base, or moving dev-base's default to it, removes the download from the hot path entirely rather than making it survivable. That is not this repo's to make, so it wants an issue against the dev-base image. ### One process note I committed the retry once with a gate failure in it - my own new comment block was over the two-line cap - because the command chain did not stop on the failed gate. Caught, fixed, amended before push. Root cause worth recording for anyone else working from a temp clone: **`pre-commit install` had never been run in it**, so the git hook was absent and nothing enforced the gate at commit time. Every commit this session was gated by running `ward exec precommit` manually, which is fine until the one time the result is not checked. The hook is installed in the clone now, and `AGENTS.md`'s checkout-residency section is the natural place to say so if this keeps recurring.

coilyco-ops referenced this issue from a commit

2026-08-14 05:47:24 +00:00

Revert "fix(ci): retry the pinned-toolchain fetch instead of going red on it"

coilyco-ops commented

2026-08-14 05:47:46 +00:00

Author

Member

The retry does not work, and the measurement says why. Backed out, and this is an Ops handoff.

Run 155 ran my retry and logged every attempt:

05:34:04  toolchain fetch attempt 1 failed; retrying
05:37:19  toolchain fetch attempt 2 failed; retrying
05:40:49  toolchain fetch attempt 3 failed; retrying
05:44:41  Error: could not download file from
          'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml':
          operation timed out

Three attempts, ten minutes, all timed out. The retry turned a 3-minute failure into a 10-minute one and fixed nothing. Reverted in 1549f5d - a measured-useless fix should not sit in the tree.

The diagnosis was wrong, and the correct one is narrower

I called this "a per-build dependency on a third-party CDN with no cache and no retry", which implied an occasionally slow CDN. That is not what is happening:

vantage point	result
outside the cluster, just now	HTTP 200 in 0.157s
inside the runner container, 3 attempts over 10 min	all timed out

static.rust-lang.org is healthy. The runner cannot reach it. That is an egress-path problem on the k3s runner - DNS, NAT, MTU, a proxy, or upstream throttling of the cluster's address - and no in-repo change can retry its way out of a network that does not complete a TLS body transfer.

Worth noting the failure mode is specifically "error decoding response body" / "operation timed out" after connection, not a connection refusal - so something is completing a handshake and then stalling the transfer. That smells like MTU/PMTU or a middlebox rather than DNS.

What is not the fix

Retrying. Measured, above.
Unpinning the toolchain. rust-toolchain.toml exists because a floating channel broke main on the same commit that added the clippy gate. Removing the pin trades a hard failure for a silent one.

What is the fix, and it needs Ops

Bake 1.90.0 into the dev-base image, or move dev-base's default toolchain to it. That removes the download from the hot path entirely, which is what rust-toolchain.toml's own comment anticipated:

Bumping it costs one toolchain download per build until dev-base's default catches up.

Expected evidence that it worked: ward exec ci-setup completes without a syncing channel updates line in the log.

Failing that, or alongside it: diagnose runner egress to static.rust-lang.org from inside the container. curl -v --max-time 30 https://static.rust-lang.org/dist/channel-rust-1.90.0.toml from a shell in the runner pod would separate DNS from handshake from body transfer in one command.

Impact while this stands

Five consecutive runs have not produced a usable signal - two toolchain timeouts, two supersessions from unrelated bot merges, one more timeout. publish has not run since 42ad68f, so prod is five commits behind main, including the #70 physics change and the #85 doc work. Nothing is broken on the site; it is simply not receiving what has landed.

Runner health is not the issue - pods are Running on kai-server with jobs Succeeded 3m/18m/33m before I checked, so work flows fine when it does not need that download.

## The retry does not work, and the measurement says why. Backed out, and this is an Ops handoff. Run 155 ran my retry and logged every attempt: ``` 05:34:04 toolchain fetch attempt 1 failed; retrying 05:37:19 toolchain fetch attempt 2 failed; retrying 05:40:49 toolchain fetch attempt 3 failed; retrying 05:44:41 Error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml': operation timed out ``` Three attempts, ten minutes, all timed out. **The retry turned a 3-minute failure into a 10-minute one and fixed nothing.** Reverted in `1549f5d` - a measured-useless fix should not sit in the tree. ## The diagnosis was wrong, and the correct one is narrower I called this "a per-build dependency on a third-party CDN with no cache and no retry", which implied an occasionally slow CDN. That is not what is happening: | vantage point | result | |---|---| | outside the cluster, just now | **HTTP 200 in 0.157s** | | inside the runner container, 3 attempts over 10 min | **all timed out** | `static.rust-lang.org` is healthy. **The runner cannot reach it.** That is an egress-path problem on the k3s runner - DNS, NAT, MTU, a proxy, or upstream throttling of the cluster's address - and no in-repo change can retry its way out of a network that does not complete a TLS body transfer. Worth noting the failure mode is specifically "error decoding response body" / "operation timed out" *after* connection, not a connection refusal - so something is completing a handshake and then stalling the transfer. That smells like MTU/PMTU or a middlebox rather than DNS. ## What is not the fix - **Retrying.** Measured, above. - **Unpinning the toolchain.** `rust-toolchain.toml` exists because a floating channel broke main on the same commit that added the clippy gate. Removing the pin trades a hard failure for a silent one. ## What is the fix, and it needs Ops **Bake 1.90.0 into the dev-base image**, or move dev-base's default toolchain to it. That removes the download from the hot path entirely, which is what `rust-toolchain.toml`'s own comment anticipated: > Bumping it costs one toolchain download per build **until dev-base's default catches up**. Expected evidence that it worked: `ward exec ci-setup` completes without a `syncing channel updates` line in the log. **Failing that, or alongside it:** diagnose runner egress to `static.rust-lang.org` from inside the container. `curl -v --max-time 30 https://static.rust-lang.org/dist/channel-rust-1.90.0.toml` from a shell in the runner pod would separate DNS from handshake from body transfer in one command. ## Impact while this stands Five consecutive runs have not produced a usable signal - two toolchain timeouts, two supersessions from unrelated bot merges, one more timeout. `publish` has not run since `42ad68f`, so **prod is five commits behind main**, including the #70 physics change and the #85 doc work. Nothing is broken on the site; it is simply not receiving what has landed. Runner health is not the issue - pods are Running on kai-server with jobs Succeeded 3m/18m/33m before I checked, so work flows fine when it does not need that download.

coilyco-ops referenced this issue from a commit

2026-08-14 06:09:04 +00:00

feat(dev-base): bake the Rust toolchains consumers pin

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-08-14 06:10:07 +00:00

feat(dev-base): bake the Rust toolchains consumers pin #1058

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-08-14 07:06:53 +00:00

dev-base release blocked: publish-lang-rust builds but cannot push, and kai-server root is at 87% critical #1060

coilyco-ops added the

role/engineer

priority