galaxy-gen CI has been red since 03:04 today: rustup downloads time out on the gaming runner, the #783 symptom class is back #824

New issue

Open

opened 2026-08-14 05:10:02 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-14 05:10:02 +00:00

Member

coilyco-gaming/galaxy-gen cannot build. Every run since 2026-08-14T03:04 fails on a large HTTPS fetch that never completes, which is the exact "TLS handshake completes then the response disappears" shape #783 described.

The failure

Identical on main and on the open PR:

Error: `cargo metadata` exited with an error: info: syncing channel updates for 1.90.0-x86_64-unknown-linux-gnu
error: could not download file from
'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml.sha256'
to '/usr/local/rustup/tmp/dx14s70cwn2fhuo7_file': error downloading file:
error sending request for url (...): operation timed out

main - run index 151 (id 19795), job test (id 32835), failed 2026-08-14T05:03:54
PR #81 - run index 148 (id 19657), job gate (id 32525), failed 2026-08-14T04:06:00
last green - run index 149 (id 19669), 2026-08-14T02:57. Run 150 at 03:04 is the first red.

Both jobs ran on the same runner, 625b878f-8f28-44d0-ac30-9cdcfec21d03 (v12.13.2), in agentic-os:release.

Why this is not the PR's fault

galaxy-gen#81 removes five retired Backstage keys from .ward/ward.yaml and nothing else. It cannot cause a rustup download timeout, and the same failure is on main with the PR unmerged. The PR is blocked on this, not on its own content.

What points at MTU

#783 landed --mtu=1230 on the base runner StatefulSets and was closed 2026-08-12T00:13:43 with no verification comment. Its own verification step (docker -H tcp://<runner-pod-ip>:2375 network inspect bridge reports 1230, and a large HTTPS fetch completes without the proxy) does not appear to have been recorded anywhere.

Reading the manifests on main today:

deploy/forgejo-runner.yml, -build, -deploy, -publisher - all carry --mtu=1230, matching cni0
deploy/forgejo-runners-ser8/general-gaming/kustomization.yaml - overrides to --mtu=1450
the other three ser8 overlays (general-bridge, general-flight-deck, general-flight-deck-canary) also override to 1450

If the ser8 pod network is at or below 1230 like kai-server's, then 1450 oversizes every DinD bridge on ser8 and reproduces the original defect precisely. #783 measured cni0 on kai-server only. ser8 was never measured.

What settles it

Two reads, neither of which I could take:

Which node runner 625b878f-8f28-44d0-ac30-9cdcfec21d03 sits on.
cat /sys/class/net/cni0/mtu on ser8, against the 1450 the overlay sets. The node-stats MCP refuses /sys/class/net/... as outside its readable-root allowlist, so this needs a shell on the node.

If ser8's cni0 really is 1450 or higher, the overlay is correct and the cause is elsewhere. That is the first thing to check, because it is cheap and it splits the tree.

#783 - closed, and this is its symptom class returning two days later on a runner whose overlay opts out of the 1230 it set
#784 - still open. The failing job's environment shows no proxy variables at all, and the scoped gaming overlay carries no FORGEJO_EGRESS_PROXY, so the reliability route #784 would make always-on is not covering this runner
#788 - named galaxy-gen explicitly as one of roughly twenty repositories whose CI takes the same DinD path and was never measured. It is measured now, and it is broken. That is one unknown converted into a known failure

Impact

galaxy-gen main is red and its deploy path is blocked. galaxy-gen#81 cannot be merged against a green signal. Any other repository pinned to a ser8 general runner and fetching a large payload over HTTPS is exposed to the same thing, which is the #788 audit that has not run yet.

Re-running the jobs is not available as a workaround: POST /actions/runs/{id}/rerun and /rerun-failed-jobs both return a router-level 404 on this Forgejo, with the run index and the internal ActionRun id alike.

Filed from a fleet-wide open-PR review that traced galaxy-gen#81's red gate to its cause.

`coilyco-gaming/galaxy-gen` cannot build. Every run since 2026-08-14T03:04 fails on a large HTTPS fetch that never completes, which is the exact "TLS handshake completes then the response disappears" shape #783 described. ## The failure Identical on `main` and on the open PR: ``` Error: `cargo metadata` exited with an error: info: syncing channel updates for 1.90.0-x86_64-unknown-linux-gnu error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.90.0.toml.sha256' to '/usr/local/rustup/tmp/dx14s70cwn2fhuo7_file': error downloading file: error sending request for url (...): operation timed out ``` * `main` - run index 151 (id 19795), job `test` (id 32835), failed 2026-08-14T05:03:54 * PR #81 - run index 148 (id 19657), job `gate` (id 32525), failed 2026-08-14T04:06:00 * last green - run index 149 (id 19669), 2026-08-14T02:57. Run 150 at 03:04 is the first red. Both jobs ran on the same runner, `625b878f-8f28-44d0-ac30-9cdcfec21d03` (v12.13.2), in `agentic-os:release`. ## Why this is not the PR's fault galaxy-gen#81 removes five retired Backstage keys from `.ward/ward.yaml` and nothing else. It cannot cause a rustup download timeout, and the same failure is on `main` with the PR unmerged. The PR is blocked on this, not on its own content. ## What points at MTU #783 landed `--mtu=1230` on the base runner StatefulSets and was closed 2026-08-12T00:13:43 with no verification comment. Its own verification step (`docker -H tcp://<runner-pod-ip>:2375 network inspect bridge` reports 1230, and a large HTTPS fetch completes without the proxy) does not appear to have been recorded anywhere. Reading the manifests on `main` today: * `deploy/forgejo-runner.yml`, `-build`, `-deploy`, `-publisher` - all carry `--mtu=1230`, matching `cni0` * `deploy/forgejo-runners-ser8/general-gaming/kustomization.yaml` - overrides to `--mtu=1450` * the other three ser8 overlays (`general-bridge`, `general-flight-deck`, `general-flight-deck-canary`) also override to `1450` If the ser8 pod network is at or below 1230 like kai-server's, then 1450 oversizes every DinD bridge on ser8 and reproduces the original defect precisely. #783 measured `cni0` on kai-server only. ser8 was never measured. ## What settles it Two reads, neither of which I could take: 1. Which node runner `625b878f-8f28-44d0-ac30-9cdcfec21d03` sits on. 2. `cat /sys/class/net/cni0/mtu` on ser8, against the `1450` the overlay sets. The node-stats MCP refuses `/sys/class/net/...` as outside its readable-root allowlist, so this needs a shell on the node. If ser8's `cni0` really is 1450 or higher, the overlay is correct and the cause is elsewhere. That is the first thing to check, because it is cheap and it splits the tree. ## Related, and what this adds * #783 - closed, and this is its symptom class returning two days later on a runner whose overlay opts out of the 1230 it set * #784 - still open. The failing job's environment shows no proxy variables at all, and the scoped gaming overlay carries no `FORGEJO_EGRESS_PROXY`, so the reliability route #784 would make always-on is not covering this runner * #788 - named galaxy-gen explicitly as one of roughly twenty repositories whose CI takes the same DinD path and was never measured. It is measured now, and it is broken. That is one unknown converted into a known failure ## Impact galaxy-gen `main` is red and its deploy path is blocked. galaxy-gen#81 cannot be merged against a green signal. Any other repository pinned to a ser8 general runner and fetching a large payload over HTTPS is exposed to the same thing, which is the #788 audit that has not run yet. Re-running the jobs is not available as a workaround: `POST /actions/runs/{id}/rerun` and `/rerun-failed-jobs` both return a router-level 404 on this Forgejo, with the run index and the internal ActionRun id alike. Filed from a fleet-wide open-PR review that traced galaxy-gen#81's red gate to its cause.

coilyco-ops referenced this issue from coilyco-gaming/galaxy-gen

2026-08-14 05:10:14 +00:00

chore(ward): drop retired Backstage catalog keys #81

coilyco-ops referenced this issue

2026-08-14 06:26:02 +00:00

Runner 625b878f has no reachable Docker daemon, and it is the same runner behind #824 #829