coilyco-flight-deck/infrastructure

Watch

Forgejo runner egress to external hosts is degraded, and it is currently blocking dev-base publication #987

New issue

Open

opened 2026-08-29 17:14:08 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-29 17:14:08 +00:00

Owner

Four distinct failures today, four different hosts, one shape: a fetch from inside a Forgejo Actions runner stalls or is reset, while the same URL answers immediately from a laptop.

Observations

download.docker.com - agentic-os run 4439, jobs 47145/47146/47147/47148 (go, node, python, rust payloads), all four identical:
```
curl: (35) Recv failure: Connection reset by peer   (x6, all retries)
```
From this laptop the same index answered 200 in 0.11s. This one is live and blocking: it fails install-docker.sh before buildx exists, so every language payload dies and publish-full, publish-release, and assert-release-moved stay blocked. dev-base cannot publish while this persists.
github.com releases - galaxy-gen runs 157, 158, 159, 160, 163, 164, 165. wasm-pack's wasm-bindgen download stalls with no timeout. Run 164 spent 6.76s compiling and then roughly 29 minutes on that one fetch before the runner killed it at its 30m budget.
ghcr.io - lunch-money-k8s run 75, job 47078:
```
Head "https://ghcr.io/v2/astral-sh/uv/manifests/latest": net/http: TLS handshake timeout
```
Succeeded on retry twenty minutes later with no change, so that one was intermittent.
static.rust-lang.org - galaxy-gen run 155: rustup channel sync for 1.90.0 timed out. This is galaxy-gen#84 recurring, and the reason RUST_PINNED_VERSIONS exists.

Why this is filed here rather than fixed

Each instance has been absorbed repo-side by baking the artifact into dev-base so the fetch stops happening: pinned rust toolchains (galaxy-gen#84), and wasm-bindgen today (agentic-os PR #1420). That mitigation is real and worth keeping, but it treats each host separately and cannot cover a fetch dev-base itself performs, which is exactly case 1.

The runner's own egress is the shared cause, and it sits on live infrastructure rather than in a repository. I did not touch it.

Worth looking at

Whether the runner leaves via a proxy, NAT, or egress policy that resets long or large transfers. Case 1 resets on a small HTML index, cases 2 and 4 stall on large transfers, so it does not look purely size-related.
Whether a pull-through registry or artifact mirror inside the cluster is the general answer rather than per-tool pinning. That would cover the dev-base build path too.
install-docker.sh scrapes the docker index and takes the newest version, so the publisher also depends on that host being reachable and on whatever version it serves that day. Pinning it would remove one of the two fetches even without fixing egress.

Immediate state

agentic-os run 4439 failed. I re-dispatched the publish as run 4440 using the resumable path. If egress is still bad it will fail the same way, and the moving :release tag stays on the previous image.

Found while fixing CI in coilyco-flight-deck/lunch-money-k8s and coilyco-gaming/galaxy-gen (galaxy-gen#89).

Four distinct failures today, four different hosts, one shape: a fetch from inside a Forgejo Actions runner stalls or is reset, while the same URL answers immediately from a laptop. ## Observations 1. **download.docker.com** - agentic-os run 4439, jobs 47145/47146/47147/47148 (go, node, python, rust payloads), all four identical: curl: (35) Recv failure: Connection reset by peer (x6, all retries) From this laptop the same index answered **200 in 0.11s**. This one is live and blocking: it fails `install-docker.sh` before buildx exists, so every language payload dies and `publish-full`, `publish-release`, and `assert-release-moved` stay blocked. **dev-base cannot publish while this persists.** 2. **github.com releases** - galaxy-gen runs 157, 158, 159, 160, 163, 164, 165. wasm-pack's wasm-bindgen download stalls with no timeout. Run 164 spent 6.76s compiling and then roughly 29 minutes on that one fetch before the runner killed it at its 30m budget. 3. **ghcr.io** - lunch-money-k8s run 75, job 47078: Head "https://ghcr.io/v2/astral-sh/uv/manifests/latest": net/http: TLS handshake timeout Succeeded on retry twenty minutes later with no change, so that one was intermittent. 4. **static.rust-lang.org** - galaxy-gen run 155: rustup channel sync for 1.90.0 timed out. This is galaxy-gen#84 recurring, and the reason `RUST_PINNED_VERSIONS` exists. ## Why this is filed here rather than fixed Each instance has been absorbed repo-side by baking the artifact into dev-base so the fetch stops happening: pinned rust toolchains (galaxy-gen#84), and wasm-bindgen today (agentic-os PR #1420). That mitigation is real and worth keeping, but it treats each host separately and cannot cover a fetch dev-base itself performs, which is exactly case 1. The runner's own egress is the shared cause, and it sits on live infrastructure rather than in a repository. I did not touch it. ## Worth looking at - Whether the runner leaves via a proxy, NAT, or egress policy that resets long or large transfers. Case 1 resets on a small HTML index, cases 2 and 4 stall on large transfers, so it does not look purely size-related. - Whether a pull-through registry or artifact mirror inside the cluster is the general answer rather than per-tool pinning. That would cover the dev-base build path too. - `install-docker.sh` scrapes the docker index and takes the newest version, so the publisher also depends on that host being reachable and on whatever version it serves that day. Pinning it would remove one of the two fetches even without fixing egress. ## Immediate state agentic-os run 4439 failed. I re-dispatched the publish as run 4440 using the resumable path. If egress is still bad it will fail the same way, and the moving `:release` tag stays on the previous image. Found while fixing CI in coilyco-flight-deck/lunch-money-k8s and coilyco-gaming/galaxy-gen (galaxy-gen#89).

coilyco-ops added the

autonomy

headless

priority

labels

2026-08-29 17:14:08 +00:00

coilyco-ops commented

2026-08-29 17:53:01 +00:00

Author

Owner

Three consecutive publish attempts on the same commit, none completing. Egress is not a brief blip.

Run 4439 - go, node, python, rust all failed. 0 of 4 payloads published.
Run 4440 - dotnet, go, node, rust succeeded; python failed. publish-full, retag-full, and release skipped, so the moving :release tag did not move.
Run 4441 - dotnet and node failed, having succeeded in 4440.

Two things this makes clear that the original report did not.

It is a per-job coin flip, not a converging retry. Every payload job runs install-docker.sh before it can do anything, so each job draws independently against the degraded egress. Run 4441 did worse than 4440 on the same inputs. With five payload jobs each needing that download, the odds of a clean sweep stay low while this persists.

The resume did not skip completed work. docs/dev-base-image.md describes the graph as artifact-resumable, checking the registry manifest first and skipping payloads whose checkpoint exists. Run 4441 re-ran every payload rather than skipping the four that published in 4440. Plausible reading is that the manifest check itself needs the Docker CLI that install-docker.sh is failing to install, so the skip path cannot be reached when this fails - the resume is gated behind the exact thing that is broken. Worth confirming, because if so, the resumability the design relies on is unavailable in precisely the failure it exists for.

Consequence: :release still points at the pre-#1420 image. coilyco-gaming/galaxy-gen therefore still stalls on its wasm-bindgen download (galaxy-gen#89), even though the fix is merged and promoted.

I have stopped dispatching. Further retries are noise against a degraded network rather than diagnosis.

Three consecutive publish attempts on the same commit, none completing. Egress is not a brief blip. - Run 4439 - go, node, python, rust all failed. 0 of 4 payloads published. - Run 4440 - dotnet, go, node, rust succeeded; python failed. `publish-full`, `retag-full`, and `release` skipped, so the moving `:release` tag did **not** move. - Run 4441 - dotnet and node failed, having succeeded in 4440. Two things this makes clear that the original report did not. **It is a per-job coin flip, not a converging retry.** Every payload job runs `install-docker.sh` before it can do anything, so each job draws independently against the degraded egress. Run 4441 did worse than 4440 on the same inputs. With five payload jobs each needing that download, the odds of a clean sweep stay low while this persists. **The resume did not skip completed work.** `docs/dev-base-image.md` describes the graph as artifact-resumable, checking the registry manifest first and skipping payloads whose checkpoint exists. Run 4441 re-ran every payload rather than skipping the four that published in 4440. Plausible reading is that the manifest check itself needs the Docker CLI that `install-docker.sh` is failing to install, so the skip path cannot be reached when this fails - the resume is gated behind the exact thing that is broken. Worth confirming, because if so, the resumability the design relies on is unavailable in precisely the failure it exists for. Consequence: `:release` still points at the pre-#1420 image. coilyco-gaming/galaxy-gen therefore still stalls on its wasm-bindgen download (galaxy-gen#89), even though the fix is merged and promoted. I have stopped dispatching. Further retries are noise against a degraded network rather than diagnosis.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#987

No description provided.

Rows
Columns