Runner egress refuses github.com and archive.ubuntu.com in 2-11ms, blocking every dev-base publish #929

Open
opened 2026-08-25 10:57:56 +00:00 by coilyco-ops · 4 comments
Owner

The dev-base publish graph cannot complete. Three attempts this session, each dying on a different external host, all with the same signature: connection refused in single-digit to low-double-digit milliseconds, not a timeout.

Evidence

run job host failure
3879 publish-lang-dotnet github.com:443 Failed to connect ... after 11 ms
3882 publish-full archive.ubuntu.com:80 Connection refused (v4) + Network is unreachable (v6)
3882 publish-full proxy.golang.org go install ./... return code 1, all go: downloading
3885 publish-lang-dotnet github.com:443 Failed to connect ... after 2 ms

A 2ms refusal is not a network being slow or a remote being down. Something local is answering immediately and saying no: an egress proxy that is down or has an allowlist that does not cover these hosts, or a DNS answer pointing at a closed port.

Why it is not simply "the internet was flaky"

Run 3882's five language payload jobs succeeded at ~10:40, and those jobs pull the Go toolchain from go.dev, Node from nodejs.org, the dotnet installer from dot.net, and rustup from sh.rustup.rs. Seven minutes later, in the same run, publish-full could not reach archive.ubuntu.com. Then at 10:56 the same payload jobs that had just succeeded failed on github.com.

So egress works for some hosts, some of the time, and refuses others instantly. That reads as partial or intermittent proxying rather than an outage.

Note also tests/test_ci_egress_proxy.py exists in agentic-os, so there is a proxy in this path by design. This issue is about whether it is healthy and whether its allowlist covers github.com (needed for actions/checkout@v6), archive.ubuntu.com, security.ubuntu.com, and the Go module proxy.

What it is blocking

coilyco-flight-deck/agentic-os#1233 merged, moving the dev-base pin to agent-compose 2.47.0 so the image stops shipping a pre-reflow Core Roster. The release branch advanced to e2c3ea03, but the image never rebuilt, so :release still carries 2.32.0.

assert-release-moved correctly failed on the first attempt and caught the mismatch, which is the pipeline working. But it leaves branch and image disagreeing, and every consumer that composes in :release still bakes the old roster.

Downstream, coilyco-gaming/sirens-echo#1151 is the migration that needs the new image and cannot go green until it lands. coilyco-gaming/sirens-echo#1147, #1148, #1149 and #1150 sit behind that.

Also worth checking

I dispatched dev-base-publish.yml with inputs.tier=full to resume only the failed tier, via --body-file since the aosguard verb has no inputs flag. The language payload jobs ran anyway. Either the input did not reach the workflow or the resume path does not skip them, and it matters because it turns a narrow retry into a full graph rebuild against the exact network that is failing.

What I did not do

Stopped after three dispatches rather than looping. Nothing here is fixable from the repository side: the pin is correct and merged, and the build is refused before it can use it. Filed rather than retried.

The dev-base publish graph cannot complete. Three attempts this session, each dying on a different external host, all with the same signature: **connection refused in single-digit to low-double-digit milliseconds**, not a timeout. ## Evidence | run | job | host | failure | | --- | --- | --- | --- | | 3879 | publish-lang-dotnet | github.com:443 | `Failed to connect ... after 11 ms` | | 3882 | publish-full | archive.ubuntu.com:80 | `Connection refused` (v4) + `Network is unreachable` (v6) | | 3882 | publish-full | proxy.golang.org | `go install ./...` return code 1, all `go: downloading` | | 3885 | publish-lang-dotnet | github.com:443 | `Failed to connect ... after 2 ms` | A 2ms refusal is not a network being slow or a remote being down. Something local is answering immediately and saying no: an egress proxy that is down or has an allowlist that does not cover these hosts, or a DNS answer pointing at a closed port. ## Why it is not simply "the internet was flaky" Run 3882's five language payload jobs **succeeded** at ~10:40, and those jobs pull the Go toolchain from `go.dev`, Node from `nodejs.org`, the dotnet installer from `dot.net`, and rustup from `sh.rustup.rs`. Seven minutes later, in the same run, `publish-full` could not reach `archive.ubuntu.com`. Then at 10:56 the same payload jobs that had just succeeded failed on `github.com`. So egress works for some hosts, some of the time, and refuses others instantly. That reads as partial or intermittent proxying rather than an outage. Note also `tests/test_ci_egress_proxy.py` exists in agentic-os, so there is a proxy in this path by design. This issue is about whether it is healthy and whether its allowlist covers `github.com` (needed for `actions/checkout@v6`), `archive.ubuntu.com`, `security.ubuntu.com`, and the Go module proxy. ## What it is blocking `coilyco-flight-deck/agentic-os#1233` merged, moving the dev-base pin to agent-compose 2.47.0 so the image stops shipping a pre-reflow Core Roster. The `release` **branch** advanced to `e2c3ea03`, but the **image** never rebuilt, so `:release` still carries 2.32.0. `assert-release-moved` correctly failed on the first attempt and caught the mismatch, which is the pipeline working. But it leaves branch and image disagreeing, and every consumer that composes in `:release` still bakes the old roster. Downstream, `coilyco-gaming/sirens-echo#1151` is the migration that needs the new image and cannot go green until it lands. `coilyco-gaming/sirens-echo#1147`, `#1148`, `#1149` and `#1150` sit behind that. ## Also worth checking I dispatched `dev-base-publish.yml` with `inputs.tier=full` to resume only the failed tier, via `--body-file` since the aosguard verb has no inputs flag. The language payload jobs ran anyway. Either the input did not reach the workflow or the resume path does not skip them, and it matters because it turns a narrow retry into a full graph rebuild against the exact network that is failing. ## What I did not do Stopped after three dispatches rather than looping. Nothing here is fixable from the repository side: the pin is correct and merged, and the build is refused before it can use it. Filed rather than retried.
Author
Owner

Escalating. I filed this as "publishes are failing"; it is now agentic-os main is red and its release train is stalled, and a second stranded pin has piled up behind the same rebuild.

The second symptom looks unrelated and is not

Run 3888, gate job, on main: every test in tests/test_aosguard.py errors with

subprocess.CalledProcessError: Command '['specgen', '--project-root',
  '.../.specgen/guardfiles', 'build', '--out', '.../aosguard']'
  returned non-zero exit status 1

No network in it, so it reads as an application bug. It is not. b5fbf5ee (#1234) correctly moved SPECGEN_VERSION in the Dockerfile and .specgen/guardfiles/specverb.lock in the same commit, which is the right way to author that change. But the image was never rebuilt with the new pin, so CI's :release container still carries the old specgen and cannot build the new lock.

Reproduced locally to be sure rather than inferring it: specgen version v0.167.0 (umbra ref v0.167.0) builds that exact lock with rc=0. The binary in the image is what differs.

Two pins now stranded behind one rebuild

pin moved by landed
AGENT_COMPOSE_VERSION=2.47.0 #1233 e2c3ea03
SPECGEN_VERSION=0.167.0 #1234 b5fbf5ee

Both are correct in the repository. Neither is in the running image. Every further pin bump will queue up the same way and each one widens the gap between what main says and what CI executes.

What that costs right now

  • agentic-os main is red, so promote.yml will not fast-forward release and nothing is being released from this repository
  • no successful dev-base-publish today at all: runs 3879, 3882 and 3885 all failed
  • coilyco-gaming/sirens-echo#1151 still blocked, with #1147, #1148, #1149 and #1150 behind it

This is the same shape as coilyco-flight-deck/ward sitting red for eight days: a repository cannot self-heal when the thing that would fix it is the thing that is broken.

One more note for the fix

#1234 merged onto a main that its own change turned red, and nothing refused it, which is coilyco-flight-deck/infrastructure#927. Not the cause here, but it is why a stale-image break gets discovered later rather than at the gate.

Escalating. I filed this as "publishes are failing"; it is now **agentic-os `main` is red and its release train is stalled**, and a second stranded pin has piled up behind the same rebuild. ## The second symptom looks unrelated and is not Run 3888, `gate` job, on `main`: every test in `tests/test_aosguard.py` errors with ``` subprocess.CalledProcessError: Command '['specgen', '--project-root', '.../.specgen/guardfiles', 'build', '--out', '.../aosguard']' returned non-zero exit status 1 ``` No network in it, so it reads as an application bug. It is not. `b5fbf5ee` (#1234) correctly moved `SPECGEN_VERSION` in the Dockerfile **and** `.specgen/guardfiles/specverb.lock` in the same commit, which is the right way to author that change. But the image was never rebuilt with the new pin, so CI's `:release` container still carries the **old** specgen and cannot build the **new** lock. Reproduced locally to be sure rather than inferring it: `specgen version v0.167.0 (umbra ref v0.167.0)` builds that exact lock with `rc=0`. The binary in the image is what differs. ## Two pins now stranded behind one rebuild | pin | moved by | landed | | --- | --- | --- | | `AGENT_COMPOSE_VERSION=2.47.0` | #1233 | `e2c3ea03` | | `SPECGEN_VERSION=0.167.0` | #1234 | `b5fbf5ee` | Both are correct in the repository. Neither is in the running image. Every further pin bump will queue up the same way and each one widens the gap between what `main` says and what CI executes. ## What that costs right now - **agentic-os `main` is red**, so `promote.yml` will not fast-forward `release` and nothing is being released from this repository - no successful `dev-base-publish` today at all: runs 3879, 3882 and 3885 all failed - `coilyco-gaming/sirens-echo#1151` still blocked, with #1147, #1148, #1149 and #1150 behind it This is the same shape as `coilyco-flight-deck/ward` sitting red for eight days: a repository cannot self-heal when the thing that would fix it is the thing that is broken. ## One more note for the fix #1234 merged onto a `main` that its own change turned red, and nothing refused it, which is `coilyco-flight-deck/infrastructure#927`. Not the cause here, but it is why a stale-image break gets discovered later rather than at the gate.
Author
Owner

Fourth attempt failed the same way, plus evidence that moves the suspect off the egress proxy.

Run 3889 (13:57:56Z), dispatched right after the escalation above

All five publish-lang-* tiers died at 13:58:12Z, fifteen seconds after start, before any step ran. Each one on the runner's own action fetch:

  • ☁️ git fetch 'https://github.com/actions/checkout' # ref=v6
  • could not fetch remote 'origin': fatal: unable to access 'https://github.com/actions/checkout/': Failed to connect to github.com port 443 after 2 ms: Could not connect to server

publish-full, publish-release and assert-release-moved stayed blocked. Same 2 ms refusal as runs 3879 and 3885.

The proxy allowlist is not the cause

The refusal reaches workloads that never touch FORGEJO_EGRESS_PROXY. From kubectl events -n forgejo on kai-server, external-secrets hit it twice this afternoon:

  • UpdateFailed ExternalSecret/forgejo-runner-deploy-secrets-eco-app - Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.217.78.180:443: connect: connection refused (approximately 13:52Z)
  • UpdateFailed ExternalSecret/forgejo-runner-deploy-secrets-agent-proxy - Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.220.36.112:443: connect: connection refused (approximately 14:14Z)

Both destination addresses are genuine AWS us-east-1, so DNS answered correctly and the refusal happened on the connect. external-secrets carries no proxy env, so a Squid allowlist gap cannot explain it. Every ExternalSecret currently reads SecretSynced=True, which means retries recover, and that matches the "some hosts, some of the time" shape in the issue body.

ser8 egress was healthy in the same window. coilyco-gaming/sirens-echo run 1589, job image-build, runs-on: docker, completed git fetch 'https://github.com/actions/checkout' at 13:47:00Z and went on to reach Dockerfile step 26. The ser8 pools serve docker. Only kai-server refuses.

Node state does not explain a 2 ms refusal either: kai-server is Ready, MemoryPressure, DiskPressure and PIDPressure all False, PSI cpu some avg60 9.73, io some avg60 1.23, load 8.5 at 23.6 percent CPU.

A containment that does not require solving the kai-server fault first

docker-build is declared on kai-server alone, by forgejo-runner-build (0/0) and forgejo-runner-build-flight-deck (1/1). No ser8 runner config declares it: ser8-flight-deck, ser8-canary-flight-deck, ser8-gaming and ser8-bridge all run default labels.

docs/forgejo-runners.md already places the Flight Deck image-build lane on ser8, and #693 tracks the move. Finishing it routes the publish graph onto the plane whose egress works, which lands both stranded pins without first solving why kai-server refuses.

Confirmed from the sirens-echo side

Every sirens-echo pull-request run dies at the same line, in both the test job and Dockerfile step 26:

role "engineer": pattern "tooling-ops-*" matches nothing in /tmp/aos-catalog

:release still carries the pre-reflow Core Roster, which still declares engineer, while the catalogue the build clones is post-reflow and renamed tooling-ops-* to tooling-sysadmin-* in 82967ca6. Runs 1582, 1584, 1585, 1587 and 1589 across #1150, #1151 and #1152.

#1151 fails one line earlier for the mirror-image reason, stage-compose-sources: role engineer exists in both rosters, because it was authored against the seven-seat roster while CI executes the eight-seat one. So #1151 cannot be evaluated on its merits until the image lands, and re-running it proves nothing.

Fourth attempt failed the same way, plus evidence that moves the suspect off the egress proxy. ## Run 3889 (13:57:56Z), dispatched right after the escalation above All five `publish-lang-*` tiers died at 13:58:12Z, fifteen seconds after start, before any step ran. Each one on the runner's own action fetch: * `☁️ git fetch 'https://github.com/actions/checkout' # ref=v6` * `could not fetch remote 'origin': fatal: unable to access 'https://github.com/actions/checkout/': Failed to connect to github.com port 443 after 2 ms: Could not connect to server` `publish-full`, `publish-release` and `assert-release-moved` stayed blocked. Same 2 ms refusal as runs 3879 and 3885. ## The proxy allowlist is not the cause The refusal reaches workloads that never touch `FORGEJO_EGRESS_PROXY`. From `kubectl events -n forgejo` on kai-server, external-secrets hit it twice this afternoon: * `UpdateFailed ExternalSecret/forgejo-runner-deploy-secrets-eco-app` - `Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.217.78.180:443: connect: connection refused` (approximately 13:52Z) * `UpdateFailed ExternalSecret/forgejo-runner-deploy-secrets-agent-proxy` - `Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.220.36.112:443: connect: connection refused` (approximately 14:14Z) Both destination addresses are genuine AWS us-east-1, so DNS answered correctly and the refusal happened on the connect. external-secrets carries no proxy env, so a Squid allowlist gap cannot explain it. Every ExternalSecret currently reads `SecretSynced=True`, which means retries recover, and that matches the "some hosts, some of the time" shape in the issue body. ## It is kai-server, not the house uplink ser8 egress was healthy in the same window. `coilyco-gaming/sirens-echo` run 1589, job `image-build`, `runs-on: docker`, completed `git fetch 'https://github.com/actions/checkout'` at 13:47:00Z and went on to reach Dockerfile step 26. The ser8 pools serve `docker`. Only kai-server refuses. Node state does not explain a 2 ms refusal either: kai-server is `Ready`, MemoryPressure, DiskPressure and PIDPressure all `False`, PSI cpu `some` avg60 9.73, io `some` avg60 1.23, load 8.5 at 23.6 percent CPU. ## A containment that does not require solving the kai-server fault first `docker-build` is declared on kai-server alone, by `forgejo-runner-build` (0/0) and `forgejo-runner-build-flight-deck` (1/1). No ser8 runner config declares it: `ser8-flight-deck`, `ser8-canary-flight-deck`, `ser8-gaming` and `ser8-bridge` all run default labels. [`docs/forgejo-runners.md`](../deploy/forgejo-runner-build.yml) already places the Flight Deck image-build lane on ser8, and #693 tracks the move. Finishing it routes the publish graph onto the plane whose egress works, which lands both stranded pins without first solving why kai-server refuses. ## Confirmed from the sirens-echo side Every sirens-echo pull-request run dies at the same line, in both the `test` job and Dockerfile step 26: ``` role "engineer": pattern "tooling-ops-*" matches nothing in /tmp/aos-catalog ``` `:release` still carries the pre-reflow Core Roster, which still declares `engineer`, while the catalogue the build clones is post-reflow and renamed `tooling-ops-*` to `tooling-sysadmin-*` in `82967ca6`. Runs 1582, 1584, 1585, 1587 and 1589 across #1150, #1151 and #1152. #1151 fails one line earlier for the mirror-image reason, `stage-compose-sources: role engineer exists in both rosters`, because it was authored against the seven-seat roster while CI executes the eight-seat one. So #1151 cannot be evaluated on its merits until the image lands, and re-running it proves nothing.
Author
Owner

Opened #930, which routes around this rather than fixing it. The kai-server fault stays open here.

deploy/forgejo-runners-ser8/image-build-flight-deck puts the docker-build label on ser8, and the kai-server overlay drops to zero replicas in the same commit, because leaving a replica there would keep handing roughly half of every five-tier publish graph to the plane that cannot reach github.com.

The lane needed no secret bootstrap. ser8 runs no external-secrets controller, so the overlay drops the base ExternalSecret the way the general pools do, and reuses forgejo-runner-secrets-ser8-flight-deck, since a registration token is scoped to the organization rather than to one pool.

Capacity checked first, since the runbook makes it the precondition. ser8 is Ready with no pressure: 16 CPU, 60 GiB memory, 931 GiB allocatable ephemeral storage, current requests at 44 percent CPU and 26 percent memory and 15 percent ephemeral, root filesystem 45.9 percent used with 507 GB free. The lane adds roughly 1.5 CPU and 3.2 GiB of requests plus a disposable 10 GiB claim. Node memory limits are already at 96 percent and this pushes overcommit further, which is the one number in the change worth a second opinion.

One thing moved that should not have to. .forgejo/workflows/build-runner-deploy.yml shares the docker-build label and pushes to the standalone registry NodePort on kai-server's LAN, which ser8 cannot route to. It was already failing here on this same egress fault, so it is a change of failure mode rather than a new outage, and it is filed at #931 rather than left in a commit message.

Render and pre-commit are green. Neither proves live health, and the merge is the live action, so I stopped at the pull request per this repo's lane.

Opened #930, which routes around this rather than fixing it. The kai-server fault stays open here. `deploy/forgejo-runners-ser8/image-build-flight-deck` puts the `docker-build` label on ser8, and the kai-server overlay drops to zero replicas in the same commit, because leaving a replica there would keep handing roughly half of every five-tier publish graph to the plane that cannot reach github.com. The lane needed no secret bootstrap. ser8 runs no external-secrets controller, so the overlay drops the base ExternalSecret the way the general pools do, and reuses `forgejo-runner-secrets-ser8-flight-deck`, since a registration token is scoped to the organization rather than to one pool. Capacity checked first, since the runbook makes it the precondition. ser8 is `Ready` with no pressure: 16 CPU, 60 GiB memory, 931 GiB allocatable ephemeral storage, current requests at 44 percent CPU and 26 percent memory and 15 percent ephemeral, root filesystem 45.9 percent used with 507 GB free. The lane adds roughly 1.5 CPU and 3.2 GiB of requests plus a disposable 10 GiB claim. Node memory *limits* are already at 96 percent and this pushes overcommit further, which is the one number in the change worth a second opinion. One thing moved that should not have to. `.forgejo/workflows/build-runner-deploy.yml` shares the `docker-build` label and pushes to the standalone registry NodePort on kai-server's LAN, which ser8 cannot route to. It was already failing here on this same egress fault, so it is a change of failure mode rather than a new outage, and it is filed at #931 rather than left in a commit message. Render and pre-commit are green. Neither proves live health, and the merge is the live action, so I stopped at the pull request per this repo's lane.
Author
Owner

The publish lane is working again. #930 merged, and run 3890 completed the entire dev-base-publish graph on ser8.

  • plan-draft, all five publish-lang-* tiers, publish-full, publish-release, assert-release-moved, plan-release, retag-full, release - every one green
  • publish-full log records agent-compose version printing v2.47.0, so the stranded agent-compose pin is in the running image
  • assert-release-moved passing is the branch-and-image disagreement in the issue body closing

The ser8 lane came up clean and needed no bootstrap. forgejo-runner-build-ser8-flight-deck-0 reached 2/2 in seventeen seconds and registered with labels: [docker-build], declared successfully against the existing org Secret. forgejo-runner-build-flight-deck on kai-server is 0/0.

The second stranded pin is still stranded

release sits at e2c3ea03, which carries the agent-compose bump but predates the specgen one. The new image records specgen version v0.161.0 (umbra ref v0.161.0), so agentic-os main stays red exactly as the escalation above describes: CI's container cannot build main's .specgen/guardfiles/specverb.lock from b5fbf5ee.

Publishing main's tree onto the :release tag would move the image ahead of the branch, which is the disagreement assert-release-moved exists to catch, so I did not do that. Breaking it needs a repository-side decision on agentic-os rather than an operator action here: either split #1234 so the Dockerfile pin lands and promotes before the lock does, or revert the lock, promote, publish, and reland it. That is the platform seat's call, not this issue's.

What this issue still owns

kai-server's egress fault is untouched. It still refuses outbound 443 in single-digit milliseconds, and external-secrets still logs connection refused to AWS SSM. #930 routed the publish lane around it, which is containment rather than a fix, so this stays open.

Two small surface notes for whoever picks it up

  • the Forgejo API on this instance answers POST runs/{id}/rerun and runs/{id}/rerun-failed-jobs with 404, so aosguard ops forgejo fetch actions-rerun cannot retry a run. A retry needs a push or a workflow_dispatch, which matches the dispatch-input trouble noted in the issue body
  • #931 carries build-runner-deploy, which shares the docker-build label and cannot follow it to ser8 until its image leaves the standalone registry
The publish lane is working again. #930 merged, and run **3890** completed the entire `dev-base-publish` graph on ser8. * `plan-draft`, all five `publish-lang-*` tiers, `publish-full`, `publish-release`, `assert-release-moved`, `plan-release`, `retag-full`, `release` - every one green * `publish-full` log records `agent-compose version` printing **v2.47.0**, so the stranded agent-compose pin is in the running image * `assert-release-moved` passing is the branch-and-image disagreement in the issue body closing The ser8 lane came up clean and needed no bootstrap. `forgejo-runner-build-ser8-flight-deck-0` reached 2/2 in seventeen seconds and registered `with labels: [docker-build], declared successfully` against the existing org Secret. `forgejo-runner-build-flight-deck` on kai-server is 0/0. ## The second stranded pin is still stranded `release` sits at `e2c3ea03`, which carries the agent-compose bump but predates the specgen one. The new image records `specgen version v0.161.0 (umbra ref v0.161.0)`, so agentic-os `main` stays red exactly as the escalation above describes: CI's container cannot build `main`'s `.specgen/guardfiles/specverb.lock` from `b5fbf5ee`. Publishing `main`'s tree onto the `:release` tag would move the image ahead of the branch, which is the disagreement `assert-release-moved` exists to catch, so I did not do that. Breaking it needs a repository-side decision on agentic-os rather than an operator action here: either split #1234 so the Dockerfile pin lands and promotes before the lock does, or revert the lock, promote, publish, and reland it. That is the platform seat's call, not this issue's. ## What this issue still owns **kai-server's egress fault is untouched.** It still refuses outbound 443 in single-digit milliseconds, and external-secrets still logs `connection refused` to AWS SSM. #930 routed the publish lane around it, which is containment rather than a fix, so this stays open. ## Two small surface notes for whoever picks it up * the Forgejo API on this instance answers `POST runs/{id}/rerun` and `runs/{id}/rerun-failed-jobs` with 404, so `aosguard ops forgejo fetch actions-rerun` cannot retry a run. A retry needs a push or a `workflow_dispatch`, which matches the dispatch-input trouble noted in the issue body * #931 carries `build-runner-deploy`, which shares the `docker-build` label and cannot follow it to ser8 until its image leaves the standalone registry
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/infrastructure#929
No description provided.