Runner egress refuses github.com and archive.ubuntu.com in 2-11ms, blocking every dev-base publish #929
Labels
No labels
burndown-2026-06
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/advocate
role/director
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/science
role/sysadmin
state
ambient
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
coilyco-flight-deck/infrastructure#929
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The dev-base publish graph cannot complete. Three attempts this session, each dying on a different external host, all with the same signature: connection refused in single-digit to low-double-digit milliseconds, not a timeout.
Evidence
Failed to connect ... after 11 msConnection refused(v4) +Network is unreachable(v6)go install ./...return code 1, allgo: downloadingFailed to connect ... after 2 msA 2ms refusal is not a network being slow or a remote being down. Something local is answering immediately and saying no: an egress proxy that is down or has an allowlist that does not cover these hosts, or a DNS answer pointing at a closed port.
Why it is not simply "the internet was flaky"
Run 3882's five language payload jobs succeeded at ~10:40, and those jobs pull the Go toolchain from
go.dev, Node fromnodejs.org, the dotnet installer fromdot.net, and rustup fromsh.rustup.rs. Seven minutes later, in the same run,publish-fullcould not reacharchive.ubuntu.com. Then at 10:56 the same payload jobs that had just succeeded failed ongithub.com.So egress works for some hosts, some of the time, and refuses others instantly. That reads as partial or intermittent proxying rather than an outage.
Note also
tests/test_ci_egress_proxy.pyexists in agentic-os, so there is a proxy in this path by design. This issue is about whether it is healthy and whether its allowlist coversgithub.com(needed foractions/checkout@v6),archive.ubuntu.com,security.ubuntu.com, and the Go module proxy.What it is blocking
coilyco-flight-deck/agentic-os#1233merged, moving the dev-base pin to agent-compose 2.47.0 so the image stops shipping a pre-reflow Core Roster. Thereleasebranch advanced toe2c3ea03, but the image never rebuilt, so:releasestill carries 2.32.0.assert-release-movedcorrectly failed on the first attempt and caught the mismatch, which is the pipeline working. But it leaves branch and image disagreeing, and every consumer that composes in:releasestill bakes the old roster.Downstream,
coilyco-gaming/sirens-echo#1151is the migration that needs the new image and cannot go green until it lands.coilyco-gaming/sirens-echo#1147,#1148,#1149and#1150sit behind that.Also worth checking
I dispatched
dev-base-publish.ymlwithinputs.tier=fullto resume only the failed tier, via--body-filesince the aosguard verb has no inputs flag. The language payload jobs ran anyway. Either the input did not reach the workflow or the resume path does not skip them, and it matters because it turns a narrow retry into a full graph rebuild against the exact network that is failing.What I did not do
Stopped after three dispatches rather than looping. Nothing here is fixable from the repository side: the pin is correct and merged, and the build is refused before it can use it. Filed rather than retried.
Escalating. I filed this as "publishes are failing"; it is now agentic-os
mainis red and its release train is stalled, and a second stranded pin has piled up behind the same rebuild.The second symptom looks unrelated and is not
Run 3888,
gatejob, onmain: every test intests/test_aosguard.pyerrors withNo network in it, so it reads as an application bug. It is not.
b5fbf5ee(#1234) correctly movedSPECGEN_VERSIONin the Dockerfile and.specgen/guardfiles/specverb.lockin the same commit, which is the right way to author that change. But the image was never rebuilt with the new pin, so CI's:releasecontainer still carries the old specgen and cannot build the new lock.Reproduced locally to be sure rather than inferring it:
specgen version v0.167.0 (umbra ref v0.167.0)builds that exact lock withrc=0. The binary in the image is what differs.Two pins now stranded behind one rebuild
AGENT_COMPOSE_VERSION=2.47.0e2c3ea03SPECGEN_VERSION=0.167.0b5fbf5eeBoth are correct in the repository. Neither is in the running image. Every further pin bump will queue up the same way and each one widens the gap between what
mainsays and what CI executes.What that costs right now
mainis red, sopromote.ymlwill not fast-forwardreleaseand nothing is being released from this repositorydev-base-publishtoday at all: runs 3879, 3882 and 3885 all failedcoilyco-gaming/sirens-echo#1151still blocked, with #1147, #1148, #1149 and #1150 behind itThis is the same shape as
coilyco-flight-deck/wardsitting red for eight days: a repository cannot self-heal when the thing that would fix it is the thing that is broken.One more note for the fix
#1234 merged onto a
mainthat its own change turned red, and nothing refused it, which iscoilyco-flight-deck/infrastructure#927. Not the cause here, but it is why a stale-image break gets discovered later rather than at the gate.Fourth attempt failed the same way, plus evidence that moves the suspect off the egress proxy.
Run 3889 (13:57:56Z), dispatched right after the escalation above
All five
publish-lang-*tiers died at 13:58:12Z, fifteen seconds after start, before any step ran. Each one on the runner's own action fetch:☁️ git fetch 'https://github.com/actions/checkout' # ref=v6could not fetch remote 'origin': fatal: unable to access 'https://github.com/actions/checkout/': Failed to connect to github.com port 443 after 2 ms: Could not connect to serverpublish-full,publish-releaseandassert-release-movedstayed blocked. Same 2 ms refusal as runs 3879 and 3885.The proxy allowlist is not the cause
The refusal reaches workloads that never touch
FORGEJO_EGRESS_PROXY. Fromkubectl events -n forgejoon kai-server, external-secrets hit it twice this afternoon:UpdateFailed ExternalSecret/forgejo-runner-deploy-secrets-eco-app-Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.217.78.180:443: connect: connection refused(approximately 13:52Z)UpdateFailed ExternalSecret/forgejo-runner-deploy-secrets-agent-proxy-Post "https://ssm.us-east-1.amazonaws.com/": dial tcp 13.220.36.112:443: connect: connection refused(approximately 14:14Z)Both destination addresses are genuine AWS us-east-1, so DNS answered correctly and the refusal happened on the connect. external-secrets carries no proxy env, so a Squid allowlist gap cannot explain it. Every ExternalSecret currently reads
SecretSynced=True, which means retries recover, and that matches the "some hosts, some of the time" shape in the issue body.It is kai-server, not the house uplink
ser8 egress was healthy in the same window.
coilyco-gaming/sirens-echorun 1589, jobimage-build,runs-on: docker, completedgit fetch 'https://github.com/actions/checkout'at 13:47:00Z and went on to reach Dockerfile step 26. The ser8 pools servedocker. Only kai-server refuses.Node state does not explain a 2 ms refusal either: kai-server is
Ready, MemoryPressure, DiskPressure and PIDPressure allFalse, PSI cpusomeavg60 9.73, iosomeavg60 1.23, load 8.5 at 23.6 percent CPU.A containment that does not require solving the kai-server fault first
docker-buildis declared on kai-server alone, byforgejo-runner-build(0/0) andforgejo-runner-build-flight-deck(1/1). No ser8 runner config declares it:ser8-flight-deck,ser8-canary-flight-deck,ser8-gamingandser8-bridgeall run default labels.docs/forgejo-runners.mdalready places the Flight Deck image-build lane on ser8, and #693 tracks the move. Finishing it routes the publish graph onto the plane whose egress works, which lands both stranded pins without first solving why kai-server refuses.Confirmed from the sirens-echo side
Every sirens-echo pull-request run dies at the same line, in both the
testjob and Dockerfile step 26::releasestill carries the pre-reflow Core Roster, which still declaresengineer, while the catalogue the build clones is post-reflow and renamedtooling-ops-*totooling-sysadmin-*in82967ca6. Runs 1582, 1584, 1585, 1587 and 1589 across #1150, #1151 and #1152.#1151 fails one line earlier for the mirror-image reason,
stage-compose-sources: role engineer exists in both rosters, because it was authored against the seven-seat roster while CI executes the eight-seat one. So #1151 cannot be evaluated on its merits until the image lands, and re-running it proves nothing.Opened #930, which routes around this rather than fixing it. The kai-server fault stays open here.
deploy/forgejo-runners-ser8/image-build-flight-deckputs thedocker-buildlabel on ser8, and the kai-server overlay drops to zero replicas in the same commit, because leaving a replica there would keep handing roughly half of every five-tier publish graph to the plane that cannot reach github.com.The lane needed no secret bootstrap. ser8 runs no external-secrets controller, so the overlay drops the base ExternalSecret the way the general pools do, and reuses
forgejo-runner-secrets-ser8-flight-deck, since a registration token is scoped to the organization rather than to one pool.Capacity checked first, since the runbook makes it the precondition. ser8 is
Readywith no pressure: 16 CPU, 60 GiB memory, 931 GiB allocatable ephemeral storage, current requests at 44 percent CPU and 26 percent memory and 15 percent ephemeral, root filesystem 45.9 percent used with 507 GB free. The lane adds roughly 1.5 CPU and 3.2 GiB of requests plus a disposable 10 GiB claim. Node memory limits are already at 96 percent and this pushes overcommit further, which is the one number in the change worth a second opinion.One thing moved that should not have to.
.forgejo/workflows/build-runner-deploy.ymlshares thedocker-buildlabel and pushes to the standalone registry NodePort on kai-server's LAN, which ser8 cannot route to. It was already failing here on this same egress fault, so it is a change of failure mode rather than a new outage, and it is filed at #931 rather than left in a commit message.Render and pre-commit are green. Neither proves live health, and the merge is the live action, so I stopped at the pull request per this repo's lane.
The publish lane is working again. #930 merged, and run 3890 completed the entire
dev-base-publishgraph on ser8.plan-draft, all fivepublish-lang-*tiers,publish-full,publish-release,assert-release-moved,plan-release,retag-full,release- every one greenpublish-fulllog recordsagent-compose versionprinting v2.47.0, so the stranded agent-compose pin is in the running imageassert-release-movedpassing is the branch-and-image disagreement in the issue body closingThe ser8 lane came up clean and needed no bootstrap.
forgejo-runner-build-ser8-flight-deck-0reached 2/2 in seventeen seconds and registeredwith labels: [docker-build], declared successfullyagainst the existing org Secret.forgejo-runner-build-flight-deckon kai-server is 0/0.The second stranded pin is still stranded
releasesits ate2c3ea03, which carries the agent-compose bump but predates the specgen one. The new image recordsspecgen version v0.161.0 (umbra ref v0.161.0), so agentic-osmainstays red exactly as the escalation above describes: CI's container cannot buildmain's.specgen/guardfiles/specverb.lockfromb5fbf5ee.Publishing
main's tree onto the:releasetag would move the image ahead of the branch, which is the disagreementassert-release-movedexists to catch, so I did not do that. Breaking it needs a repository-side decision on agentic-os rather than an operator action here: either split #1234 so the Dockerfile pin lands and promotes before the lock does, or revert the lock, promote, publish, and reland it. That is the platform seat's call, not this issue's.What this issue still owns
kai-server's egress fault is untouched. It still refuses outbound 443 in single-digit milliseconds, and external-secrets still logs
connection refusedto AWS SSM. #930 routed the publish lane around it, which is containment rather than a fix, so this stays open.Two small surface notes for whoever picks it up
POST runs/{id}/rerunandruns/{id}/rerun-failed-jobswith 404, soaosguard ops forgejo fetch actions-reruncannot retry a run. A retry needs a push or aworkflow_dispatch, which matches the dispatch-input trouble noted in the issue bodybuild-runner-deploy, which shares thedocker-buildlabel and cannot follow it to ser8 until its image leaves the standalone registry