fix(runners): serve the image-build lane from ser8, not the plane refusing 443 #930

Merged

coilyco-ops merged 1 commit from aos/claude/ser8-image-build-929 into main

2026-08-25 16:49:37 +00:00

coilyco-ops commented

2026-08-25 14:28:52 +00:00

Owner

Moves the Flight Deck image-build lane to ser8, the plane docs/forgejo-runners.md already names as its home, so the stalled dev-base publish train stops depending on the kai-server egress fault being solved first.

Why now

kai-server refuses outbound 443 in single-digit milliseconds, intermittently and across unrelated destinations. Evidence is in #929, including the reads I added at #issuecomment-76035:

run 3889, all five publish-lang-* tiers dead fifteen seconds in on Failed to connect to github.com port 443 after 2 ms
ExternalSecret reconciles hitting the same instant refusal to ssm.us-east-1.amazonaws.com, which is not behind FORGEJO_EGRESS_PROXY, so a Squid allowlist gap does not explain it
ser8 reaching github.com successfully in the same windows

Two correct pins are stranded in agentic-os but absent from the running image, AGENT_COMPOSE_VERSION=2.47.0 and SPECGEN_VERSION=0.167.0. Downstream, every coilyco-gaming/sirens-echo pull request dies on role "engineer": pattern "tooling-ops-*" matches nothing, because :release still bakes the pre-reflow Core Roster against a post-reflow catalogue.

What changed

deploy/forgejo-runners-ser8/image-build-flight-deck wraps the shared build base for ser8: the public Forgejo instance URL rather than the in-cluster Service, cni0's 1450 MTU rather than kai-server's 1230, and no carve-out for the other cluster's standalone registry
it reuses forgejo-runner-secrets-ser8-flight-deck, the org Secret the general Flight Deck pool already carries, so there is no bootstrap step. ser8 runs no external-secrets controller, so the base ExternalSecret is dropped the way the general overlay drops it
deploy/forgejo-runners-scoped/image-build-flight-deck drops to zero replicas. Leaving a replica would keep handing roughly half of every five-tier publish graph back to the failing plane
the Flux slice gains the overlay and clusters/ser8/runners-general.yaml gains its health check
just forgejo-ser8-runners-render <slice> is the new read-only render verb, mirroring forgejo-scoped-runners-render

Capacity, since the runbook makes it the precondition

ser8 is Ready with no pressure: 16 CPU, 60 GiB memory, 931 GiB allocatable ephemeral storage. Current requests are 44 percent CPU, 26 percent memory, 15 percent ephemeral storage, and the root filesystem is 45.9 percent used with 507 GB free. The lane adds roughly 1.5 CPU and 3.2 GiB of requests plus a disposable 10 GiB docker-lib claim. Memory limits on the node are already at 96 percent and this pushes overcommit further, which is worth a second opinion.

What this does not fix and does not claim

the kai-server egress fault is untouched. #929 stays open
.forgejo/workflows/build-runner-deploy.yml shares the docker-build label and moves with it. It pushes to the standalone registry NodePort on kai-server's LAN, which ser8 cannot reach, so that job stays down until its image moves to Forgejo OCI. It was already failing on the same egress fault before this change. Filed separately
render and pre-commit prove repository validity, not live health. Nothing here is verified against a running cluster, and the merge is the live action

Rollback

Remove the overlay from deploy/_flux/ser8-runners-general/kustomization.yaml and restore replicas: 1 on the kai-server overlay. That returns the lane to a plane that currently cannot reach github.com, so it is a rollback to the broken state rather than to a good one.

Verification the merge needs

forgejo-runner-build-ser8-flight-deck-0 reaches 2/2 on ser8 and registers with the docker-build label
forgejo-runner-build-flight-deck on kai-server reports 0/0
a dev-base-publish dispatch on release completes publish-full, publish-release and assert-release-moved
agentic-os:release reports agent-compose version v2.47.0
a sirens-echo pull-request run clears Dockerfile step 26

Moves the Flight Deck image-build lane to ser8, the plane `docs/forgejo-runners.md` already names as its home, so the stalled dev-base publish train stops depending on the kai-server egress fault being solved first. ## Why now kai-server refuses outbound 443 in single-digit milliseconds, intermittently and across unrelated destinations. Evidence is in #929, including the reads I added at `#issuecomment-76035`: * run 3889, all five `publish-lang-*` tiers dead fifteen seconds in on `Failed to connect to github.com port 443 after 2 ms` * `ExternalSecret` reconciles hitting the same instant refusal to `ssm.us-east-1.amazonaws.com`, which is not behind `FORGEJO_EGRESS_PROXY`, so a Squid allowlist gap does not explain it * ser8 reaching github.com successfully in the same windows Two correct pins are stranded in agentic-os but absent from the running image, `AGENT_COMPOSE_VERSION=2.47.0` and `SPECGEN_VERSION=0.167.0`. Downstream, every `coilyco-gaming/sirens-echo` pull request dies on `role "engineer": pattern "tooling-ops-*" matches nothing`, because `:release` still bakes the pre-reflow Core Roster against a post-reflow catalogue. ## What changed * `deploy/forgejo-runners-ser8/image-build-flight-deck` wraps the shared build base for ser8: the public Forgejo instance URL rather than the in-cluster Service, cni0's 1450 MTU rather than kai-server's 1230, and no carve-out for the other cluster's standalone registry * it reuses `forgejo-runner-secrets-ser8-flight-deck`, the org Secret the general Flight Deck pool already carries, so there is no bootstrap step. ser8 runs no external-secrets controller, so the base ExternalSecret is dropped the way the general overlay drops it * `deploy/forgejo-runners-scoped/image-build-flight-deck` drops to zero replicas. Leaving a replica would keep handing roughly half of every five-tier publish graph back to the failing plane * the Flux slice gains the overlay and `clusters/ser8/runners-general.yaml` gains its health check * `just forgejo-ser8-runners-render <slice>` is the new read-only render verb, mirroring `forgejo-scoped-runners-render` ## Capacity, since the runbook makes it the precondition ser8 is `Ready` with no pressure: 16 CPU, 60 GiB memory, 931 GiB allocatable ephemeral storage. Current requests are 44 percent CPU, 26 percent memory, 15 percent ephemeral storage, and the root filesystem is 45.9 percent used with 507 GB free. The lane adds roughly 1.5 CPU and 3.2 GiB of requests plus a disposable 10 GiB docker-lib claim. Memory limits on the node are already at 96 percent and this pushes overcommit further, which is worth a second opinion. ## What this does not fix and does not claim * the kai-server egress fault is untouched. #929 stays open * `.forgejo/workflows/build-runner-deploy.yml` shares the `docker-build` label and moves with it. It pushes to the standalone registry NodePort on kai-server's LAN, which ser8 cannot reach, so that job stays down until its image moves to Forgejo OCI. It was already failing on the same egress fault before this change. Filed separately * render and pre-commit prove repository validity, not live health. Nothing here is verified against a running cluster, and the merge is the live action ## Rollback Remove the overlay from `deploy/_flux/ser8-runners-general/kustomization.yaml` and restore `replicas: 1` on the kai-server overlay. That returns the lane to a plane that currently cannot reach github.com, so it is a rollback to the broken state rather than to a good one. ## Verification the merge needs 1. `forgejo-runner-build-ser8-flight-deck-0` reaches 2/2 on ser8 and registers with the `docker-build` label 2. `forgejo-runner-build-flight-deck` on kai-server reports 0/0 3. a `dev-base-publish` dispatch on `release` completes `publish-full`, `publish-release` and `assert-release-moved` 4. `agentic-os:release` reports `agent-compose version v2.47.0` 5. a sirens-echo pull-request run clears Dockerfile step 26

coilyco-ops added 1 commit

2026-08-25 14:28:52 +00:00

fix(runners): serve the image-build lane from ser8, not the plane refusing 443

TruffleHog / Scan for secrets (pull_request) Successful in 28s

Details

CI / lint (pull_request) Successful in 50s

Details

b915e08bb0

kai-server refuses outbound 443 in single-digit milliseconds, intermittently
and across unrelated destinations, so every dev-base publish since the
agent-compose pin landed has died before it could use the pin. ser8 egress is
healthy in the same windows, and docs/forgejo-runners.md already names ser8 as
this lane's home, so the lane moves rather than waiting on that fault.

deploy/forgejo-runners-ser8/image-build-flight-deck wraps the shared build base
with ser8's shape: the public Forgejo instance URL rather than the in-cluster
Service, the org Secret the general Flight Deck pool already carries rather than
an ExternalSecret ser8 has no controller for, cni0's 1450 MTU, and no carve-out
for the other cluster's standalone registry.

The kai-server overlay drops to zero replicas in the same change. Leaving a
replica there would keep handing roughly half of every five-tier publish graph
to the plane that cannot reach github.com.

That takes build-runner-deploy.yml with it, since it shares the label and pushes
to a NodePort on kai-server's LAN. It was already failing on the same egress
fault, and its image needs to move to Forgejo OCI before its lane can follow.

Refs coilyco-flight-deck/infrastructure#929, coilyco-flight-deck/infrastructure#693

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin

coilyco-ops referenced this pull request

2026-08-25 14:29:11 +00:00

build-runner-deploy pushes to a kai-server LAN NodePort, so its lane cannot follow docker-build to ser8 #931

coilyco-ops referenced this pull request

2026-08-25 14:29:26 +00:00

Runner egress refuses github.com and archive.ubuntu.com in 2-11ms, blocking every dev-base publish #929

coilyco-ops merged commit 01cc99d41c into main

2026-08-25 16:49:37 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-25 16:49:38 +00:00

fix(runners): serve the image-build lane from ser8, not the plane refusing 443 (#930)

coilyco-ops referenced this pull request