build-runner-deploy pushes to a kai-server LAN NodePort, so its lane cannot follow docker-build to ser8 #931

New issue

Open

opened 2026-08-25 14:29:11 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-25 14:29:11 +00:00

Owner

.forgejo/workflows/build-runner-deploy.yml builds and pushes the tool-bearing CD deploy-runner image, and scripts/actions/build-runner-deploy.sh pushes it to the standalone registry NodePort on kai-server's LAN. The kai-server image-build runner's dind carries the matching --insecure-registry flag, so that job only works from that cluster.

#930 moves the docker-build label to ser8, because kai-server refuses outbound 443 and stalls every dev-base publish (#929). The deploy-runner build shares that label, so it moves too, and ser8 cannot route to the other cluster's LAN NodePort.

What that costs

The job is down until this lands. It was already down before the move, on the same egress fault, since its actions/checkout needs github.com from kai-server. So this is a change of failure mode rather than a new outage, but it is a real gap and it will not fix itself when #929 does.

It is path-triggered on deploy/forgejo-runner-deploy-image/**, scripts/actions/build-runner-deploy.sh, and its own workflow file, so nothing fires until one of those changes. The currently deployed forgejo-runner-deploy:latest image keeps running, since the running deploy runners already hold it.

What resolves it

Move the deploy-runner image to Forgejo OCI. docs/k3s-deploy-notes-topology.md and the change-infrastructure skill both name Forgejo OCI the application-image authority and the standalone registry a rollback bridge, so this is the direction already chosen rather than a new one. That means the push target in scripts/actions/build-runner-deploy.sh, the REGISTRY_TOKEN the publisher base already provides, the pull side in deploy/forgejo-runner-deploy.yml, and whatever imagePullSecret the deploy runners need.

Alternatively, keep one kai-server replica alive purely for this job under a distinct label. That is smaller, and it leaves a lane on a plane whose egress is unreliable.

Acceptance

build-runner-deploy completes on a ser8 runner and pushes a tag plus latest to Forgejo OCI
a deploy runner pod pulls the new image and reaches Running
the standalone registry NodePort is no longer named by this workflow or its script

`.forgejo/workflows/build-runner-deploy.yml` builds and pushes the tool-bearing CD deploy-runner image, and `scripts/actions/build-runner-deploy.sh` pushes it to the standalone registry NodePort on kai-server's LAN. The kai-server image-build runner's dind carries the matching `--insecure-registry` flag, so that job only works from that cluster. #930 moves the `docker-build` label to ser8, because kai-server refuses outbound 443 and stalls every dev-base publish (#929). The deploy-runner build shares that label, so it moves too, and ser8 cannot route to the other cluster's LAN NodePort. ## What that costs The job is down until this lands. It was already down before the move, on the same egress fault, since its `actions/checkout` needs github.com from kai-server. So this is a change of failure mode rather than a new outage, but it is a real gap and it will not fix itself when #929 does. It is path-triggered on `deploy/forgejo-runner-deploy-image/**`, `scripts/actions/build-runner-deploy.sh`, and its own workflow file, so nothing fires until one of those changes. The currently deployed `forgejo-runner-deploy:latest` image keeps running, since the running deploy runners already hold it. ## What resolves it Move the deploy-runner image to Forgejo OCI. `docs/k3s-deploy-notes-topology.md` and the change-infrastructure skill both name Forgejo OCI the application-image authority and the standalone registry a rollback bridge, so this is the direction already chosen rather than a new one. That means the push target in `scripts/actions/build-runner-deploy.sh`, the `REGISTRY_TOKEN` the publisher base already provides, the pull side in `deploy/forgejo-runner-deploy.yml`, and whatever imagePullSecret the deploy runners need. Alternatively, keep one kai-server replica alive purely for this job under a distinct label. That is smaller, and it leaves a lane on a plane whose egress is unreliable. ## Acceptance * `build-runner-deploy` completes on a ser8 runner and pushes a tag plus `latest` to Forgejo OCI * a deploy runner pod pulls the new image and reaches Running * the standalone registry NodePort is no longer named by this workflow or its script

coilyco-ops referenced this issue

2026-08-25 14:29:26 +00:00

Runner egress refuses github.com and archive.ubuntu.com in 2-11ms, blocking every dev-base publish #929

coilyco-ops referenced this issue

2026-08-25 17:41:45 +00:00

Runner egress refuses github.com and archive.ubuntu.com in 2-11ms, blocking every dev-base publish #929

coilyco-ops added this to the Forgejo runners milestone

2026-08-29 05:40:47 +00:00