coilyco-flight-deck/galaxy-gen

Fork 0

Cut over #22 stock-caddy restructure to the cluster #23

New issue

Closed

opened 2026-06-06 03:20:25 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-06-06 03:20:25 +00:00

Owner

Context

The code for #22 (serve dist from stock caddy:2-alpine + mounted bundle) is on main (commit 5197f8b): Dockerfile stage 2 is a pure busybox data image, deploy/main.yml runs stock caddy + a bundle-loader initContainer + two emptyDirs. This issue tracks the deploy cutover of that restructure to the live cluster.

Deploy cutover (the load-bearing part)

CI on push to main (.forgejo/workflows/build-publish-deploy.yml) only runs kubectl set image - it does not kubectl apply the restructured deploy/main.yml. So a plain merge breaks serving: set image points the current live single-container deployment's coilysiren-galaxy-gen container at the new non-serving busybox image, and the new pod spec (initContainer + volumes + stock caddy) never gets applied.

A deploy verb (coily exec deploy -> make deploy) is being added to .coily/coily.yaml so the apply is a first-class command. Cutover sequence:

coily exec deploy - build + push the busybox bundle image for HEAD sha, then kubectl apply the rendered new main.yml and rollout status. This stands up the initContainer + emptyDir + stock-caddy structure and rolls it healthy.
From here, CI's per-push set image matches the initContainer by name, so ongoing rolls are clean.

Future simplification

Once kai-server is on k8s 1.33+ (ImageVolume GA), swap the initContainer + emptyDir for a volumes[].image mount and change Dockerfile stage 2 from busybox to FROM scratch - the bundle image is already pure data. Same pattern is slated for the repos.coilysiren.me catalog-graph viewer (see #22 cross-link).

Split out

The broken v0.14.0 pre-commit WIP adoption moved to #24.

## Context The code for #22 (serve dist from stock `caddy:2-alpine` + mounted bundle) is on `main` (commit `5197f8b`): Dockerfile stage 2 is a pure busybox data image, `deploy/main.yml` runs stock caddy + a bundle-loader initContainer + two emptyDirs. This issue tracks the **deploy cutover** of that restructure to the live cluster. ## Deploy cutover (the load-bearing part) CI on push to `main` (`.forgejo/workflows/build-publish-deploy.yml`) only runs `kubectl set image` - it does **not** `kubectl apply` the restructured `deploy/main.yml`. So a plain merge breaks serving: `set image` points the *current* live single-container deployment's `coilysiren-galaxy-gen` container at the new non-serving busybox image, and the new pod spec (initContainer + volumes + stock caddy) never gets applied. A `deploy` verb (`coily exec deploy` -> `make deploy`) is being added to `.coily/coily.yaml` so the apply is a first-class command. Cutover sequence: 1. `coily exec deploy` - build + push the busybox bundle image for HEAD sha, then `kubectl apply` the rendered new `main.yml` and `rollout status`. This stands up the initContainer + emptyDir + stock-caddy structure and rolls it healthy. 2. From here, CI's per-push `set image` matches the initContainer by name, so ongoing rolls are clean. ## Future simplification Once kai-server is on k8s 1.33+ (ImageVolume GA), swap the initContainer + emptyDir for a `volumes[].image` mount and change Dockerfile stage 2 from `busybox` to `FROM scratch` - the bundle image is already pure data. Same pattern is slated for the `repos.coilysiren.me` catalog-graph viewer (see #22 cross-link). ## Split out - The broken `v0.14.0` pre-commit WIP adoption moved to #24.

coilysiren changed title from ~~Cut over #22 stock-caddy restructure to the cluster (+ finish v0.14.0 pre-commit adoption)~~ to Cut over #22 stock-caddy restructure to the cluster

2026-06-06 03:33:46 +00:00

coilysiren referenced this issue from a commit

2026-06-06 03:36:14 +00:00

chore(coily): add `deploy` verb for the cluster cutover

coilysiren commented

2026-06-06 03:42:51 +00:00

Author

Owner

Reopening - the cutover did not complete. The deploy verb landed (dcb1533) but running it surfaced a stack of pre-existing deploy-pipeline breakage, none of it from the #22 restructure:

Forgejo Actions runner is not picking up pushes. My pushes 5197f8b and dcb1533 triggered no CI runs - newest run is run#14 on the old 5b03ec2. So nothing rebuilds on push right now.
The CI deploy job has never succeeded. Every deploy task in actions task list is failure (run#14/#13/#12...); only test passes. galaxy-gen has never actually deployed via this pipeline.
The in-cluster registry is empty (/v2/_catalog -> only probe). No galaxy-gen image was ever pushed, so an apply-only cutover is impossible - the initContainer would have nothing to pull.
Local coily exec deploy (the manual path) failed twice:
- run 1: build OK, then docker push to 192.168.0.194:30500 -> unexpected EOF (.publish Error 1).
- run 2: build failed resolving docker/dockerfile:1.7 from docker.io -> macOS keychain error getting credentials ... User canceled (-128). Needs an interactive terminal to approve the keychain prompt (or docker logout / login fix), and the registry-push EOF still needs explaining.

So the code is correct and merged; the cutover is blocked on deploy-infra that's out of band from #22. Likely needs: (a) bring the Forgejo Actions runner back / confirm it's processing, (b) fix the CI deploy job (separate long-standing issue), or (c) run the manual coily exec deploy interactively on a host with working docker.io creds + registry push.

Reopening - the cutover did not complete. The `deploy` verb landed (`dcb1533`) but running it surfaced a stack of pre-existing deploy-pipeline breakage, none of it from the #22 restructure: 1. **Forgejo Actions runner is not picking up pushes.** My pushes `5197f8b` and `dcb1533` triggered **no** CI runs - newest run is run#14 on the old `5b03ec2`. So nothing rebuilds on push right now. 2. **The CI `deploy` job has never succeeded.** Every `deploy` task in `actions task list` is `failure` (run#14/#13/#12...); only `test` passes. galaxy-gen has never actually deployed via this pipeline. 3. **The in-cluster registry is empty** (`/v2/_catalog` -> only `probe`). No galaxy-gen image was ever pushed, so an apply-only cutover is impossible - the initContainer would have nothing to pull. 4. **Local `coily exec deploy` (the manual path) failed twice:** - run 1: build OK, then `docker push` to `192.168.0.194:30500` -> `unexpected EOF` (`.publish` Error 1). - run 2: build failed resolving `docker/dockerfile:1.7` from docker.io -> macOS keychain `error getting credentials ... User canceled (-128)`. Needs an interactive terminal to approve the keychain prompt (or `docker logout` / login fix), and the registry-push EOF still needs explaining. So the code is correct and merged; the cutover is blocked on deploy-infra that's out of band from #22. Likely needs: (a) bring the Forgejo Actions runner back / confirm it's processing, (b) fix the CI deploy job (separate long-standing issue), or (c) run the manual `coily exec deploy` interactively on a host with working docker.io creds + registry push.

coilysiren commented

2026-06-06 04:01:09 +00:00

Author

Owner

Push now fails on VM->LAN routing, not TLS. After whitelisting the registry as insecure (OrbStack ~/.orbstack/config/docker.json + an engine restart via orbctl stop && orbctl start - note orb restart only restarts Linux machines, not the docker engine), the HTTPS mismatch is gone. But the blob push now dial tcp 192.168.0.194:30500: i/o timeout.

Diagnosis:

macOS host -> http://192.168.0.194:30500/v2/: rock solid (HTTP 200, ~5ms connect, 3/3).
OrbStack VM (docker run busybox wget http://192.168.0.194:30500/v2/): times out.
OrbStack network.subnet4 = 192.168.138.0/23 - no overlap with the LAN 192.168.0.0/24, so not a subnet conflict.
The engine restart emitted start VM: timed out waiting for VM to start, so the VM likely came up with degraded networking.

So the docker daemon (inside OrbStack's VM) can't reach the in-cluster registry on the LAN, even though the host can. The #22 image builds fine; it just can't be pushed from this Mac's OrbStack VM in its current state.

Paths forward:

Clean OrbStack restart (fully quit + relaunch the app, or reboot), re-test VM->registry, retry coily exec deploy. Cheapest if it's just degraded VM net.
docker save the image on the Mac (build works), transfer to kai-server over tailnet, docker load / push to the registry from kai-server where it's localhost. Bypasses VM->LAN entirely.
Fix the intended in-cluster CI path: the Forgejo Actions runner isn't picking up pushes (no run for 5197f8b/dcb1533) and the deploy job has always failed. This is the sustainable mechanism (build+push is node-local, no OrbStack), and would retire the manual local deploy.

**Push now fails on VM->LAN routing, not TLS.** After whitelisting the registry as insecure (OrbStack `~/.orbstack/config/docker.json` + an *engine* restart via `orbctl stop && orbctl start` - note `orb restart` only restarts Linux machines, not the docker engine), the HTTPS mismatch is gone. But the blob push now `dial tcp 192.168.0.194:30500: i/o timeout`. Diagnosis: - macOS **host** -> `http://192.168.0.194:30500/v2/`: rock solid (HTTP 200, ~5ms connect, 3/3). - OrbStack **VM** (`docker run busybox wget http://192.168.0.194:30500/v2/`): **times out**. - OrbStack `network.subnet4 = 192.168.138.0/23` - no overlap with the LAN `192.168.0.0/24`, so not a subnet conflict. - The engine restart emitted `start VM: timed out waiting for VM to start`, so the VM likely came up with degraded networking. So the docker daemon (inside OrbStack's VM) can't reach the in-cluster registry on the LAN, even though the host can. The #22 image builds fine; it just can't be pushed from this Mac's OrbStack VM in its current state. Paths forward: 1. Clean OrbStack restart (fully quit + relaunch the app, or reboot), re-test VM->registry, retry `coily exec deploy`. Cheapest if it's just degraded VM net. 2. `docker save` the image on the Mac (build works), transfer to kai-server over tailnet, `docker load` / push to the registry from kai-server where it's localhost. Bypasses VM->LAN entirely. 3. Fix the intended in-cluster CI path: the Forgejo Actions runner isn't picking up pushes (no run for 5197f8b/dcb1533) and the deploy job has always failed. This is the sustainable mechanism (build+push is node-local, no OrbStack), and would retire the manual local deploy.

coilysiren referenced this issue from a commit

2026-06-06 04:10:10 +00:00

fix(deploy): pin local docker build to linux/amd64

coilysiren closed this issue

2026-06-06 04:10:10 +00:00

coilysiren commented

2026-06-06 04:41:59 +00:00

Author

Owner

Cutover is live and verified. https://galaxy-gen.coilysiren.me now serves from stock caddy + mounted bundle.

Final state:

New pod coilysiren-galaxy-gen-app-* is 2/2 Running; the 11-day-old single-container pod is retired. rollout status = successfully rolled out.
Containers: serving = stock caddy:2-alpine (unmodified), initContainer = the bundle data image 192.168.0.194:30500/coilysiren-galaxy-gen:1cbaca9... (amd64), plus the ts sidecar.
/usr/share/caddy (the emptyDir the initContainer filled) holds the real bundle: *.module.wasm, index.html, index.js, worker + vendor chunks.
Ingress /healthz -> 200; / -> <title>galaxy-gen</title>; the real *.wasm -> 200 content-type: application/wasm + x-content-type-options: nosniff, so the Caddyfile traveled in the bundle and its rules are live.

What it took (all deploy-infra, none from the #22 code):

Architecture: local arm64 build pushed a manifest the amd64 node rejected -> fixed by pinning --platform linux/amd64 in the makefile (1cbaca9).
OrbStack: registry whitelisted as insecure + an engine restart (orbctl stop && start); the daemon also needed a clean OrbStack relaunch to restore VM->LAN routing.
kubectl context: the makefile .deploy targets kai-server:6443 (tailnet 100.69.164.66), which times out from this Mac. The apply was completed via coily ops kubectl (LAN 192.168.0.194:6443). So coily exec deploy's apply step still fails locally as-is - see follow-up.

**Cutover is live and verified.** `https://galaxy-gen.coilysiren.me` now serves from stock caddy + mounted bundle. Final state: - New pod `coilysiren-galaxy-gen-app-*` is `2/2 Running`; the 11-day-old single-container pod is retired. `rollout status` = successfully rolled out. - Containers: serving = **stock `caddy:2-alpine`** (unmodified), initContainer = the bundle data image `192.168.0.194:30500/coilysiren-galaxy-gen:1cbaca9...` (amd64), plus the ts sidecar. - `/usr/share/caddy` (the emptyDir the initContainer filled) holds the real bundle: `*.module.wasm`, `index.html`, `index.js`, worker + vendor chunks. - Ingress `/healthz` -> 200; `/` -> `<title>galaxy-gen</title>`; the real `*.wasm` -> 200 `content-type: application/wasm` + `x-content-type-options: nosniff`, so the Caddyfile traveled in the bundle and its rules are live. What it took (all deploy-infra, none from the #22 code): 1. Architecture: local arm64 build pushed a manifest the amd64 node rejected -> fixed by pinning `--platform linux/amd64` in the makefile (`1cbaca9`). 2. OrbStack: registry whitelisted as insecure + an *engine* restart (`orbctl stop && start`); the daemon also needed a clean OrbStack relaunch to restore VM->LAN routing. 3. kubectl context: the makefile `.deploy` targets `kai-server:6443` (tailnet `100.69.164.66`), which **times out** from this Mac. The apply was completed via `coily ops kubectl` (LAN `192.168.0.194:6443`). So `coily exec deploy`'s apply step still fails locally as-is - see follow-up.

coilysiren referenced this issue

2026-06-06 04:42:17 +00:00

Fix .deploy kubectl context: target LAN API endpoint, not the flaky tailnet route #25

coilysiren referenced this issue

2026-06-06 05:02:36 +00:00

Deploy-path infra: get CI green + document local-deploy host prereqs #26

coilysiren referenced this issue from a commit

2026-06-06 05:03:15 +00:00

fix(deploy): target the LAN k3s API in .deploy, not the flaky tailnet route