Restore repository-scoped image publishers after runner migration

coilyco-ops commented

2026-07-28 22:25:33 +00:00

Member

Incident

The completed scoped-runner migration retired every global deploy:host registration after canarying only coilyco-bridge/deploy and coilyco-gaming/sirens-discord-ops. A canonical workflow inventory across all 29 active repositories found twelve repositories selecting runs-on: deploy. Ten image publishers therefore have no eligible runner, including the Eco application publisher.

Outcome

Keep coilyco-bridge/deploy as the sole runner with Kubernetes deployment identity.
Give all eleven existing image publishers exact-repository runner registrations.
Give publisher runners Docker, REGISTRY_TOKEN, and FORGEJO_EGRESS_PROXY, without a Kubernetes service-account token, AWS credentials, home-network values, or private-source read credentials.
Preserve the existing deploy workflow label during the compatibility migration.
Provision distinct registration-token paths through the guarded file-source flow.
Reconcile the full publisher roster through Flux and verify waiting jobs start.
Update validation, token inventory, scoped-runner documentation, and FEATURES.

Publisher roster

coilyco-flight-deck/bluesky-mcp
coilyco-flight-deck/lunch-money-k8s
coilyco-flight-deck/node-stats-mcp
coilyco-flight-deck/reddit-mcp
coilyco-flight-deck/ward-mcp
coilyco-bridge/atlas
coilyco-gaming/eco-app
coilyco-gaming/factory-game-v3
coilyco-gaming/galaxy-gen
coilyco-gaming/sirens-discord-ops
coilyco-gaming/steam-ops

## Incident The completed scoped-runner migration retired every global `deploy:host` registration after canarying only `coilyco-bridge/deploy` and `coilyco-gaming/sirens-discord-ops`. A canonical workflow inventory across all 29 active repositories found twelve repositories selecting `runs-on: deploy`. Ten image publishers therefore have no eligible runner, including the Eco application publisher. ## Outcome * Keep `coilyco-bridge/deploy` as the sole runner with Kubernetes deployment identity. * Give all eleven existing image publishers exact-repository runner registrations. * Give publisher runners Docker, `REGISTRY_TOKEN`, and `FORGEJO_EGRESS_PROXY`, without a Kubernetes service-account token, AWS credentials, home-network values, or private-source read credentials. * Preserve the existing `deploy` workflow label during the compatibility migration. * Provision distinct registration-token paths through the guarded file-source flow. * Reconcile the full publisher roster through Flux and verify waiting jobs start. * Update validation, token inventory, scoped-runner documentation, and FEATURES. ## Publisher roster * coilyco-flight-deck/bluesky-mcp * coilyco-flight-deck/lunch-money-k8s * coilyco-flight-deck/node-stats-mcp * coilyco-flight-deck/reddit-mcp * coilyco-flight-deck/ward-mcp * coilyco-bridge/atlas * coilyco-gaming/eco-app * coilyco-gaming/factory-game-v3 * coilyco-gaming/galaxy-gen * coilyco-gaming/sirens-discord-ops * coilyco-gaming/steam-ops Related: #653, #658, #665, coilyco-gaming/eco-app#142.

coilyco-ops referenced this issue from a commit

2026-07-28 22:32:27 +00:00

fix(runners): restore repository image publishers

coilyco-ops referenced this issue from a commit

2026-07-28 22:36:26 +00:00

fix(runners): restore repository image publishers

coilyco-ops referenced this issue from a commit

2026-07-28 22:45:20 +00:00

fix(runners): harden publisher rollout capacity

coilyco-ops commented

2026-07-28 22:47:02 +00:00

Author

Member

Rollout checkpoint: commits 6b8e3bd and 4f272e3 are on canonical main. Ten missing exact-repository registration tokens are stored in SSM through newline-free file sources. Flux created all publisher resources and every ExternalSecret reports SecretSynced. Live verification found kai-server at its kubelet 110-pod ceiling. Six publisher pods scheduled and four remain Pending with Too many pods. The tracked k3s-config role now owns the systemd start wrapper and raises max-pods to 180 while publisher idle memory requests are reduced. Repository render and full pre-commit pass. Remaining attended action: ward exec ansible-sync apply tags=k3s-config hosts=kai-server ask_pass. After the restart, Ops must verify node pod capacity, all eleven publisher StatefulSets, repository runner visibility, and the Eco workflow before closing this issue.

Rollout checkpoint: commits 6b8e3bd and 4f272e3 are on canonical main. Ten missing exact-repository registration tokens are stored in SSM through newline-free file sources. Flux created all publisher resources and every ExternalSecret reports SecretSynced. Live verification found kai-server at its kubelet 110-pod ceiling. Six publisher pods scheduled and four remain Pending with Too many pods. The tracked k3s-config role now owns the systemd start wrapper and raises max-pods to 180 while publisher idle memory requests are reduced. Repository render and full pre-commit pass. Remaining attended action: ward exec ansible-sync apply tags=k3s-config hosts=kai-server ask_pass. After the restart, Ops must verify node pod capacity, all eleven publisher StatefulSets, repository runner visibility, and the Eco workflow before closing this issue.

coilyco-ops referenced this issue from a commit

2026-07-28 23:07:41 +00:00

fix(ansible): preserve existing Traefik config

coilyco-ops referenced this issue from a commit

2026-07-28 23:09:34 +00:00

fix(ansible): flush core k3s handlers early

coilyco-ops referenced this issue

2026-07-29 00:24:49 +00:00

Autoscale scoped Forgejo runners to a four-pod idle floor #682

coilyco-ops referenced this issue from coilyco-gaming/eco-app

2026-07-29 00:57:58 +00:00

Fix mod publishers invoking Ward outside the repository #167

coilyco-ops commented

2026-07-29 00:58:21 +00:00

Author

Member

Live rollout update: kai-server now reports pod capacity 180, all eleven repository-scoped publisher runners and the scoped deploy runner are Running 2/2, and Ops restarted all twelve after their ExternalSecrets refreshed the rotated registry credential.

Eco verification run 143 passed test, frontend, and build-image. Docker authentication and immutable app-image publication are restored. The four downstream mod publishers fail later inside eco-app because Ward is invoked from an extracted temporary directory without a reachable repository config. coilyco-gaming/eco-app#167 owns that repository fix.

Keep this issue open until eco-app#167 lands and a full Eco publish run returns green. The original scheduling and registry-credential blockers are cleared.

Evidence: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/143

Live rollout update: kai-server now reports pod capacity 180, all eleven repository-scoped publisher runners and the scoped deploy runner are `Running 2/2`, and Ops restarted all twelve after their ExternalSecrets refreshed the rotated registry credential. Eco verification run 143 passed test, frontend, and `build-image`. Docker authentication and immutable app-image publication are restored. The four downstream mod publishers fail later inside eco-app because Ward is invoked from an extracted temporary directory without a reachable repository config. coilyco-gaming/eco-app#167 owns that repository fix. Keep this issue open until eco-app#167 lands and a full Eco publish run returns green. The original scheduling and registry-credential blockers are cleared. Evidence: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/143

coilyco-ops added the

labels

2026-07-29 03:46:37 +00:00

coilyco-ops commented

2026-07-30 08:20:02 +00:00

Author

Member

Closing verification is complete.

kai-server reports pod capacity 180 and Ready with DiskPressure false.
All eleven repository-scoped publisher runners and the scoped eco-app deploy runner are Running 2/2.
Eco-app canonical main commit 3093efe89381cca84b6b04d6e351f1509a68e712 completed Forgejo Actions run 148 successfully.
test, frontend, build-image, and all four mod package publishers succeeded.

The scheduling, registration, registry authentication, and downstream publication blockers are cleared. Evidence: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/148

Closing verification is complete. * kai-server reports pod capacity 180 and Ready with DiskPressure false. * All eleven repository-scoped publisher runners and the scoped eco-app deploy runner are Running 2/2. * Eco-app canonical main commit 3093efe89381cca84b6b04d6e351f1509a68e712 completed Forgejo Actions run 148 successfully. * test, frontend, build-image, and all four mod package publishers succeeded. The scheduling, registration, registry authentication, and downstream publication blockers are cleared. Evidence: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/148

coilyco-ops closed this issue

2026-07-30 08:20:07 +00:00

Rows
Columns

Restore repository-scoped image publishers after runner migration #675

Incident

Outcome

Publisher roster