Reconcile Forgejo runner scopes for active deploy workflows #955

New issue

Closed

opened 2026-08-27 03:24:56 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-27 03:24:56 +00:00

Owner

Outcome

Restore runner visibility for every active trusted Forgejo workflow without recreating an instance-global or organization-wide deploy execution surface.

Parent boundary: coilysiren/inbox#280

Incident repair: #665

Current evidence

An audit after the scoped runner credential rollout checked 32 active repositories and 126 runs-on declarations.

All live scoped runner StatefulSets are ready and their ExternalSecrets are synced. No task was waiting at inspection time. The current exact-repository deploy runners serve only:

coilyco-bridge/deploy
coilyco-gaming/sirens-discord-ops

The following active main-push workflows still request runs-on: deploy but cannot see either runner:

coilyco-bridge/atlas
coilyco-flight-deck/bluesky-mcp
coilyco-flight-deck/lunch-money-k8s
coilyco-flight-deck/node-stats-mcp
coilyco-flight-deck/reddit-mcp
coilyco-flight-deck/ward-mcp
coilyco-gaming/eco-app
coilyco-gaming/factory-game-v3
coilyco-gaming/galaxy-gen
coilyco-gaming/steam-ops
coilysiren/website

coilysiren/website also requests runs-on: docker, but user-owned coilysiren repositories intentionally have no general runner scope.

The repaired Sirens workflow completed successfully in https://forgejo.coilysiren.me/coilyco-gaming/sirens-discord-ops/actions/runs/38.

Decision boundary

For each affected source repository:

Engineering retires or moves the deploy or publish workflow into coilyco-bridge/deploy when that repository already owns the canonical deployment.
Otherwise, Ops creates the narrowest exact-repository publisher runner. Ops does not grant organization-wide deploy.
Engineering keeps image publication separate from cluster mutation when a workflow needs only Docker plus the package-write credential.

Kai decides whether coilysiren/website receives exact-repository Docker and publisher scopes or moves its Forgejo workflows under an already scoped owner.

Complete when

Every active Forgejo workflow label maps to a runner visible at that repository scope.
No user-controlled or unrelated repository can select a privileged runner.
Engineering removes duplicate source-repository deployment workflows where coilyco-bridge/deploy is canonical.
Representative general, image-build, deploy or publish, and website workflows pass after the change.
A fleet audit reports zero unmatched runs-on labels.

## Outcome Restore runner visibility for every active trusted Forgejo workflow without recreating an instance-global or organization-wide deploy execution surface. Parent boundary: https://forgejo.coilysiren.me/coilysiren/inbox/issues/280 Incident repair: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/665 ## Current evidence An audit after the scoped runner credential rollout checked 32 active repositories and 126 `runs-on` declarations. All live scoped runner StatefulSets are ready and their ExternalSecrets are synced. No task was waiting at inspection time. The current exact-repository deploy runners serve only: * `coilyco-bridge/deploy` * `coilyco-gaming/sirens-discord-ops` The following active main-push workflows still request `runs-on: deploy` but cannot see either runner: * `coilyco-bridge/atlas` * `coilyco-flight-deck/bluesky-mcp` * `coilyco-flight-deck/lunch-money-k8s` * `coilyco-flight-deck/node-stats-mcp` * `coilyco-flight-deck/reddit-mcp` * `coilyco-flight-deck/ward-mcp` * `coilyco-gaming/eco-app` * `coilyco-gaming/factory-game-v3` * `coilyco-gaming/galaxy-gen` * `coilyco-gaming/steam-ops` * `coilysiren/website` `coilysiren/website` also requests `runs-on: docker`, but user-owned `coilysiren` repositories intentionally have no general runner scope. The repaired Sirens workflow completed successfully in https://forgejo.coilysiren.me/coilyco-gaming/sirens-discord-ops/actions/runs/38. ## Decision boundary For each affected source repository: 1. Engineering retires or moves the deploy or publish workflow into `coilyco-bridge/deploy` when that repository already owns the canonical deployment. 2. Otherwise, Ops creates the narrowest exact-repository publisher runner. Ops does not grant organization-wide `deploy`. 3. Engineering keeps image publication separate from cluster mutation when a workflow needs only Docker plus the package-write credential. Kai decides whether `coilysiren/website` receives exact-repository Docker and publisher scopes or moves its Forgejo workflows under an already scoped owner. ## Complete when * Every active Forgejo workflow label maps to a runner visible at that repository scope. * No user-controlled or unrelated repository can select a privileged runner. * Engineering removes duplicate source-repository deployment workflows where `coilyco-bridge/deploy` is canonical. * Representative general, image-build, deploy or publish, and website workflows pass after the change. * A fleet audit reports zero unmatched `runs-on` labels.

coilyco-ops referenced this issue

2026-08-28 21:12:35 +00:00

forgejo-runner-deploy-atlas has crash-looped for 14h against a repository that does not exist #834

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops commented

2026-08-29 04:20:54 +00:00

Author

Owner

Resolved. Verified live 2026-08-29 ~04:20Z against the kai-server API.

This issue listed eleven active main-push workflows requesting runs-on: deploy that could see no runner. Every one of them now resolves.

Exact-repository deploy runners running on kai-server, all 1/1, all recycled in the 2026-08-28T16:15Z window:

forgejo-runner-deploy-bluesky-mcp-0
forgejo-runner-deploy-lunch-money-k8s-0
forgejo-runner-deploy-node-stats-mcp-0
forgejo-runner-deploy-ward-mcp-0
forgejo-runner-deploy-eco-app-0
forgejo-runner-deploy-factory-game-v3-0
forgejo-runner-deploy-galaxy-gen-0
forgejo-runner-deploy-steam-ops-0
forgejo-runner-deploy-website-0

Plus forgejo-runner-deploy-agent-proxy-0, -sirens-echo-0, -sirens-discord-ops-0 and -scoped-0, which this issue did not list.

The two names on the list with no runner are resolved rather than outstanding:

coilyco-bridge/atlas - the runner and the repository are both gone. #834 closed on the StatefulSet no longer existing.
coilyco-flight-deck/reddit-mcp - retired by coilyco-bridge/deploy#608.

coilysiren/website received its own exact-repository runner, which is the second of the two options this issue put to Kai. The decision is recorded by the deployed state.

No organization-wide deploy scope was created. kai-server carries only exact-repository deploy runners plus the DinD-free tap writer, and the general pools moved to ser8 under #693.

What this close does not assert

The final acceptance line, "a fleet audit reports zero unmatched runs-on labels", was not re-run. What was verified is that every repository this issue named now has a runner visible at its scope. A fresh unmatched label filed since 2026-08-27 would not have been caught here.

Closing as resolved. Reopen freely if the audit finds a straggler.

## Resolved. Verified live 2026-08-29 ~04:20Z against the kai-server API. This issue listed eleven active main-push workflows requesting `runs-on: deploy` that could see no runner. Every one of them now resolves. **Exact-repository deploy runners running on kai-server**, all `1/1`, all recycled in the 2026-08-28T16:15Z window: * `forgejo-runner-deploy-bluesky-mcp-0` * `forgejo-runner-deploy-lunch-money-k8s-0` * `forgejo-runner-deploy-node-stats-mcp-0` * `forgejo-runner-deploy-ward-mcp-0` * `forgejo-runner-deploy-eco-app-0` * `forgejo-runner-deploy-factory-game-v3-0` * `forgejo-runner-deploy-galaxy-gen-0` * `forgejo-runner-deploy-steam-ops-0` * `forgejo-runner-deploy-website-0` Plus `forgejo-runner-deploy-agent-proxy-0`, `-sirens-echo-0`, `-sirens-discord-ops-0` and `-scoped-0`, which this issue did not list. The two names on the list with no runner are resolved rather than outstanding: * `coilyco-bridge/atlas` - the runner and the repository are both gone. #834 closed on the StatefulSet no longer existing. * `coilyco-flight-deck/reddit-mcp` - retired by `coilyco-bridge/deploy#608`. `coilysiren/website` received its own exact-repository runner, which is the second of the two options this issue put to Kai. The decision is recorded by the deployed state. **No organization-wide `deploy` scope was created.** kai-server carries only exact-repository deploy runners plus the DinD-free tap writer, and the general pools moved to ser8 under #693. ### What this close does not assert The final acceptance line, "a fleet audit reports zero unmatched `runs-on` labels", was not re-run. What was verified is that every repository this issue named now has a runner visible at its scope. A fresh unmatched label filed since 2026-08-27 would not have been caught here. Closing as resolved. Reopen freely if the audit finds a straggler.

coilyco-ops closed this issue

2026-08-29 04:21:00 +00:00

coilyco-ops referenced this issue

2026-08-29 04:22:59 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981