Fix provision-forgejo-runner-registration-token.sh: stale --output/--query flags break the token-restore bootstrap #539

New issue

Closed

opened 2026-07-10 17:27:47 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-10 17:27:47 +00:00

Member

Bug

scripts/provision-forgejo-runner-registration-token.sh is broken against the current ward and fails silently at exactly the moment it is needed - restoring the Forgejo runner registration token after it goes missing from SSM (e.g. the post-reboot incident in infrastructure#538).

The script mints the token with:

generate_args=(ward ops forgejo actions generate-runner-token)
token="$("${generate_args[@]}" --output text --query token)"

But ward ops forgejo actions generate-runner-token is an exec-dialect verb (it wraps the in-pod forgejo CLI via kubectl exec), not a spec-driven HTTP verb, so it does not define --output / --query. The run dies with:

Minting Forgejo runner registration token 'forgejo-runner-registration'...
Incorrect Usage: flag provided but not defined: -output
ward: flag provided but not defined: -output

Reproduced live on 2026-07-10 during the infrastructure#538 recovery: with /forgejo/runner-registration-token absent from SSM, forgejo-runner-3 was stuck Init:CreateContainerConfigError (couldn't find key runner-token in Secret forgejo/forgejo-runner-secrets), and the operator bootstrap script could not re-mint the token to fix it. The operator had to hand-edit the script to kubectl -n forgejo exec deploy/forgejo -- forgejo actions generate-runner-token to recover.

Do

Fix the mint call to match the current ward ops forgejo actions generate-runner-token output contract. Check what the verb actually prints today (--help + a bare run), then extract the token from that output instead of the non-existent --output text --query token flags. A robust extraction is grep -Eo '[0-9a-f]{40}' | head -1, which works whether the verb prints the bare token or wraps it in a line.
Keep the existing ^[0-9a-f]{40}$ validation and the SecureString ward ops aws ssm put-parameter store unchanged.
If the ward verb is not reliably present on operator shells, add a documented kubectl -n forgejo exec deploy/forgejo -- forgejo actions generate-runner-token fallback (the in-pod path the wrapper is meant to abstract), guarded so the ward path is preferred when available.
Verify end to end: run the script, confirm the token lands in SSM /forgejo/runner-registration-token, the forgejo-runner-secrets / forgejo-runner-deploy-secrets ExternalSecrets sync the runner-token key, and a fresh forgejo-runner-* pod reaches 2/2 Running.

Acceptance

bash scripts/provision-forgejo-runner-registration-token.sh mints and stores a valid registration token on a current-ward operator shell with no hand-editing.
The docs that point at this bootstrap path (docs/cd-runner.md, docs/FEATURES.md runner entries) still describe the actual invocation.

Context

Surfaced during the infrastructure#538 post-reboot recovery. The runner-token restore is the operator path that unwedges the whole Forgejo Actions runner fleet after the SSM param is lost, so a silent failure here blocks CI recovery for every repo. This is a headless code fix - the script and its extraction logic, no host access required to author it. Refs: infrastructure#538, deploy/forgejo-runner.yml, deploy/forgejo-runner-deploy.yml, docs/cd-runner.md.

## Bug `scripts/provision-forgejo-runner-registration-token.sh` is broken against the current `ward` and fails **silently at exactly the moment it is needed** - restoring the Forgejo runner registration token after it goes missing from SSM (e.g. the post-reboot incident in infrastructure#538). The script mints the token with: ```bash generate_args=(ward ops forgejo actions generate-runner-token) token="$("${generate_args[@]}" --output text --query token)" ``` But `ward ops forgejo actions generate-runner-token` is an **exec-dialect** verb (it wraps the in-pod `forgejo` CLI via `kubectl exec`), not a spec-driven HTTP verb, so it does not define `--output` / `--query`. The run dies with: ``` Minting Forgejo runner registration token 'forgejo-runner-registration'... Incorrect Usage: flag provided but not defined: -output ward: flag provided but not defined: -output ``` Reproduced live on 2026-07-10 during the infrastructure#538 recovery: with `/forgejo/runner-registration-token` absent from SSM, `forgejo-runner-3` was stuck `Init:CreateContainerConfigError` (`couldn't find key runner-token in Secret forgejo/forgejo-runner-secrets`), and the operator bootstrap script could not re-mint the token to fix it. The operator had to hand-edit the script to `kubectl -n forgejo exec deploy/forgejo -- forgejo actions generate-runner-token` to recover. ## Do - Fix the mint call to match the current `ward ops forgejo actions generate-runner-token` output contract. Check what the verb actually prints today (`--help` + a bare run), then extract the token from that output instead of the non-existent `--output text --query token` flags. A robust extraction is `grep -Eo '[0-9a-f]{40}' | head -1`, which works whether the verb prints the bare token or wraps it in a line. - Keep the existing `^[0-9a-f]{40}$` validation and the SecureString `ward ops aws ssm put-parameter` store unchanged. - If the ward verb is not reliably present on operator shells, add a documented `kubectl -n forgejo exec deploy/forgejo -- forgejo actions generate-runner-token` fallback (the in-pod path the wrapper is meant to abstract), guarded so the ward path is preferred when available. - Verify end to end: run the script, confirm the token lands in SSM `/forgejo/runner-registration-token`, the `forgejo-runner-secrets` / `forgejo-runner-deploy-secrets` ExternalSecrets sync the `runner-token` key, and a fresh `forgejo-runner-*` pod reaches `2/2 Running`. ## Acceptance - `bash scripts/provision-forgejo-runner-registration-token.sh` mints and stores a valid registration token on a current-ward operator shell with no hand-editing. - The docs that point at this bootstrap path (`docs/cd-runner.md`, `docs/FEATURES.md` runner entries) still describe the actual invocation. ## Context Surfaced during the infrastructure#538 post-reboot recovery. The runner-token restore is the operator path that unwedges the whole Forgejo Actions runner fleet after the SSM param is lost, so a silent failure here blocks CI recovery for every repo. This is a headless code fix - the script and its extraction logic, no host access required to author it. Refs: infrastructure#538, `deploy/forgejo-runner.yml`, `deploy/forgejo-runner-deploy.yml`, `docs/cd-runner.md`.

coilyco-ops added the

headless

labels

2026-07-10 17:27:47 +00:00

coilyco-ops commented

2026-07-10 17:27:58 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-claude-infrastructure-539 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness claude (reserved 2026-07-10T17:27:57Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#539 · branch issue-539 · harness claude · workflow direct-main
Run: engineer-claude-infrastructure-539 · ward v0.584.0 · dispatched 2026-07-10T17:27:57Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0).

— Claude (she/her), via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-claude-infrastructure-539` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness claude` (reserved 2026-07-10T17:27:57Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#539` · branch `issue-539` · harness `claude` · workflow `direct-main` - **Run:** `engineer-claude-infrastructure-539` · ward `v0.584.0` · dispatched `2026-07-10T17:27:57Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0). </details> </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 17:35:04 +00:00

fix(scripts): mint runner token without stale --output/--query flags

coilyco-ops closed this issue

2026-07-10 17:35:04 +00:00