coilyco-flight-deck/ward

Archived

Watch

Ward release ops #1117

Closed

opened 2026-07-11 22:01:01 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-07-11 22:01:01 +00:00

Owner

The problem

See: coilyco-flight-deck/agentic-os#469 (comment)

We probably want to cut the tags off of release, not off of main. But it should be chained, and the full CI suit should run twice (merging to release triggers ci same way main does)

Then we can add the super heavy test that asks goose to one shot a ticket inside of k3s, against the release branch.

Release branch should merge queue instead of overlap and cancel. Or whatever the name for those release ops methods are. I don't think there's a simple flag that flips that on for free, we probably need a dedicated runner for it. Or, probably better, I think we have a (which will now be incorrectly named) runner called "tap runner"

All of this we should do for aos as well

Proposed change

See above

Alternatives considered

No response

Before filing

I searched existing issues and this is not a duplicate.
This stays within ward's scope (the dev-verb gate / agent driver), not a personal-infra or downstream-repo verb.

### The problem See: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/469#issuecomment-31878 We probably want to cut the tags off of release, not off of main. But it should be chained, and the full CI suit should run *twice* (merging to release triggers ci same way main does) Then we can add the super heavy test that asks goose to one shot a ticket inside of k3s, against the release branch. Release branch should merge queue instead of overlap and cancel. Or whatever the name for those release ops methods are. I don't think there's a simple flag that flips that on for free, we probably need a dedicated runner for it. Or, probably better, I think we have a (which will now be incorrectly named) runner called "tap runner" All of this we should do for aos as well ### Proposed change See above ### Alternatives considered _No response_ ### Before filing - [x] I searched existing issues and this is not a duplicate. - [x] This stays within ward's scope (the dev-verb gate / agent driver), not a personal-infra or downstream-repo verb.

coilysiren referenced this issue from coilyco-flight-deck/agentic-os

2026-07-11 22:01:46 +00:00

aos images should publish a "latest" tag #469

coilysiren referenced this issue from a commit

2026-07-12 05:59:47 +00:00

feat(release): two-stage release - gate-green main promotes to a queued release branch

coilysiren referenced this issue from a commit

2026-07-12 06:10:47 +00:00

feat(release): two-stage release - gate-green main promotes to a queued release branch

coilysiren referenced this issue

2026-07-12 06:11:04 +00:00

feat(release): two-stage release - gate-green main promotes to a queued release branch #1126

coilysiren commented

2026-07-12 06:44:20 +00:00

Author

Owner

Two-stage flow live on ward, one blocker found: the promote token cannot enqueue workflows

Landed tonight (2026-07-12 UTC):

ward#1126 - promote.yml (stage 1) + release.yml on release pushes under a no-cancel concurrency queue (Forgejo v14+ semantics, no dedicated runner needed).
ward#1130 - github.server_url on this runner is the in-cluster http URL; the promote push URL now keeps the server''s own scheme.
aos#475 (open) - the same flow for aos, held until CI_RELEASE_TOKEN exists there.

Live result: promote gates green and fast-forwards release (runs 1831, 1833 - branch tip tracks main exactly). But release.yml never fires on those pushes: the repo activity feed shows the release-branch pushes with an empty actor. /forgejo/ci-release-token lacks read:user (GET /api/v1/user with it returns "token does not have at least one of required scope(s): [read:user]"), so Forgejo cannot attribute the push and silently enqueues nothing - the same class as the aos#240 job-token suppression, one layer deeper.

Unblock (Kai):

Remint /forgejo/ci-release-token as a PAT with write:repository + read:user, update SSM.
ward exec sync-actions-secrets (aos#479) - pushes it to aos + ward, plus the Telegram alert creds both repos were missing.
Next merge to either main then promotes AND releases end to end.

Until then: promote runs stay green, release keeps tracking gate-green main, and no tags are cut (ward releases paused at v0.630.0).

— Claude (she/her), via interactive session

## Two-stage flow live on ward, one blocker found: the promote token cannot enqueue workflows Landed tonight (2026-07-12 UTC): * ward#1126 - promote.yml (stage 1) + release.yml on `release` pushes under a no-cancel concurrency queue (Forgejo v14+ semantics, no dedicated runner needed). * ward#1130 - `github.server_url` on this runner is the in-cluster http URL; the promote push URL now keeps the server''s own scheme. * aos#475 (open) - the same flow for aos, held until CI_RELEASE_TOKEN exists there. **Live result:** promote gates green and fast-forwards `release` (runs 1831, 1833 - branch tip tracks main exactly). But release.yml never fires on those pushes: the repo activity feed shows the release-branch pushes with an **empty actor**. `/forgejo/ci-release-token` lacks `read:user` (`GET /api/v1/user` with it returns "token does not have at least one of required scope(s): [read:user]"), so Forgejo cannot attribute the push and silently enqueues nothing - the same class as the aos#240 job-token suppression, one layer deeper. **Unblock (Kai):** 1. Remint `/forgejo/ci-release-token` as a PAT with `write:repository` + `read:user`, update SSM. 2. `ward exec sync-actions-secrets` (aos#479) - pushes it to aos + ward, plus the Telegram alert creds both repos were missing. 3. Next merge to either main then promotes AND releases end to end. Until then: promote runs stay green, `release` keeps tracking gate-green main, and no tags are cut (ward releases paused at v0.630.0). — Claude (she/her), via interactive session

coilyco-ops commented

2026-07-12 21:41:20 +00:00

Owner

Two-stage flow unblocked end to end - root cause was one layer deeper than the token scopes

Done tonight (2026-07-12, claude-windows-kai-desktop-tower-bc51):

Reminted the promote PAT on the coilyco-ops bot - token endpoints take basic auth only and only the bot's password lives in SSM, so CI_RELEASE_TOKEN is now a bot PAT with write:repository + read:user, verified via GET /user, stashed to SSM. New verb: aos ward exec remint-ci-release-token.
ward exec sync-actions-secrets fanned it plus the Telegram creds to aos, ward, and cli-guard (cli-guard joined the mapping - it now runs the same two-stage flow, landed on its main).
Merged aos#475, and found the real blocker: the release push still enqueued nothing even with correct scopes. actions/checkout persists the job token as an http extraheader in the local git config, and that Authorization header overrides the oauth2 PAT in the promote push URL. The push landed as the internal Actions user, whose pushes Forgejo deliberately skips workflow enqueue for - the aos#240 class one layer deeper again. The read:user diagnosis was necessary for attribution but was never the whole story: the PAT was not being used at all.
Fix: persist-credentials: false on the promote checkout. Landed on aos, ward, cli-guard. Confirmed live on aos: promote green, release branch fast-forwarded, release.yml + ci.yml enqueued and running on the release ref.

Also scaled the stale forgejo/comfyui deployment to 0 per infrastructure#547 (875 restarts) - runner pods show no restarts in 2d3h.

Leaving this open until ward and cli-guard show their first tag cut off the release branch.

Claude (she/her)

## Two-stage flow unblocked end to end - root cause was one layer deeper than the token scopes Done tonight (2026-07-12, claude-windows-kai-desktop-tower-bc51): 1. **Reminted the promote PAT on the coilyco-ops bot** - token endpoints take basic auth only and only the bot's password lives in SSM, so CI_RELEASE_TOKEN is now a bot PAT with write:repository + read:user, verified via GET /user, stashed to SSM. New verb: aos `ward exec remint-ci-release-token`. 2. **`ward exec sync-actions-secrets`** fanned it plus the Telegram creds to aos, ward, and cli-guard (cli-guard joined the mapping - it now runs the same two-stage flow, landed on its main). 3. **Merged aos#475**, and found the real blocker: the release push still enqueued nothing even with correct scopes. **actions/checkout persists the job token as an http extraheader in the local git config, and that Authorization header overrides the oauth2 PAT in the promote push URL.** The push landed as the internal Actions user, whose pushes Forgejo deliberately skips workflow enqueue for - the aos#240 class one layer deeper again. The read:user diagnosis was necessary for attribution but was never the whole story: the PAT was not being used at all. 4. **Fix: `persist-credentials: false` on the promote checkout.** Landed on aos, ward, cli-guard. Confirmed live on aos: promote green, release branch fast-forwarded, release.yml + ci.yml enqueued and running on the release ref. Also scaled the stale forgejo/comfyui deployment to 0 per infrastructure#547 (875 restarts) - runner pods show no restarts in 2d3h. Leaving this open until ward and cli-guard show their first tag cut off the release branch. - Claude (she/her)

coilyco-ops commented

2026-07-12 22:16:19 +00:00

Owner

Shipped - all three repos released end to end off the release branch tonight

ward v0.631.0, cli-guard v0.96.0, aos v0.247.0 - each cut by stage 2 on a gate-green release push, under the no-cancel queue.
aos needed two re-dispatches on the way: run 1102 hit the shared runner's 15m cap (infrastructure#551) and run 1103 hit the registry's concurrent blob-upload race (infrastructure#552). Both are pre-existing infra flake classes, not two-stage defects - the recovery lever (workflow_dispatch against the release ref) worked as designed.
The remaining spec item (goose one-shots a ticket inside k3s against release, gating the tag) is carved out to ward#1135.

Closing: the two-stage flow specced here is live on ward, aos, and cli-guard.

Claude (she/her), claude-windows-kai-desktop-tower-bc51

## Shipped - all three repos released end to end off the release branch tonight * **ward v0.631.0**, **cli-guard v0.96.0**, **aos v0.247.0** - each cut by stage 2 on a gate-green `release` push, under the no-cancel queue. * aos needed two re-dispatches on the way: run 1102 hit the shared runner's 15m cap (infrastructure#551) and run 1103 hit the registry's concurrent blob-upload race (infrastructure#552). Both are pre-existing infra flake classes, not two-stage defects - the recovery lever (workflow_dispatch against the release ref) worked as designed. * The remaining spec item (goose one-shots a ticket inside k3s against release, gating the tag) is carved out to ward#1135. Closing: the two-stage flow specced here is live on ward, aos, and cli-guard. - Claude (she/her), claude-windows-kai-desktop-tower-bc51

coilyco-ops closed this issue

2026-07-12 22:16:21 +00:00

Commenting is not possible because the repository is archived.

No milestone

No project

No assignees

2 participants

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#1117

No description provided.

Rows
Columns