coilyco-flight-deck/ward

Fork 0

fix(release): bump-formula stalled since v0.96.0 - homebrew-tap not bumped for v0.97-v0.102 #237

New issue

Closed

opened 2026-06-23 09:02:22 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-06-23 09:02:22 +00:00

Member

Symptom

brew update && brew upgrade ward is a no-op - it reports 0.96.0 already installed. The ward binary self-reports a newer release (v0.102.0) because it checks the release endpoint directly, but the Homebrew tap formula is stuck at 0.96.0, so brew has nothing newer to install. Every host on the brew surface is pinned at 0.96.0 and cannot upgrade.

Evidence (the bump-formula step is stalled, not merely lagging)

Tap formula coilyco-flight-deck/homebrew-tap/Formula/ward.rb declares stable 0.96.0.
homebrew-tap repo updated_at = 2026-06-19T10:29:55Z - seconds after the v0.96.0 release (2026-06-19T10:29:43Z) - and untouched since.
Six releases have shipped since with zero tap updates:
- v0.97.0 2026-06-23T07:17:06Z
- v0.98.0 2026-06-23T08:22:35Z
- v0.99.0 2026-06-23T08:31:07Z
- v0.100.0 2026-06-23T08:41:05Z
- v0.101.0 2026-06-23T08:41:31Z
- v0.102.0 2026-06-23T08:55:13Z
The oldest unbumped release (v0.97.0) is ~1h43m old as of filing, so this is a failure, not in-flight latency.

Last good bump: v0.96.0 on 2026-06-19. The breakage landed sometime between then and the v0.97.0 release.

Leading hypothesis (check first)

A FORGEJO_TOKEN -> ward-kdl migration is in flight across the fleet (agentic-os landed its half in e1ff479 "route forgejo I/O through ward-kdl, drop FORGEJO_TOKEN"). If ward's bump-formula step authenticated to write Formula/ward.rb in coilyco-flight-deck/homebrew-tap (Contents API) using FORGEJO_TOKEN, and that token was dropped or rotated as part of the same migration, the formula write would fail - silently if the step does not gate the release on its exit code. The timing fits exactly. Verify before assuming.

Where to look (front-loaded - do not guess)

ward's release pipeline: .forgejo/workflows/ (or .github/workflows/), the release workflow and the bump-formula job/step.
How that step authenticates to coilyco-flight-deck/homebrew-tap - which token/secret, and whether a FORGEJO_TOKEN reference survived the ward-kdl migration.
Whether the bump step's failure is swallowed (no set -e / unchecked API status) - it should fail the release loudly.

Two outcomes - one is NOT agent-fixable

Workflow rewiring (agent can fix): the YAML still references a removed token / wrong auth path and needs pointing at the ward-kdl-routed credential.
Missing CI secret (human only): if the fix is re-provisioning a Forgejo Actions secret, an agent cannot mint or install it. Flag it for Kai and stop - do not attempt to put a secret yourself.

Acceptance

A new ward release bumps Formula/ward.rb in homebrew-tap automatically.
The tap is backfilled to v0.102.0 so brew update && brew upgrade ward installs the current release.
The bump step fails the release loudly on a write error, so this cannot silently recur.

## Symptom `brew update && brew upgrade ward` is a no-op - it reports `0.96.0 already installed`. The ward binary self-reports a newer release (`v0.102.0`) because it checks the release endpoint directly, but the Homebrew tap formula is stuck at `0.96.0`, so brew has nothing newer to install. Every host on the brew surface is pinned at 0.96.0 and cannot upgrade. ## Evidence (the bump-formula step is stalled, not merely lagging) - Tap formula `coilyco-flight-deck/homebrew-tap/Formula/ward.rb` declares `stable 0.96.0`. - `homebrew-tap` repo `updated_at` = **2026-06-19T10:29:55Z** - seconds after the v0.96.0 release (2026-06-19T10:29:43Z) - and untouched since. - Six releases have shipped since with **zero** tap updates: - v0.97.0 2026-06-23T07:17:06Z - v0.98.0 2026-06-23T08:22:35Z - v0.99.0 2026-06-23T08:31:07Z - v0.100.0 2026-06-23T08:41:05Z - v0.101.0 2026-06-23T08:41:31Z - v0.102.0 2026-06-23T08:55:13Z - The oldest unbumped release (v0.97.0) is ~1h43m old as of filing, so this is a failure, not in-flight latency. Last good bump: v0.96.0 on 2026-06-19. The breakage landed sometime between then and the v0.97.0 release. ## Leading hypothesis (check first) A `FORGEJO_TOKEN` -> ward-kdl migration is in flight across the fleet (agentic-os landed its half in `e1ff479` "route forgejo I/O through ward-kdl, drop FORGEJO_TOKEN"). If ward's `bump-formula` step authenticated to write `Formula/ward.rb` in `coilyco-flight-deck/homebrew-tap` (Contents API) using `FORGEJO_TOKEN`, and that token was dropped or rotated as part of the same migration, the formula write would fail - silently if the step does not gate the release on its exit code. The timing fits exactly. **Verify before assuming.** ## Where to look (front-loaded - do not guess) - ward's release pipeline: `.forgejo/workflows/` (or `.github/workflows/`), the `release` workflow and the `bump-formula` job/step. - How that step authenticates to `coilyco-flight-deck/homebrew-tap` - which token/secret, and whether a `FORGEJO_TOKEN` reference survived the ward-kdl migration. - Whether the bump step's failure is swallowed (no `set -e` / unchecked API status) - it should fail the release loudly. ## Two outcomes - one is NOT agent-fixable 1. **Workflow rewiring** (agent can fix): the YAML still references a removed token / wrong auth path and needs pointing at the ward-kdl-routed credential. 2. **Missing CI secret** (human only): if the fix is re-provisioning a Forgejo Actions secret, an agent cannot mint or install it. Flag it for Kai and stop - do not attempt to put a secret yourself. ## Acceptance - A new ward release bumps `Formula/ward.rb` in `homebrew-tap` automatically. - The tap is backfilled to v0.102.0 so `brew update && brew upgrade ward` installs the current release. - The bump step fails the release loudly on a write error, so this cannot silently recur.

coilyco-ops commented

2026-06-23 09:02:42 +00:00

Author

Member

🔒 Reserved by ward agent claude — container ward-ward-issue-237-claude-c37da700 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-23T09:02:41Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent claude` — container `ward-ward-issue-237-claude-c37da700` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-23T09:02:41Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-06-24 04:06:14 +00:00

Author

Member

🔒 Reserved by ward agent claude — container ward-ward-issue-237-claude-c615f345 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-24T04:06:13Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent claude` — container `ward-ward-issue-237-claude-c615f345` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-24T04:06:13Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilyco-ops added the

interactive

labels

2026-06-24 04:15:56 +00:00

coilyco-ops commented

2026-06-24 04:35:33 +00:00

Author

Member

🔒 Reserved by ward agent claude — container ward-ward-issue-237-claude-dcba30cf on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-24T04:35:32Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent claude` — container `ward-ward-issue-237-claude-dcba30cf` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-24T04:35:32Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-06-24 04:50:47 +00:00

fix(release): fail the tap bump loudly when the write does not land (ward#237)

coilysiren closed this issue

2026-06-24 04:51:38 +00:00

coilysiren commented

2026-06-24 04:52:51 +00:00

Owner

Fixed the swallow-the-error half of this in 1ddd726 (released as v0.107.0). Root-cause + what remains:

Confirmed root cause

Not a code regression. .forgejo/workflows/release.yml was unchanged across the whole stall (last touched 2026-06-17, well before v0.96.0). The tell: both bump-tap-formula and publish-binaries flipped from green to red at the same release — v0.96.0 (run 299) was the last green of each, v0.97.0 (run 301) the first red of each — and they share nothing but a Forgejo write credential. So a credential that both jobs depend on broke after v0.96.0 (the FORGEJO_TOKEN migration window in the issue body lines up). The formula-rewrite logic itself is fine: replaying it against the live tap produces a correct url+sha256 diff for v0.102.0.

This is still true today: every release v0.97.0 → v0.107.0 has failed at bump-tap-formula, and the tap is still pinned at v0.96.0.

What shipped (acceptance #3 — fail loudly)

bump-tap-formula no longer lets a write failure hide behind a green release:

set -euo pipefail so a failed tarball fetch can't hash an empty body into a bogus sha256 and ship a broken formula;
reject any sha256 that isn't a 64-hex digest before writing;
annotate + fail on a non-zero git push, naming the likely cause;
read the tap back after pushing and assert main serves the new tag — a push that reports success but doesn't land now fails the release.

Verified live: the v0.107.0 release (run 321) bump-tap-formula failed loudly instead of silently.

What remains (acceptance #1 + #2 — gated on infra)

The bump is idempotent and backfilling — it rewrites url/sha256 against whatever the live tap holds — so the next green release auto-advances the tap to the latest tag, no manual catch-up needed. But it can't go green until the underlying write credential is restored:

Operator action: restore the tap-writer forgejo-runner's git credential (the infrastructure deploy that provisions forgejo-runner-tap-writer.yml's credential helper). The same broken credential is also why publish-binaries upload fails — both recover together.

I did not hand-edit coilyco-flight-deck/homebrew-tap to backfill it: that's a different repo, outside this container's autonomy wall (infra/other-repo writes). Once the credential is back, the first release backfills the tap and brew upgrade ward works again.

docs/release.md now documents this failure mode + recovery.

Fixed the swallow-the-error half of this in 1ddd726 (released as v0.107.0). Root-cause + what remains: ## Confirmed root cause Not a code regression. `.forgejo/workflows/release.yml` was unchanged across the whole stall (last touched 2026-06-17, well before v0.96.0). The tell: **both** `bump-tap-formula` *and* `publish-binaries` flipped from green to red at the *same* release — v0.96.0 (run 299) was the last green of each, v0.97.0 (run 301) the first red of each — and they share nothing but a Forgejo write credential. So a credential that both jobs depend on broke after v0.96.0 (the `FORGEJO_TOKEN` migration window in the issue body lines up). The formula-rewrite logic itself is fine: replaying it against the live tap produces a correct `url`+`sha256` diff for v0.102.0. This is still true today: every release v0.97.0 → v0.107.0 has failed at `bump-tap-formula`, and the tap is still pinned at v0.96.0. ## What shipped (acceptance #3 — fail loudly) `bump-tap-formula` no longer lets a write failure hide behind a green release: - `set -euo pipefail` so a failed tarball fetch can't hash an empty body into a bogus `sha256` and ship a broken formula; - reject any `sha256` that isn't a 64-hex digest before writing; - annotate + fail on a non-zero `git push`, naming the likely cause; - **read the tap back after pushing and assert `main` serves the new tag** — a push that reports success but doesn't land now fails the release. Verified live: the v0.107.0 release (run 321) `bump-tap-formula` failed *loudly* instead of silently. ## What remains (acceptance #1 + #2 — gated on infra) The bump is idempotent and backfilling — it rewrites `url`/`sha256` against whatever the live tap holds — so the **next green release auto-advances the tap to the latest tag**, no manual catch-up needed. But it can't go green until the underlying write credential is restored: - **Operator action:** restore the `tap-writer` forgejo-runner's git credential (the `infrastructure` deploy that provisions `forgejo-runner-tap-writer.yml`'s credential helper). The same broken credential is also why `publish-binaries` upload fails — both recover together. I did not hand-edit `coilyco-flight-deck/homebrew-tap` to backfill it: that's a different repo, outside this container's autonomy wall (infra/other-repo writes). Once the credential is back, the first release backfills the tap and `brew upgrade ward` works again. docs/release.md now documents this failure mode + recovery.

coilysiren referenced this issue

2026-07-04 09:11:56 +00:00

advisor ref-mode returns docker exit 125 through the dispatch broker (TTY/attach in non-interactive host-forward context) #605

coilysiren referenced this issue

2026-07-04 09:12:00 +00:00

advisor ref-mode returns docker exit 125 through the dispatch broker (TTY/attach in non-interactive host-forward context) #606