The release pipeline has failed on every main push for a week, so nothing has shipped since v2.19.0 #281

Closed
opened 2026-08-14 03:16:41 +00:00 by coilyco-ops · 1 comment
Member

🤖 Filed by Claude Code on Kai's behalf.

Every release run on main is failing, and has been for at least the last ten pushes. The most recent successful release is v2.19.0, 2026-08-07.

What I can show

Ten consecutive release runs, newest first, all status=failure:

71b112b0  feat(schema): let a request name the seat it composes   run 234
d5783b7a  chore: refresh the managed agentic-os hook block        run 233
b486f888  refactor!: retire the driver-plus-reviewer eval stack    run 232
0445e5ad  docs(agents): state the temp-clone expectation           run 231
8fedc3ec / 017c5d28 / a19ae629 / e8444c83 / ca223ad3 / 0dd9fcc7

Run 234 started 03:14:29Z and failed 03:14:54Z25 seconds, which is short for a job that builds and tests, so it looks like a setup failure rather than a build one.

Corroborating: coilyco-flight-deck/agentic-os pins AGENT_COMPOSE_VERSION=2.19.0 at docker/dev-base/full/Dockerfile:69, exactly where releases stopped.

What I could not determine

The cause. /api/v1/.../actions/runs/234 and /actions/runs/234/logs both return resource does not exist, and the tasks endpoint exposes no per-job breakdown, so I could not read a single line of failure output. Everything above is from run metadata alone.

One hypothesis I could not test, offered as a lead rather than a finding: .forgejo/workflows/release.yml runs the job inside container: forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release. A pull failure there would produce a fast setup-time failure like this one, and it would be circular — agent-compose's release runs inside an image that ships agent-compose. Someone with runner log access should check that before chasing the workflow body.

Why it matters beyond the version number

The pipeline is the delivery path for everything downstream. agentic-os:release ships the binary; coilyco-gaming/sirens-echo composes bundles with it at image build. So a change landing on main here reaches no consumer until this is fixed.

Concretely blocked right now: #280's seat-identity override landed on main as 71b112b, and cannot reach sirens-echo — which needs it to stop introducing Olaf in front of the Sirens Discord. Nothing in that chain is broken except this.

A week of green-looking merges that shipped nothing is also the reporting half of this. main looks healthy from the commit log.

🤖 Filed by Claude Code on Kai's behalf.

> 🤖 Filed by Claude Code on Kai's behalf. Every `release` run on `main` is failing, and has been for at least the last ten pushes. The most recent successful release is **v2.19.0, 2026-08-07**. ## What I can show Ten consecutive `release` runs, newest first, all `status=failure`: ``` 71b112b0 feat(schema): let a request name the seat it composes run 234 d5783b7a chore: refresh the managed agentic-os hook block run 233 b486f888 refactor!: retire the driver-plus-reviewer eval stack run 232 0445e5ad docs(agents): state the temp-clone expectation run 231 8fedc3ec / 017c5d28 / a19ae629 / e8444c83 / ca223ad3 / 0dd9fcc7 ``` Run 234 started `03:14:29Z` and failed `03:14:54Z` — **25 seconds**, which is short for a job that builds and tests, so it looks like a setup failure rather than a build one. Corroborating: `coilyco-flight-deck/agentic-os` pins `AGENT_COMPOSE_VERSION=2.19.0` at `docker/dev-base/full/Dockerfile:69`, exactly where releases stopped. ## What I could not determine **The cause.** `/api/v1/.../actions/runs/234` and `/actions/runs/234/logs` both return `resource does not exist`, and the tasks endpoint exposes no per-job breakdown, so I could not read a single line of failure output. Everything above is from run metadata alone. One hypothesis I could not test, offered as a lead rather than a finding: `.forgejo/workflows/release.yml` runs the job inside `container: forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release`. A pull failure there would produce a fast setup-time failure like this one, and it would be circular — agent-compose's release runs inside an image that ships agent-compose. Someone with runner log access should check that before chasing the workflow body. ## Why it matters beyond the version number The pipeline is the delivery path for everything downstream. `agentic-os:release` ships the binary; `coilyco-gaming/sirens-echo` composes bundles with it at image build. So a change landing on `main` here reaches no consumer until this is fixed. Concretely blocked right now: #280's seat-identity override landed on `main` as `71b112b`, and cannot reach `sirens-echo` — which needs it to stop introducing Olaf in front of the Sirens Discord. Nothing in that chain is broken except this. A week of green-looking merges that shipped nothing is also the reporting half of this. `main` looks healthy from the commit log. > 🤖 Filed by Claude Code on Kai's behalf.
Author
Member

🤖 Filed by Claude Code on Kai's behalf.

Pulled the full run history rather than the first page, and it gives a bisect boundary nine minutes wide.

success  1e5c409c  2026-08-07T03:59:40Z  Merge pull request 'test(evals): add an exec...
failure  53be5372  2026-08-07T04:08:48Z  Merge pull request 'feat(person): add shared...

Every run before 1e5c409c is a normal mix of success and failure. Every run from 53be5372 onward is a failure, without exception — that is 33 consecutive failing runs across a week, not an intermittent break.

v2.19.0 published at 03:50:17Z, and 1e5c409c succeeded at 03:59:40Z after it. So the last green run is after the last release, which rules out the release step itself having been mid-break.

That narrows it to whatever changed between those two commits, or to something environmental that happened in that nine-minute window. The commit range is the cheaper thing to check first.

Still no logs — the runs and logs API endpoints both return resource does not exist, so I cannot confirm any of this against actual failure output. The boundary is from run metadata alone.

One correction to my hypothesis above, since the history contradicts part of it: I guessed a pull failure on container: agentic-os:release. That image has presumably been pulled successfully by other repositories' workflows all week, so a plain unavailable-image story does not fit on its own. Treat the commit range as the stronger lead.

🤖 Filed by Claude Code on Kai's behalf.

> 🤖 Filed by Claude Code on Kai's behalf. Pulled the full run history rather than the first page, and it gives a bisect boundary nine minutes wide. ``` success 1e5c409c 2026-08-07T03:59:40Z Merge pull request 'test(evals): add an exec... failure 53be5372 2026-08-07T04:08:48Z Merge pull request 'feat(person): add shared... ``` Every run before `1e5c409c` is a normal mix of success and failure. **Every run from `53be5372` onward is a failure, without exception** — that is 33 consecutive failing runs across a week, not an intermittent break. `v2.19.0` published at `03:50:17Z`, and `1e5c409c` succeeded at `03:59:40Z` after it. So the last green run is *after* the last release, which rules out the release step itself having been mid-break. That narrows it to whatever changed between those two commits, or to something environmental that happened in that nine-minute window. The commit range is the cheaper thing to check first. Still no logs — the runs and logs API endpoints both return `resource does not exist`, so I cannot confirm any of this against actual failure output. The boundary is from run metadata alone. One correction to my hypothesis above, since the history contradicts part of it: I guessed a pull failure on `container: agentic-os:release`. That image has presumably been pulled successfully by other repositories' workflows all week, so a plain unavailable-image story does not fit on its own. Treat the commit range as the stronger lead. > 🤖 Filed by Claude Code on Kai's behalf.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/agent-compose#281
No description provided.