coilyco-flight-deck/agentic-os

Watch

One transient language-payload failure costs the whole dev-base publish and a manual re-dispatch #987

New issue

Closed

opened 2026-08-10 00:45:09 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-10 00:45:09 +00:00

Member

Observation

dev-base-publish run 3169 (id 16863) on 7547dd34 failed. Same commit, re-dispatched as run 3170, went green with no code change.

Job-level detail from aosguard ops forgejo tasks list:

publish-lang-go - failure - started 22:01:09Z, ended 22:05:02Z (about 4 minutes)
publish-lang-node, publish-lang-dotnet, publish-lang-python, publish-lang-rust - success, same window
publish-full, retag-full, release - skipped, because the matrix did not come back clean
run 3170 at 22:13:45Z rebuilt all five payloads and the full image, ending green at 22:43:30Z

Identical inputs, different outcome, so the failure is transient rather than a defect in the tree.

Why it is worth more than a re-run

publish-language-payloads sets fail-fast: false, but publish-full declares needs: [plan-draft, publish-language-payloads], so one payload failing skips the full image and the entire release. Recovery is a manual workflow_dispatch that rebuilds all five payloads from scratch: about 30 minutes of runner time to recover from one job that failed in four.

actions/publish-dev-base already documents the property that would make this cheap. Its own description says promotion is restricted to the already-pushed draft, "which is what makes the job safe to retry". Nothing acts on that: there is no per-job retry, and no path that lets publish-full consume the four payloads that did succeed.

What blocked a root cause

The failure text is unreachable. Job logs 404 on this Forgejo:

GET /repos/coilyco-flight-deck/agentic-os/actions/runs/16863/jobs  -> 404
GET /repos/coilyco-flight-deck/agentic-os/actions/runs/16863/logs  -> 404

The server runs Forgejo 15, whose swagger exposes neither route. #970 noted this in passing and said it deserves its own issue. coilyco-gaming/galaxy-gen#73 is blocked the same way. Until a run's logs are readable, every CI failure here is diagnosed from status codes and timings alone, which is how this issue ends without a cause.

Candidate causes, unconfirmed

Both are hypotheses from reading the build, not from logs:

apt-get update / apt-get install in every language stage of docker/dev-base/Dockerfile carry no retry, while every curl in the same file uses --retry 5 --retry-all-errors --retry-delay 2. An apt mirror blip is a classic CI transient and would fail in about the right window. The Go stage's go.dev download is already retried, which argues against the toolchain download itself.
A registry push transient against the Forgejo registry.

Suggested direction

Give the payload matrix a bounded automatic retry. The action already asserts the job is safe to retry.
Or let publish-full proceed from the last good payload draft instead of requiring the whole matrix green in one run, so a single transient costs one job rather than the release.
Separately, make Actions job logs reachable, whether by a Forgejo upgrade or a supported fallback. Without it the next occurrence produces exactly this issue again.

Add retry to apt-get in the language stages regardless. It is cheap, matches the curl treatment beside it, and removes one candidate.

Frequency caveat

The Actions API returns page 1 only, about the last 50 runs, which covers 3 dev-base-publish runs: 2 success, 1 failure. This documents one instance and its blast radius. It does not establish a rate.

## Observation `dev-base-publish` run 3169 (id 16863) on `7547dd34` failed. Same commit, re-dispatched as run 3170, went green with no code change. Job-level detail from `aosguard ops forgejo tasks list`: * `publish-lang-go` - **failure** - started 22:01:09Z, ended 22:05:02Z (about 4 minutes) * `publish-lang-node`, `publish-lang-dotnet`, `publish-lang-python`, `publish-lang-rust` - success, same window * `publish-full`, `retag-full`, `release` - **skipped**, because the matrix did not come back clean * run 3170 at 22:13:45Z rebuilt all five payloads and the full image, ending green at 22:43:30Z Identical inputs, different outcome, so the failure is transient rather than a defect in the tree. ## Why it is worth more than a re-run `publish-language-payloads` sets `fail-fast: false`, but `publish-full` declares `needs: [plan-draft, publish-language-payloads]`, so one payload failing skips the full image and the entire release. Recovery is a manual `workflow_dispatch` that rebuilds all five payloads from scratch: about 30 minutes of runner time to recover from one job that failed in four. `actions/publish-dev-base` already documents the property that would make this cheap. Its own description says promotion is restricted to the already-pushed draft, "which is what makes the job safe to retry". Nothing acts on that: there is no per-job retry, and no path that lets `publish-full` consume the four payloads that did succeed. ## What blocked a root cause The failure text is unreachable. Job logs 404 on this Forgejo: ``` GET /repos/coilyco-flight-deck/agentic-os/actions/runs/16863/jobs -> 404 GET /repos/coilyco-flight-deck/agentic-os/actions/runs/16863/logs -> 404 ``` The server runs Forgejo 15, whose swagger exposes neither route. `#970` noted this in passing and said it deserves its own issue. `coilyco-gaming/galaxy-gen#73` is blocked the same way. Until a run's logs are readable, every CI failure here is diagnosed from status codes and timings alone, which is how this issue ends without a cause. ## Candidate causes, unconfirmed Both are hypotheses from reading the build, not from logs: * `apt-get update` / `apt-get install` in every language stage of `docker/dev-base/Dockerfile` carry no retry, while every `curl` in the same file uses `--retry 5 --retry-all-errors --retry-delay 2`. An apt mirror blip is a classic CI transient and would fail in about the right window. The Go stage's `go.dev` download is already retried, which argues against the toolchain download itself. * A registry push transient against the Forgejo registry. ## Suggested direction 1. Give the payload matrix a bounded automatic retry. The action already asserts the job is safe to retry. 2. Or let `publish-full` proceed from the last good payload draft instead of requiring the whole matrix green in one run, so a single transient costs one job rather than the release. 3. Separately, make Actions job logs reachable, whether by a Forgejo upgrade or a supported fallback. Without it the next occurrence produces exactly this issue again. Add retry to `apt-get` in the language stages regardless. It is cheap, matches the `curl` treatment beside it, and removes one candidate. ## Frequency caveat The Actions API returns page 1 only, about the last 50 runs, which covers 3 `dev-base-publish` runs: 2 success, 1 failure. This documents one instance and its blast radius. It does not establish a rate.

coilyco-ops referenced this issue from a commit

2026-08-10 01:05:17 +00:00

fix(dev-base): resolve both python paths from the interpreter, not by hand

coilyco-ops referenced this issue

2026-08-10 01:28:38 +00:00

Stop dev-base PYTHONPATH from overriding pinned Python tools #771

coilyco-ops added the

labels

2026-08-17 02:36:15 +00:00

coilyco-ops referenced this issue

2026-08-22 20:54:50 +00:00

Lane: headless engineer burn-down in agentic-os, 28 issues at P1 and P2, ordered so the toolchain unblocks itself first #1177

coilyco-ops added this to the Headless engineer lane: toolchain first milestone

2026-08-22 20:58:16 +00:00

coilyco-ops commented

2026-08-22 21:50:10 +00:00

Author

Member

Director support before this gets worked. Two of the three things this issue is built on have changed, and I have the root cause.

The logs are reachable now

This issue says job logs 404 and that "every CI failure here is diagnosed from status codes and timings alone". That is no longer true. Both routes answer today:

$ aosguard ops forgejo action-run-job list coilyco-flight-deck agentic-os 16863
plan-draft           success
publish-lang-go      failure
publish-lang-node    success
...

$ aosguard ops forgejo action-run logs coilyco-flight-deck agentic-os 16863 > run.zip
122266 bytes, unpacks to one .log per job

coilyco-gaming/galaxy-gen#73, named here as blocked the same way, has since closed. Direction 3 in the suggestions is already satisfied and should come off the list.

Root cause of run 16863, from the log this issue could not read

publish-lang-go-26623-attempt-1.log, the last 60 lines:

#16 exporting to image
#16 pushing layers 14.5s done
#16 ERROR: failed to push forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:lang-go-draft-7547dd34: unknown

#18 exporting cache to registry
#18 sending cache export 14.5s done
#18 ERROR: error writing layer blob: failed to copy: failed to do request:
    Put "https://forgejo.coilysiren.me/v2/coilyco-flight-deck/agentic-os/blobs/uploads/tmvpv3xjlqn3g13srsm9xd?digest=sha256:2e2886..."

The Go image built fine. Every apt step, the go.dev download, the whole build graph completed. It died at the registry, 14.5 seconds into pushing layers, and the image push and the cache export failed at the same instant.

What that changes

The apt hypothesis is dead, and so is the recommendation attached to it. This issue says "Add retry to apt-get in the language stages regardless. It is cheap ... and removes one candidate." It removes a candidate the log already excludes. Adding it is still defensible on its own merits beside the curl retries, and it should not be sold as addressing this failure, because it does not.
The second candidate is the actual one. "A registry push transient against the Forgejo registry" was listed second and unweighted. It is the cause.
Two independent uploads failing in the same second points server-side, not at the client. Worth checking against coilyco-flight-deck/infrastructure#870, which reports kai-server's root filesystem chronically at the 85% critical line. A blob write failing unknown under storage pressure fits, and I have not verified the registry's backing store or what disk looked like on 2026-08-09, so treat that as the next thing to check rather than as the answer.

Suggested direction, revised

Retry the push, not the build. Suggestion 1 in this issue (bounded automatic retry on the payload matrix) is still right and is now better targeted: the action's own description says promotion is restricted to the already-pushed draft, "which is what makes the job safe to retry", and the failure is in exactly the step that claim covers.
Leave suggestion 2 alone for now. Letting publish-full consume the last good payload draft is a larger change that weakens the commit-scoped guarantee the release path depends on, and it is not needed if a retry costs one job instead of thirty minutes.
Drop suggestion 3. Already fixed.
Hand the registry question to infrastructure rather than solving it here. Whether the Forgejo registry drops blob writes under pressure is not this repo's to answer, and a retry here is the right local mitigation either way.

The log archive is one command. Whoever picks this up should read it before writing any code, because the issue's own diagnosis section is now the least reliable part of it.

**Director support before this gets worked. Two of the three things this issue is built on have changed, and I have the root cause.** ## The logs are reachable now This issue says job logs 404 and that "every CI failure here is diagnosed from status codes and timings alone". That is no longer true. Both routes answer today: ``` $ aosguard ops forgejo action-run-job list coilyco-flight-deck agentic-os 16863 plan-draft success publish-lang-go failure publish-lang-node success ... $ aosguard ops forgejo action-run logs coilyco-flight-deck agentic-os 16863 > run.zip 122266 bytes, unpacks to one .log per job ``` `coilyco-gaming/galaxy-gen#73`, named here as blocked the same way, has since closed. Direction 3 in the suggestions is already satisfied and should come off the list. ## Root cause of run 16863, from the log this issue could not read `publish-lang-go-26623-attempt-1.log`, the last 60 lines: ``` #16 exporting to image #16 pushing layers 14.5s done #16 ERROR: failed to push forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:lang-go-draft-7547dd34: unknown #18 exporting cache to registry #18 sending cache export 14.5s done #18 ERROR: error writing layer blob: failed to copy: failed to do request: Put "https://forgejo.coilysiren.me/v2/coilyco-flight-deck/agentic-os/blobs/uploads/tmvpv3xjlqn3g13srsm9xd?digest=sha256:2e2886..." ``` **The Go image built fine.** Every apt step, the `go.dev` download, the whole build graph completed. It died at the registry, 14.5 seconds into pushing layers, and the image push and the cache export failed at the same instant. ## What that changes * **The apt hypothesis is dead, and so is the recommendation attached to it.** This issue says "Add retry to `apt-get` in the language stages regardless. It is cheap ... and removes one candidate." It removes a candidate the log already excludes. Adding it is still defensible on its own merits beside the `curl` retries, and it should not be sold as addressing this failure, because it does not. * **The second candidate is the actual one.** "A registry push transient against the Forgejo registry" was listed second and unweighted. It is the cause. * **Two independent uploads failing in the same second points server-side**, not at the client. Worth checking against `coilyco-flight-deck/infrastructure#870`, which reports kai-server's root filesystem chronically at the 85% critical line. A blob write failing `unknown` under storage pressure fits, and I have not verified the registry's backing store or what disk looked like on 2026-08-09, so treat that as the next thing to check rather than as the answer. ## Suggested direction, revised 1. **Retry the push, not the build.** Suggestion 1 in this issue (bounded automatic retry on the payload matrix) is still right and is now better targeted: the action's own description says promotion is restricted to the already-pushed draft, "which is what makes the job safe to retry", and the failure is in exactly the step that claim covers. 2. **Leave suggestion 2 alone for now.** Letting `publish-full` consume the last good payload draft is a larger change that weakens the commit-scoped guarantee the release path depends on, and it is not needed if a retry costs one job instead of thirty minutes. 3. **Drop suggestion 3.** Already fixed. 4. **Hand the registry question to infrastructure** rather than solving it here. Whether the Forgejo registry drops blob writes under pressure is not this repo's to answer, and a retry here is the right local mitigation either way. The log archive is one command. Whoever picks this up should read it before writing any code, because the issue's own diagnosis section is now the least reliable part of it.

coilyco-ops referenced this issue from a commit

2026-08-22 21:50:16 +00:00

fix(dev-base): stop one transient payload failure costing the release (#987)

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 21:50:33 +00:00

fix(dev-base): stop one transient payload failure costing the release (#987) #1187

coilyco-ops referenced this issue

2026-08-22 21:52:58 +00:00

fix(dev-base): stop one transient payload failure costing the release (#987) #1187

coilyco-ops referenced this issue

2026-08-22 22:05:06 +00:00

fix(release): assert the promotion happened instead of believing the colour (#1032) #1188