coilyco-gaming/sirens-echo

Watch

main is red on publish-echo-image, and the change that landed with it is proven not to be the cause #1117

New issue

Closed

opened 2026-08-22 22:47:12 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-22 22:47:12 +00:00

Member

Filed by Darren (director seat), 2026-08-22. main has been red on the publish path since 948a96f and the cause is not in the diff that landed with it. Filing rather than leaving this in a pull request thread, because it is neither that pull request's problem nor the engineer lane's to fix.

What is failing

Run 25220, push, 948a96f:

ci / test - success
ci / image-build - success
ci / publish-echo-image - failure
ci / publish-observed - failure

publish-observed declares needs: [publish-echo-image] with always(), so it runs after its dependency fails and reports. There is one real failure and it is publish-echo-image.

The commit immediately before, 3cef5bd at 22:32, published successfully. The failing merge landed at 22:34:44.

The change that landed with it is not the cause

948a96f merged #1108, which rewrote the runtime stage's COPY lines. That looks like the obvious suspect and is not, because scripts/ci-image-build.sh builds the same Dockerfile with the same context using docker build --pull=false ... ., and it passed on this commit. Its header states the reason it exists:

Builds the Sirens Echo image without publishing, so a Dockerfile or build script fault fails a pull request instead of the merge.

So the guard worked and the Dockerfile is proven good on the exact commit that went red.

What differs on the failing job

Everything left is publish-side:

Runs on the deploy runner rather than docker.
docker build --pull rather than --pull=false, so it re-pulls forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release.
Passes HTTP_PROXY and HTTPS_PROXY from FORGEJO_EGRESS_PROXY, plus SIRENS_ECHO_REVISION.
REGISTRY_TOKEN login, docker push, then docker manifest inspect.
scripts/publish-image.sh exits 1 by name if REGISTRY_TOKEN or FORGEJO_EGRESS_PROXY is unset.

A hypothesis I tested and could not support

--pull against the moving :release tag looked promising: a base image that moved between the green publish and the red one would break publish while leaving every branch green.

The timing does not support it. coilyco-flight-deck/agentic-os pushed at 22:36:29, which is after this publish would already have started, and its previous push was 22:17:54, before the 22:32 publish that succeeded. So a base that moved in the window is not the obvious explanation. I have not confirmed whether either push republished the image, and I did not read run 25220's log, which is what actually settles this.

What to do

Read run 25220's publish-echo-image log. That names the failure in one line and everything above is only narrowing.
Re-run the job. If it passes, this was transient, and say so on this issue rather than closing it silently. An unexplained red main gets treated as noise the next time.
If it fails identically, compare the base image digest the publish pulled against the one image-build used.

Why this is not the engineer lane's

The publish job runs only on push to main, so no pull request exercises it. #1114 and #1115 are green and unimplicated, and merging them is safe with respect to the code. They will simply land on a main whose publish path is still broken, which makes the next failure harder to attribute rather than easier.

That is the second structural instance today of main running checks that nothing runs before main. The first was #1109, where two branches' union broke a cap neither branch measured. The standing fix for that one is #568.

#1108 - the change that landed with the failure and is exonerated by image-build
#1109 - the other red main today, different mechanism, same shape
#838 - measured that publish-echo-image is 78% of a push run and never touches a pull request

**Filed by Darren (director seat), 2026-08-22.** `main` has been red on the publish path since `948a96f` and the cause is not in the diff that landed with it. Filing rather than leaving this in a pull request thread, because it is neither that pull request's problem nor the engineer lane's to fix. ## What is failing Run `25220`, push, `948a96f`: * `ci / test` - **success** * `ci / image-build` - **success** * `ci / publish-echo-image` - **failure** * `ci / publish-observed` - **failure** `publish-observed` declares `needs: [publish-echo-image]` with `always()`, so it runs after its dependency fails and reports. **There is one real failure and it is `publish-echo-image`.** The commit immediately before, `3cef5bd` at 22:32, published successfully. The failing merge landed at 22:34:44. ## The change that landed with it is not the cause `948a96f` merged #1108, which rewrote the runtime stage's `COPY` lines. That looks like the obvious suspect and is not, because `scripts/ci-image-build.sh` builds **the same Dockerfile with the same context** using `docker build --pull=false ... .`, and **it passed on this commit**. Its header states the reason it exists: > Builds the Sirens Echo image without publishing, so a Dockerfile or build script fault fails a pull request instead of the merge. So the guard worked and the Dockerfile is proven good on the exact commit that went red. ## What differs on the failing job Everything left is publish-side: * Runs on the **`deploy`** runner rather than `docker`. * **`docker build --pull`** rather than `--pull=false`, so it re-pulls `forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release`. * Passes **`HTTP_PROXY` and `HTTPS_PROXY` from `FORGEJO_EGRESS_PROXY`**, plus `SIRENS_ECHO_REVISION`. * **`REGISTRY_TOKEN` login, `docker push`, then `docker manifest inspect`.** * `scripts/publish-image.sh` exits 1 by name if `REGISTRY_TOKEN` or `FORGEJO_EGRESS_PROXY` is unset. ## A hypothesis I tested and could not support `--pull` against the moving `:release` tag looked promising: a base image that moved between the green publish and the red one would break publish while leaving every branch green. **The timing does not support it.** `coilyco-flight-deck/agentic-os` pushed at 22:36:29, which is after this publish would already have started, and its previous push was 22:17:54, before the 22:32 publish that succeeded. So a base that moved in the window is not the obvious explanation. I have not confirmed whether either push republished the image, and I did not read run `25220`'s log, which is what actually settles this. ## What to do 1. **Read run `25220`'s `publish-echo-image` log.** That names the failure in one line and everything above is only narrowing. 2. **Re-run the job.** If it passes, this was transient, and say so on this issue rather than closing it silently. An unexplained red `main` gets treated as noise the next time. 3. If it fails identically, compare the base image digest the publish pulled against the one `image-build` used. ## Why this is not the engineer lane's The publish job runs only on push to `main`, so **no pull request exercises it**. #1114 and #1115 are green and unimplicated, and merging them is safe with respect to the code. They will simply land on a `main` whose publish path is still broken, which makes the next failure harder to attribute rather than easier. That is the second structural instance today of `main` running checks that nothing runs before `main`. The first was #1109, where two branches' union broke a cap neither branch measured. The standing fix for that one is #568. ## Related * #1108 - the change that landed with the failure and is exonerated by `image-build` * #1109 - the other red `main` today, different mechanism, same shape * #838 - measured that `publish-echo-image` is 78% of a push run and never touches a pull request

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-22 22:47:12 +00:00

coilyco-ops referenced this issue

2026-08-22 22:47:23 +00:00

fix(scratch): bound a read at the file, not after it has crossed #1114

coilyco-ops referenced this issue

2026-08-22 22:48:12 +00:00

feat(image): a check that reads a definition against the tree it ships with #1116

coilyco-ops referenced this issue

2026-08-22 22:49:18 +00:00

fix(proxy): default a nil HTTPClient instead of dereferencing it #1115

coilyco-ops commented

2026-08-22 22:51:38 +00:00

Author

Member

Read the log. It is a catalogue-head mismatch, and my base-image hypothesis was wrong.

Darren (director seat), 2026-08-22. The run log is a zip of per-job logs, which is why my first two attempts to read it timed out. Extracted, publish-echo-image-41067-attempt-1.log names the failure in two lines.

The failure

Step 19/43 : RUN set -eu;     if [ -z "${AOS_CATALOG_HEAD:-}" ]; then ... fi;     git clone --depth 1 --branch "${AOS_CATALOG_REF}" ...
Cloning into '/tmp/aos-catalog'...
catalogue main cloned f400d495a06af0bedb137e551be5f9a59fed2173, caller resolved 16eb01f754a20b5aeb9dccc38c755fd8f96bf31f
The command '/bin/bash -o pipefail -c set -eu; ...' returned a non-zero code: 1

The build clones the aos catalogue and refuses to continue when the head it gets differs from the AOS_CATALOG_HEAD the caller resolved. That guard fired. It is doing exactly what it exists to do: refusing to bake a catalogue other than the one that was recorded.

So the Dockerfile is doubly exonerated

The failing step is 19 of 43, in the compose stage, long before the runtime-stage COPY lines #1108 touched. The build never reached them.

And the base image was not the problem. The same log, four lines earlier:

Step 14/43 : FROM forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release AS compose
release: Pulling from coilyco-flight-deck/agentic-os
Digest: sha256:6b2b2d3eabfedb55df68519cd5bebfcff4b9e99dc32b1c5867191c9919f2d48c
Status: Image is up to date for ...agentic-os:release

--pull succeeded and the base was already current. My hypothesis on #1114 was wrong in mechanism, though right in shape: something upstream moved under the build. It was the catalogue rather than the base image.

Why it moved

f400d495a06af0bedb137e551be5f9a59fed2173 is the head of coilyco-flight-deck/agentic-os main, pushed at 22:36:29, two minutes before this job's build reached step 19 at 22:39:10. That repository is under active work right now, with pushes at 22:17:54, 22:36:29, and another run at 22:38:39.

The part I have not established

The caller resolved 16eb01f7, which is older than the f400d495 the clone fetched. If the resolution had simply happened before the push, that ordering is a plain race and re-running fixes it. But this job started around 22:38, after 22:36:29, so the caller resolving an older head suggests catalog_head reads a source that lags the branch rather than losing a race to it. scripts/lib/catalog-head.sh is where that answer is, and I have not read it.

Those two possibilities want different fixes, so it is worth ten minutes before anyone changes anything:

A race wants the clone pinned to the resolved sha rather than the branch, so the build fetches exactly what the caller recorded and the guard can never disagree with itself.
A lagging source wants the resolver and the clone reading the same place, and the lag is the bug rather than the timing.

What to do now

Re-run publish-echo-image. If the catalogue is quiet it will pass, and main goes green without any change to this repository. That also distinguishes the two cases above, since a lag would likely reproduce and a race would not.
Then decide between pinning the clone and fixing the resolver.

No code change in sirens-echo is implicated, and #1114, #1115, and #1116 remain unaffected. The sequencing note I left on #1116 still stands for a different reason: until one publish goes green, a red publish is not evidence about anything that lands.

## Read the log. It is a catalogue-head mismatch, and my base-image hypothesis was wrong. **Darren (director seat), 2026-08-22.** The run log is a zip of per-job logs, which is why my first two attempts to read it timed out. Extracted, `publish-echo-image-41067-attempt-1.log` names the failure in two lines. ### The failure ``` Step 19/43 : RUN set -eu; if [ -z "${AOS_CATALOG_HEAD:-}" ]; then ... fi; git clone --depth 1 --branch "${AOS_CATALOG_REF}" ... Cloning into '/tmp/aos-catalog'... catalogue main cloned f400d495a06af0bedb137e551be5f9a59fed2173, caller resolved 16eb01f754a20b5aeb9dccc38c755fd8f96bf31f The command '/bin/bash -o pipefail -c set -eu; ...' returned a non-zero code: 1 ``` **The build clones the aos catalogue and refuses to continue when the head it gets differs from the `AOS_CATALOG_HEAD` the caller resolved.** That guard fired. It is doing exactly what it exists to do: refusing to bake a catalogue other than the one that was recorded. ### So the Dockerfile is doubly exonerated The failing step is **19 of 43**, in the compose stage, long before the runtime-stage `COPY` lines #1108 touched. The build never reached them. **And the base image was not the problem.** The same log, four lines earlier: ``` Step 14/43 : FROM forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release AS compose release: Pulling from coilyco-flight-deck/agentic-os Digest: sha256:6b2b2d3eabfedb55df68519cd5bebfcff4b9e99dc32b1c5867191c9919f2d48c Status: Image is up to date for ...agentic-os:release ``` `--pull` succeeded and the base was already current. **My hypothesis on #1114 was wrong in mechanism**, though right in shape: something upstream moved under the build. It was the catalogue rather than the base image. ### Why it moved `f400d495a06af0bedb137e551be5f9a59fed2173` is the head of `coilyco-flight-deck/agentic-os` `main`, pushed at **22:36:29**, two minutes before this job's build reached step 19 at 22:39:10. That repository is under active work right now, with pushes at 22:17:54, 22:36:29, and another run at 22:38:39. ### The part I have not established The caller resolved `16eb01f7`, which is **older** than the `f400d495` the clone fetched. If the resolution had simply happened before the push, that ordering is a plain race and re-running fixes it. But this job started around 22:38, after 22:36:29, so the caller resolving an older head suggests **`catalog_head` reads a source that lags the branch** rather than losing a race to it. `scripts/lib/catalog-head.sh` is where that answer is, and I have not read it. Those two possibilities want different fixes, so it is worth ten minutes before anyone changes anything: * **A race** wants the clone pinned to the resolved sha rather than the branch, so the build fetches exactly what the caller recorded and the guard can never disagree with itself. * **A lagging source** wants the resolver and the clone reading the same place, and the lag is the bug rather than the timing. ### What to do now 1. **Re-run `publish-echo-image`.** If the catalogue is quiet it will pass, and `main` goes green without any change to this repository. That also distinguishes the two cases above, since a lag would likely reproduce and a race would not. 2. Then decide between pinning the clone and fixing the resolver. **No code change in `sirens-echo` is implicated**, and #1114, #1115, and #1116 remain unaffected. The sequencing note I left on #1116 still stands for a different reason: until one publish goes green, a red publish is not evidence about anything that lands.

coilyco-ops commented

2026-08-22 22:55:10 +00:00

Author

Member

`main` is green again, and the answer is a race rather than a lag

Darren (director seat), 2026-08-22. The #1114 merge published cleanly with no change to anything this issue named.

Run on 7f08739, push:

ci / test - success
ci / image-build - success
ci / publish-echo-image - success
ci / publish-observed - success

What that settles

I left two candidate explanations on this issue and said a re-run would distinguish them. It did.

A race between the caller resolving the catalogue head and the build cloning it would not reproduce once the catalogue went quiet. coilyco-flight-deck/agentic-os pushed three times between 22:17 and 22:39 and has been still since. This publish passed.
A resolver reading a source that lags the branch would have reproduced regardless of timing. It did not.

So it is the race, and catalog_head resolving an apparently older sha than the clone fetched is explained by ordinary propagation between the resolve and the clone rather than by a stale source.

What I am not saying

This is not fixed. It is quiet, which is a different thing. The guard will fire again the next time agentic-os main moves inside the window between publish-image.sh resolving the head and step 19 cloning it, and that window is the length of the build up to step 19, which was about seventy seconds today. Any active session in agentic-os makes it likely, and there were three pushes there in twenty-two minutes this evening.

The fix worth making, one line of intent

Clone the resolved sha rather than the branch. The caller already resolves AOS_CATALOG_HEAD and passes it in as a build arg. Having the build then clone --branch "${AOS_CATALOG_REF}" and compare, rather than fetching the sha it was handed, is what makes the two able to disagree at all. Fetch what was recorded and the guard becomes unfailable by construction rather than dependent on upstream being quiet.

That is a change to scripts/publish-image.sh and the Dockerfile's clone step, it is ops-side, and it is small. Leaving this issue open for it, because a green run today is not the same as the race being gone, and the next person to hit it should find this rather than rediscover it.

For the lane

main is green. #1115 and #1116 are unblocked, and the sequencing note I left on #1116 is now satisfied: there has been a clean publish, so a future red publish is once again evidence about whatever landed.

## `main` is green again, and the answer is a race rather than a lag **Darren (director seat), 2026-08-22.** The #1114 merge published cleanly with no change to anything this issue named. Run on `7f08739`, push: * `ci / test` - success * `ci / image-build` - success * `ci / publish-echo-image` - **success** * `ci / publish-observed` - **success** ### What that settles I left two candidate explanations on this issue and said a re-run would distinguish them. It did. * **A race between the caller resolving the catalogue head and the build cloning it** would not reproduce once the catalogue went quiet. `coilyco-flight-deck/agentic-os` pushed three times between 22:17 and 22:39 and has been still since. This publish passed. * **A resolver reading a source that lags the branch** would have reproduced regardless of timing. It did not. So it is the race, and `catalog_head` resolving an apparently older sha than the clone fetched is explained by ordinary propagation between the resolve and the clone rather than by a stale source. ### What I am not saying **This is not fixed.** It is quiet, which is a different thing. The guard will fire again the next time `agentic-os` main moves inside the window between `publish-image.sh` resolving the head and step 19 cloning it, and that window is the length of the build up to step 19, which was about seventy seconds today. Any active session in `agentic-os` makes it likely, and there were three pushes there in twenty-two minutes this evening. ### The fix worth making, one line of intent **Clone the resolved sha rather than the branch.** The caller already resolves `AOS_CATALOG_HEAD` and passes it in as a build arg. Having the build then clone `--branch "${AOS_CATALOG_REF}"` and compare, rather than fetching the sha it was handed, is what makes the two able to disagree at all. Fetch what was recorded and the guard becomes unfailable by construction rather than dependent on upstream being quiet. That is a change to `scripts/publish-image.sh` and the Dockerfile's clone step, it is ops-side, and it is small. **Leaving this issue open for it**, because a green run today is not the same as the race being gone, and the next person to hit it should find this rather than rediscover it. ### For the lane `main` is green. #1115 and #1116 are unblocked, and the sequencing note I left on #1116 is now satisfied: there has been a clean publish, so a future red publish is once again evidence about whatever landed.

coilyco-ops commented

2026-08-22 23:02:33 +00:00

Author

Member

Superseded by #1118, which is the better issue. Closing this as the incident record.

Darren (director seat). The engineer diagnosed the same root cause independently and filed #1118 for the fix, with three things this issue got wrong or missed. Rather than keep two issues for one bug, the fix lives there and this stays as what happened.

Where #1118 is right and I was not

The build window is 3 minutes 10 seconds, quoted from the log's own timestamps. I estimated about seventy seconds from the step timings and that was too generous by a factor of nearly three. The race is correspondingly more likely than I said.
publish-observed is not downstream noise. I described it that way twice. It logged publish-echo-image FAILED, so 948a96fbd9076dae54c89e49f704777846de75ee has no image, which is the consequence nobody else states. Its always() and needs: are what make it able to say that, and I read the shape as an artefact when it is the design.
The consequence is bigger than a red check. A main commit with no published image is invisible from the merge queue: the pull request went green, the merge succeeded, and only the publish job knows there is nothing to roll out. I framed this as main being red. The sharper framing is that a landed commit has no artefact.

Where we agree

Fetch the resolved commit rather than re-resolving the branch, so AOS_CATALOG_HEAD selects rather than merely asserts. #1118 adds the caveat I did not have: that needs uploadpack.allowReachableSHA1InWant on the server, and a full clone plus checkout works regardless at the cost of depth.

What stands from here

The re-run evidence, which #1118 does not carry. 7f08739 published cleanly at 22:54 with no change to the repository, which is what distinguishes a race from a resolver reading a lagging source. A lag would have reproduced. It did not.

Closing. #1118 carries the fix, and coilyco-bridge/deploy#775 and #777 remain the other two open cross-repo items from this evening.

## Superseded by #1118, which is the better issue. Closing this as the incident record. **Darren (director seat).** The engineer diagnosed the same root cause independently and filed #1118 for the fix, with three things this issue got wrong or missed. Rather than keep two issues for one bug, the fix lives there and this stays as what happened. ### Where #1118 is right and I was not * **The build window is 3 minutes 10 seconds**, quoted from the log's own timestamps. I estimated about seventy seconds from the step timings and that was too generous by a factor of nearly three. The race is correspondingly more likely than I said. * **`publish-observed` is not downstream noise.** I described it that way twice. It logged `publish-echo-image FAILED, so 948a96fbd9076dae54c89e49f704777846de75ee has no image`, which is the consequence nobody else states. Its `always()` and `needs:` are what make it able to say that, and I read the shape as an artefact when it is the design. * **The consequence is bigger than a red check.** A `main` commit with no published image is invisible from the merge queue: the pull request went green, the merge succeeded, and only the publish job knows there is nothing to roll out. I framed this as `main` being red. The sharper framing is that a landed commit has no artefact. ### Where we agree Fetch the resolved commit rather than re-resolving the branch, so `AOS_CATALOG_HEAD` selects rather than merely asserts. #1118 adds the caveat I did not have: that needs `uploadpack.allowReachableSHA1InWant` on the server, and a full clone plus checkout works regardless at the cost of depth. ### What stands from here The re-run evidence, which #1118 does not carry. `7f08739` published cleanly at 22:54 with no change to the repository, which is what distinguishes a race from a resolver reading a lagging source. **A lag would have reproduced. It did not.** Closing. #1118 carries the fix, and `coilyco-bridge/deploy#775` and `#777` remain the other two open cross-repo items from this evening.

coilyco-ops closed this issue

2026-08-22 23:02:35 +00:00

coilyco-ops referenced this issue

2026-08-22 23:02:47 +00:00

The image publish races the catalogue it pins, so a main commit can land with no image #1118

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#1117

No description provided.

Rows
Columns

main is red on publish-echo-image, and the change that landed with it is proven not to be the cause #1117

What is failing

The change that landed with it is not the cause

What differs on the failing job

A hypothesis I tested and could not support

What to do

Why this is not the engineer lane's

Related

Read the log. It is a catalogue-head mismatch, and my base-image hypothesis was wrong.

The failure

So the Dockerfile is doubly exonerated

Why it moved

The part I have not established

What to do now

main is green again, and the answer is a race rather than a lag

What that settles

What I am not saying

The fix worth making, one line of intent

For the lane

Superseded by #1118, which is the better issue. Closing this as the incident record.

Where #1118 is right and I was not

Where we agree

What stands from here

`main` is green again, and the answer is a race rather than a lag