coilyco-flight-deck/lunch-money-k8s

Watch

Image build is not reproducible from a commit sha: uv and the base image are both floating tags #46

New issue

Open

opened 2026-08-29 15:12:02 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-29 15:12:02 +00:00

Owner

docs/deploy.md states the fleet consumes "that exact immutable reference", and scripts/publish-image.sh tags by full source sha and prints verified immutable manifest. The tag is immutable. The image contents are not.

Dockerfile pulls two floating tags:

FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

and publish-image.sh runs docker build --pull, which re-resolves both on every build rather than reusing a local layer. So building the same commit twice can produce two different images under two different sha tags, and nothing in the pipeline would show it. uv:latest is the sharper of the two, since uv ships often and latest moves with it.

Observed on run 75 (job 47078), where step 2 failed:

invalid from flag value ghcr.io/astral-sh/uv:latest:
  Head "https://ghcr.io/v2/astral-sh/uv/manifests/latest": net/http: TLS handshake timeout

That failure is the reachability half rather than the reproducibility half, but both come from the same line.

Two things to decide, and I did not decide either here:

Pin uv to an exact version, and pin the base image by digest rather than 3.12-slim. Low risk, and it makes sha -> image deterministic.
Whether the build should reach ghcr.io at all. The build already needs a Python index for uv sync --frozen, so sourcing uv from there instead would cut the external host set from two to one. Mirroring uv into the Forgejo registry would cut it further, but that is a live-registry change rather than a repo change.

Not fixed in place because there is no Docker daemon on the host I was working from, so I could not build the image to validate any of it, and an unvalidated build change pushed to main is what produced the failure this came out of.

Found while fixing the run-74 format failure (ea739b6).

`docs/deploy.md` states the fleet consumes "that exact immutable reference", and `scripts/publish-image.sh` tags by full source sha and prints `verified immutable manifest`. The tag is immutable. The image contents are not. `Dockerfile` pulls two floating tags: FROM python:3.12-slim COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ and `publish-image.sh` runs `docker build --pull`, which re-resolves both on every build rather than reusing a local layer. So building the same commit twice can produce two different images under two different sha tags, and nothing in the pipeline would show it. `uv:latest` is the sharper of the two, since uv ships often and `latest` moves with it. Observed on run 75 (job 47078), where step 2 failed: invalid from flag value ghcr.io/astral-sh/uv:latest: Head "https://ghcr.io/v2/astral-sh/uv/manifests/latest": net/http: TLS handshake timeout That failure is the reachability half rather than the reproducibility half, but both come from the same line. Two things to decide, and I did not decide either here: 1. Pin `uv` to an exact version, and pin the base image by digest rather than `3.12-slim`. Low risk, and it makes sha -> image deterministic. 2. Whether the build should reach ghcr.io at all. The build already needs a Python index for `uv sync --frozen`, so sourcing uv from there instead would cut the external host set from two to one. Mirroring uv into the Forgejo registry would cut it further, but that is a live-registry change rather than a repo change. Not fixed in place because there is no Docker daemon on the host I was working from, so I could not build the image to validate any of it, and an unvalidated build change pushed to main is what produced the failure this came out of. Found while fixing the run-74 format failure (ea739b6).

coilyco-ops added the

autonomy

headless

priority

labels

2026-08-29 15:12:02 +00:00

coilyco-ops referenced this issue from a commit

2026-08-29 15:12:55 +00:00

ci: give build-publish a retry trigger, because it had none

coilyco-ops commented

2026-08-29 15:13:51 +00:00

Author

Owner

Confirmed transient. Run 76 built and published the identical Dockerfile line with no change to it:

879a15cc48af63a1a0711f58e8338dabfa42de48: digest: sha256:4a343876b946239785616ccd9c99bf4dcd3fdaf815fb0357001d59d32005605a
verified immutable manifest forgejo.coilysiren.me/coilyco-flight-deck/lunch-money-mcp:879a15cc...

So the ghcr.io TLS handshake timeout on run 75 was a blip rather than a standing egress block, and the reachability half of this issue is intermittent rather than broken. Two observations now: failed on run 75, succeeded on run 76, roughly twenty minutes apart, same runner and same unpinned tag.

That lowers the urgency but does not close it. An intermittent external fetch on the publish path still fails a run whose commit is fine, and the reproducibility half is unaffected by any of this: uv:latest still moves under docker build --pull, so two builds of one sha can still differ.

Confirmed transient. Run 76 built and published the identical Dockerfile line with no change to it: 879a15cc48af63a1a0711f58e8338dabfa42de48: digest: sha256:4a343876b946239785616ccd9c99bf4dcd3fdaf815fb0357001d59d32005605a verified immutable manifest forgejo.coilysiren.me/coilyco-flight-deck/lunch-money-mcp:879a15cc... So the ghcr.io TLS handshake timeout on run 75 was a blip rather than a standing egress block, and the reachability half of this issue is intermittent rather than broken. Two observations now: failed on run 75, succeeded on run 76, roughly twenty minutes apart, same runner and same unpinned tag. That lowers the urgency but does not close it. An intermittent external fetch on the publish path still fails a run whose commit is fine, and the reproducibility half is unaffected by any of this: `uv:latest` still moves under `docker build --pull`, so two builds of one sha can still differ.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/lunch-money-k8s#46

No description provided.

Rows
Columns