Preserve common dev-base cache across late multi-arch failures #786

New issue

Open

opened 2026-07-29 09:46:03 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-29 09:46:03 +00:00

Member

Problem

A common tool-pin change invalidates the monolithic install-common layer for every language tier and both architectures. The language matrix then cold-builds the same common surface four-wide. If verification fails near the end, BuildKit does not export the updated registry cache. A retry may retain completed layers only on the same persistent builder, while cancelled architecture work and jobs scheduled elsewhere rebuild cold.

Evidence

Runs 2392 and 2401 imported the existing lang-*-buildcache manifests and showed many cache hits, so caching is configured and reachable. Run 2392 completed much of amd64 but cancelled arm64 after the final roster verifier failed. Run 2401 reused the retained amd64 layers while downloading and unpacking the arm64 common layer again.

Done-condition

The publish graph durably seeds or exports the shared common multi-architecture cache before tier-specific verification, so a late verifier failure and retry do not repeat the common Ubuntu and tool installation across language tiers. The workflow summary reports the shared-cache provenance and write result.

Non-goals

No language runtime version changes.
No weakening of per-tier or full-image verification.
No dependence on one runner node's local BuildKit state.

## Problem A common tool-pin change invalidates the monolithic install-common layer for every language tier and both architectures. The language matrix then cold-builds the same common surface four-wide. If verification fails near the end, BuildKit does not export the updated registry cache. A retry may retain completed layers only on the same persistent builder, while cancelled architecture work and jobs scheduled elsewhere rebuild cold. ## Evidence Runs 2392 and 2401 imported the existing lang-*-buildcache manifests and showed many cache hits, so caching is configured and reachable. Run 2392 completed much of amd64 but cancelled arm64 after the final roster verifier failed. Run 2401 reused the retained amd64 layers while downloading and unpacking the arm64 common layer again. ## Done-condition The publish graph durably seeds or exports the shared common multi-architecture cache before tier-specific verification, so a late verifier failure and retry do not repeat the common Ubuntu and tool installation across language tiers. The workflow summary reports the shared-cache provenance and write result. ## Non-goals * No language runtime version changes. * No weakening of per-tier or full-image verification. * No dependence on one runner node's local BuildKit state.

coilyco-ops commented

2026-07-29 10:27:33 +00:00

Author

Member

Additional evidence from dev-base-publish run 2401, Rust job 20365: the first multi-arch image build and verification succeeded, then registry export reported . Buildx surfaced both and , despite cache-to carrying . The helper retried the whole build, and the retry could not import because the failed export never created it. The fix should separate durable common/cache seeding from the release image exporter, or otherwise ensure a cache-export fault cannot replay an already-verified tier.

coilyco-ops commented

2026-07-29 10:27:58 +00:00

Author

Member

Correction to the preceding shell-mangled comment. Additional evidence from dev-base-publish run 2401, Rust job 20365: the first multi-arch image build and verification succeeded, then registry export reported error writing layer blob: unknown. Buildx surfaced both exporting cache to registry and failed to push the lang-rust draft with unknown, despite cache-to carrying ignore-error=true. The helper retried the whole build, and the retry could not import lang-rust-buildcache because the failed export never created it. The fix should separate durable common/cache seeding from the release image exporter, or otherwise ensure a cache-export fault cannot replay an already-verified tier.

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-31 09:55:01 +00:00

coilyco-ops referenced this issue

2026-08-04 07:38:42 +00:00

Make dev-base language images cached build-only payloads #863

coilyco-ops added

priority

role/ops