Finish the dev-base image split: fan-out tier DAG, per-tier CI jobs, persistent build cache #494

Merged

coilysiren merged 1 commit from issue-491 into main

2026-07-13 05:18:31 +00:00

coilyco-ops commented

2026-07-13 04:40:16 +00:00

Member

closes #491

ward.workflow: pull-request-and-merge

What this does

Finishes the dev-base tier split along all three axes from the issue.

1. Tier DAG - fan-out/fan-in, not a chain. lang-node, lang-go, lang-dotnet, and ops are now siblings building directly on core, each carrying only its own toolchain. The composed tiers graft what they need with multi-stage COPY --from: agent (on ops) grafts the Node toolchain, full (on agent) grafts Go and dotnet, so full keeps everything it had. Node moves to a self-contained /usr/local/node prefix (on core's PATH) to make its graft one directory copy; Go and dotnet already lived in graftable prefixes. The wiring is data (TierSpec.base_tier / graft_tiers in agentic_os/dev_base.py), emitted as BASE_IMAGE / <TIER>_IMAGE build-args.

The full restructure path was taken, not the incremental keep-the-chain path: the graft shape is the standard multi-stage answer and the plan/tests hold it together, so stopping at split-jobs-on-a-chain would have left the ops/agent tiers still hostage to a dotnet flake.

2. CI - one job per tier. publish-dev-base becomes seven jobs riding the new actions/publish-dev-base-tier composite, with needs: carrying the real DAG (core -> {lang-node,lang-go,lang-dotnet,ops} -> agent -> full). Each tier gets its own logs, timeout, retry (rerun-failed-jobs reruns one tier), and its own verify (tag + alias imagetools inspect, folded into the build script). A lang-dotnet flake now costs lang-dotnet and full only - ops and agent publish anyway. The tag-cutting release job needs publish-full, so the public tag still lands only after the whole family (decoupling that further stays with #490).

3. Cache - persistent and loud. The aosbuilder buildx builder is reused: created only when absent (create race between sibling jobs absorbed), recreated only when it genuinely fails to bootstrap. Its layer cache lives in the long-lived runner dind, so warm runs stop cold-rebuilding under qemu - the old rm -f bootstrap threw that cache away every run. The type=registry cache keeps ignore-error=true (a registry hiccup must not fail a good push), but the script now probes the buildcache manifest after each pushed tier and emits ::warning:: when the write did not land, so a permanently-cold cache is observed instead of inferred. Model documented in docs/dev-base-build-cache.md.

Wall-clock evidence

Before (Forgejo actions API, publish-dev-base tasks since 2026-07-10): 47 attempts, 5 successes. Warm-registry-cache successes ran 2.9-6.1 min; cold runs (cache wiped or registry cache-write silently failing) ran into the shared 120-minute family timeout, and run 1127's lang-dotnet push failure at position 4/7 dropped ops/agent/full entirely (the aos#490 lost release).

After: cannot be measured until this lands and the promote -> release pipeline publishes with it - the numbers to record on the first post-merge release run are the per-tier job durations (expect the sibling rank to overlap, runner capacity permitting) and the second, warm run's family wall-clock (expect minutes, not a cold rebuild, since the builder now survives). Recording those on the issue after the first two release runs is the follow-up; the mechanism (builder reuse + cache probe) is in the diff and the probe makes any remaining coldness visible in the job logs rather than needing inference.

Verification

uv run pytest - 466 passed (new tests: graft plan entries, topological TIER_SPECS, Dockerfile graft stages, single-tier --tier build args, cache-probe warning).
pre-commit run --all-files - clean.
scripts/dev-base-build.py ... plan derives exactly the intended DAG (core root; four siblings on core; agent = ops + LANG_NODE_IMAGE; full = agent + LANG_GO_IMAGE + LANG_DOTNET_IMAGE).

closes #491 ward.workflow: pull-request-and-merge ## What this does Finishes the dev-base tier split along all three axes from the issue. **1. Tier DAG - fan-out/fan-in, not a chain.** `lang-node`, `lang-go`, `lang-dotnet`, and `ops` are now siblings building directly on `core`, each carrying only its own toolchain. The composed tiers graft what they need with multi-stage `COPY --from`: `agent` (on `ops`) grafts the Node toolchain, `full` (on `agent`) grafts Go and dotnet, so `full` keeps everything it had. Node moves to a self-contained `/usr/local/node` prefix (on core's `PATH`) to make its graft one directory copy; Go and dotnet already lived in graftable prefixes. The wiring is data (`TierSpec.base_tier` / `graft_tiers` in `agentic_os/dev_base.py`), emitted as `BASE_IMAGE` / `<TIER>_IMAGE` build-args. The full restructure path was taken, not the incremental keep-the-chain path: the graft shape is the standard multi-stage answer and the plan/tests hold it together, so stopping at split-jobs-on-a-chain would have left the ops/agent tiers still hostage to a dotnet flake. **2. CI - one job per tier.** `publish-dev-base` becomes seven jobs riding the new `actions/publish-dev-base-tier` composite, with `needs:` carrying the real DAG (`core` -> {`lang-node`,`lang-go`,`lang-dotnet`,`ops`} -> `agent` -> `full`). Each tier gets its own logs, timeout, retry (`rerun-failed-jobs` reruns one tier), and its own verify (tag + alias `imagetools inspect`, folded into the build script). A `lang-dotnet` flake now costs `lang-dotnet` and `full` only - `ops` and `agent` publish anyway. The tag-cutting `release` job needs `publish-full`, so the public tag still lands only after the whole family (decoupling that further stays with #490). **3. Cache - persistent and loud.** The `aosbuilder` buildx builder is reused: created only when absent (create race between sibling jobs absorbed), recreated only when it genuinely fails to bootstrap. Its layer cache lives in the long-lived runner dind, so warm runs stop cold-rebuilding under qemu - the old `rm -f` bootstrap threw that cache away every run. The `type=registry` cache keeps `ignore-error=true` (a registry hiccup must not fail a good push), but the script now probes the buildcache manifest after each pushed tier and emits `::warning::` when the write did not land, so a permanently-cold cache is observed instead of inferred. Model documented in `docs/dev-base-build-cache.md`. ## Wall-clock evidence Before (Forgejo actions API, `publish-dev-base` tasks since 2026-07-10): 47 attempts, **5 successes**. Warm-registry-cache successes ran 2.9-6.1 min; cold runs (cache wiped or registry cache-write silently failing) ran into the shared 120-minute family timeout, and run 1127's lang-dotnet push failure at position 4/7 dropped `ops`/`agent`/`full` entirely (the aos#490 lost release). After: cannot be measured until this lands and the promote -> release pipeline publishes with it - the numbers to record on the first post-merge release run are the per-tier job durations (expect the sibling rank to overlap, runner capacity permitting) and the second, warm run's family wall-clock (expect minutes, not a cold rebuild, since the builder now survives). Recording those on the issue after the first two release runs is the follow-up; the mechanism (builder reuse + cache probe) is in the diff and the probe makes any remaining coldness visible in the job logs rather than needing inference. ## Verification - `uv run pytest` - 466 passed (new tests: graft plan entries, topological TIER_SPECS, Dockerfile graft stages, single-tier `--tier` build args, cache-probe warning). - `pre-commit run --all-files` - clean. - `scripts/dev-base-build.py ... plan` derives exactly the intended DAG (core root; four siblings on core; agent = ops + LANG_NODE_IMAGE; full = agent + LANG_GO_IMAGE + LANG_DOTNET_IMAGE).

coilyco-ops added 1 commit

2026-07-13 04:40:16 +00:00

feat(dev-base): finish the tier split - fan-out DAG, per-tier CI jobs, persistent build cache

ci / gate (pull_request) Successful in 55s

Details

40f39e43ab

The seven-tier image family had names but none of the split's benefits
(aos#491): the DAG was a linear chain, the publish was one job looping
all tiers, and the buildx builder (and its layer cache) was deleted at
the top of every run.

- Tier DAG: lang-node / lang-go / lang-dotnet / ops are now siblings on
  core, each carrying only its own toolchain. agent composes ops plus a
  lang-node graft (Claude Code rides Node), and full fans in on agent
  grafting lang-go + lang-dotnet. Grafts are multi-stage COPY --from of
  self-contained prefixes (node moves to /usr/local/node, on core PATH),
  wired as TierSpec.graft_tiers and emitted as <TIER>_IMAGE build-args.
- CI: publish-dev-base splits into one job per tier via the new
  actions/publish-dev-base-tier composite, with needs: carrying the DAG.
  A lang-dotnet flake now costs lang-dotnet (and full, which contains
  dotnet) - ops and agent publish anyway, and rerun-failed-jobs retries
  a single tier. Per-tier verify folds into the build script (tag +
  alias imagetools inspect).
- Cache: the aosbuilder buildx builder is reused, not nuked - created
  only when absent, recreated only when it fails to bootstrap - so the
  layer cache in the persistent dind survives between runs. The
  type=registry cache keeps ignore-error=true but the script now probes
  the buildcache manifest after each push and warns loudly when the
  write did not land. New docs/dev-base-build-cache.md documents the
  model.

closes #491

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

coilyco-ops referenced this pull request

2026-07-13 04:42:39 +00:00

Finish the dev-base image split: linear tier DAG, unsplit CI jobs, and a build cache thrown away every run #491