coilyco-flight-deck/agentic-os

Fork 0

Finish the dev-base image split: linear tier DAG, unsplit CI jobs, and a build cache thrown away every run #491

New issue

Closed

opened 2026-07-13 04:16:31 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-13 04:16:31 +00:00

Member

The dev-base tier split is half-finished

The image family was split into seven named tiers, but neither the dependency graph nor the CI jobs nor the build cache followed. The result is seven names with none of the benefits, and it is the root cause behind agentic-os#490 (a dotnet push flake losing an entire release).

1. The tier DAG is a linear chain, not a tree

agentic_os/dev_base.py TIER_SPECS wires each tier's base_tier to the previous one:

core -> lang-node -> lang-go -> lang-dotnet -> ops -> agent -> full

Consequences:

No real modularity. ops contains Node, Go, and dotnet whether it needs them or not. There is no way to get Go without Node, or dotnet without both. The split hands out names, not separation.
Parallelism is impossible. A strict chain forces sequential builds. This is why the jobs were never split, so fixing the job layout without fixing the DAG will not buy much.
A mid-chain failure destroys everything downstream. lang-dotnet is position 4 of 7. When its push failed in run 1127, ops, agent, and full were never built - and those are the tiers ward actually consumes.

Careful, this is not a naive rename. Docker images have a single parent, so "make the lang tiers siblings on core" does not by itself work for the composed tiers: ops/agent/full genuinely need more than one toolchain. Fanning the leaves out means the composed tiers must graft toolchains in with multi-stage COPY --from=... rather than inheriting them. That is the standard shape and it is the real fix, but budget for it - do not fan out the leaves and leave full broken.

If the full restructure is too big for one pass, an incremental step that still pays off is to keep the chain but split the CI jobs anyway (below): each tier's job pulls the previously-pushed tier as its base. That buys failure isolation and per-tier retry even without parallelism. State which path was taken and why.

2. The CI jobs are not split per sub-image

publish-dev-base in release.yml is one job with one step that shells into scripts/dev-base-build.py build --push, which loops all seven tiers in-process under subprocess.run(..., check=True). There is no strategy: matrix. So:

The first failing tier aborts the loop and every remaining tier is lost.
There is no per-tier retry, no per-tier timing, and no per-tier log isolation.
The whole family shares one 120-minute timeout.

Split it into one job per tier (a matrix keyed off PUBLISHED_TIER_NAMES, with needs: expressing whatever DAG survives step 1), so a dotnet flake costs the dotnet tier and nothing else.

3. The build cache is thrown away on every run

This is the one that likely causes the flake in the first place.

# .forgejo/workflows/release.yml, "Set up qemu + buildx builder"
docker buildx rm -f aosbuilder 2>/dev/null || true
docker buildx create --use --name aosbuilder --driver docker-container --driver-opt network=host

A docker-container buildx builder holds its layer cache inside its own container. Destroying it every run discards the entire local cache every run. The Forgejo runners are long-lived pods with persistent dind sidecars (forgejo-runner-0 has been up over two days), so that cache would have survived between runs. It is being wiped to dodge a name collision left by crashed runs - the inline comment says exactly that. A collision workaround is silently paying for a full cold rebuild.

That leaves type=registry as the only cache, and scripts/dev-base-build.py sets:

"--cache-to", f"type=registry,ref={buildcache_ref},mode=max,ignore-error=true"

ignore-error=true means cache-write failures are silent. If the registry is unhealthy (and it is - see agentic-os#490), the cache quietly never populates, so the next run is cold again, so it pushes even more multi-GB blobs at the struggling registry. That is a self-reinforcing loop, and combined with linux/amd64,linux/arm64 under qemu emulation it explains both the 120-minute timeout and why lang-dotnet (the largest tier, emulated for arm64) is the one that dies.

Fix the cache:

Reuse the builder instead of nuking it. Probe it (docker buildx inspect aosbuilder) and only recreate on failure, so the warm layer cache in the persistent dind survives between runs.
Add a real disk cache on the runner (type=local with a persisted dir, or a persistent builder) rather than depending solely on a network round-trip to a registry that is currently failing.
Stop swallowing cache-write errors silently. At minimum log loudly when cache-to fails, so a permanently-cold cache is visible instead of inferred.

Relationship to agentic-os#490

agentic-os#490 asks to decouple release from publish-dev-base and add push retries. That is still right and still worth doing as defense in depth - keep it. This issue is the root cause underneath it. #490 stops one flake from costing a release. This one stops the flake from being routine.

Done condition

The lang tiers build independently of one another (or, if the chain is kept, each tier is its own CI job), a dotnet failure no longer prevents ops/agent/full from publishing, the buildx layer cache survives between runs on the same runner, and a warm run does not cold-rebuild the family. Record the measured before/after wall-clock of publish-dev-base in the PR so the caching win is evidenced rather than asserted.

## The dev-base tier split is half-finished The image family was split into seven named tiers, but neither the **dependency graph** nor the **CI jobs** nor the **build cache** followed. The result is seven names with none of the benefits, and it is the root cause behind agentic-os#490 (a dotnet push flake losing an entire release). ## 1. The tier DAG is a linear chain, not a tree `agentic_os/dev_base.py` `TIER_SPECS` wires each tier's `base_tier` to the **previous** one: ```text core -> lang-node -> lang-go -> lang-dotnet -> ops -> agent -> full ``` Consequences: * **No real modularity.** `ops` contains Node, Go, and dotnet whether it needs them or not. There is no way to get Go without Node, or dotnet without both. The split hands out names, not separation. * **Parallelism is impossible.** A strict chain forces sequential builds. This is *why* the jobs were never split, so fixing the job layout without fixing the DAG will not buy much. * **A mid-chain failure destroys everything downstream.** `lang-dotnet` is position 4 of 7. When its push failed in run 1127, `ops`, `agent`, and `full` were never built - and those are the tiers ward actually consumes. **Careful, this is not a naive rename.** Docker images have a single parent, so "make the lang tiers siblings on core" does not by itself work for the composed tiers: `ops`/`agent`/`full` genuinely need more than one toolchain. Fanning the leaves out means the composed tiers must graft toolchains in with multi-stage `COPY --from=...` rather than inheriting them. That is the standard shape and it is the real fix, but budget for it - do not fan out the leaves and leave `full` broken. If the full restructure is too big for one pass, **an incremental step that still pays off** is to keep the chain but split the CI jobs anyway (below): each tier's job pulls the previously-pushed tier as its base. That buys failure isolation and per-tier retry even without parallelism. State which path was taken and why. ## 2. The CI jobs are not split per sub-image `publish-dev-base` in [`release.yml`](.forgejo/workflows/release.yml) is **one job** with **one step** that shells into `scripts/dev-base-build.py build --push`, which loops all seven tiers in-process under `subprocess.run(..., check=True)`. There is no `strategy: matrix`. So: * The first failing tier aborts the loop and every remaining tier is lost. * There is no per-tier retry, no per-tier timing, and no per-tier log isolation. * The whole family shares one 120-minute timeout. Split it into one job per tier (a matrix keyed off `PUBLISHED_TIER_NAMES`, with `needs:` expressing whatever DAG survives step 1), so a dotnet flake costs the dotnet tier and nothing else. ## 3. The build cache is thrown away on every run This is the one that likely causes the flake in the first place. ```bash # .forgejo/workflows/release.yml, "Set up qemu + buildx builder" docker buildx rm -f aosbuilder 2>/dev/null || true docker buildx create --use --name aosbuilder --driver docker-container --driver-opt network=host ``` A `docker-container` buildx builder holds its layer cache **inside its own container**. Destroying it every run **discards the entire local cache every run**. The Forgejo runners are long-lived pods with persistent `dind` sidecars (`forgejo-runner-0` has been up over two days), so that cache **would** have survived between runs. It is being wiped to dodge a name collision left by crashed runs - the inline comment says exactly that. A collision workaround is silently paying for a full cold rebuild. That leaves `type=registry` as the only cache, and `scripts/dev-base-build.py` sets: ```python "--cache-to", f"type=registry,ref={buildcache_ref},mode=max,ignore-error=true" ``` `ignore-error=true` means **cache-write failures are silent**. If the registry is unhealthy (and it is - see agentic-os#490), the cache quietly never populates, so the next run is cold again, so it pushes even more multi-GB blobs at the struggling registry. That is a self-reinforcing loop, and combined with `linux/amd64,linux/arm64` under **qemu emulation** it explains both the 120-minute timeout and why `lang-dotnet` (the largest tier, emulated for arm64) is the one that dies. Fix the cache: * **Reuse the builder instead of nuking it.** Probe it (`docker buildx inspect aosbuilder`) and only recreate on failure, so the warm layer cache in the persistent dind survives between runs. * **Add a real disk cache** on the runner (`type=local` with a persisted dir, or a persistent builder) rather than depending solely on a network round-trip to a registry that is currently failing. * **Stop swallowing cache-write errors silently.** At minimum log loudly when `cache-to` fails, so a permanently-cold cache is visible instead of inferred. ## Relationship to agentic-os#490 agentic-os#490 asks to decouple `release` from `publish-dev-base` and add push retries. That is still right and still worth doing as defense in depth - keep it. **This issue is the root cause underneath it.** #490 stops one flake from costing a release. This one stops the flake from being routine. ## Done condition The lang tiers build independently of one another (or, if the chain is kept, each tier is its own CI job), a dotnet failure no longer prevents `ops`/`agent`/`full` from publishing, the buildx layer cache survives between runs on the same runner, and a warm run does not cold-rebuild the family. Record the measured before/after wall-clock of `publish-dev-base` in the PR so the caching win is evidenced rather than asserted.

coilyco-ops commented

2026-07-13 04:16:50 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-claude-agentic-os-491 on host KAI-DESKTOP-TOWER.

Reserved by ward agent --harness claude (reserved 2026-07-13T04:16:47Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#491 · branch issue-491 · harness claude · workflow pull-request-and-merge
Run: engineer-claude-agentic-os-491 · ward v0.633.0 · dispatched 2026-07-13T04:16:47Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.633.0).

— Claude (she/her), via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-claude-agentic-os-491` on host `KAI-DESKTOP-TOWER`. Reserved by `ward agent --harness claude` (reserved 2026-07-13T04:16:47Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#491` · branch `issue-491` · harness `claude` · workflow `pull-request-and-merge` - **Run:** `engineer-claude-agentic-os-491` · ward `v0.633.0` · dispatched `2026-07-13T04:16:47Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.633.0). </details> </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-13 04:39:34 +00:00

feat(dev-base): finish the tier split - fan-out DAG, per-tier CI jobs, persistent build cache

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-13 04:40:16 +00:00

Finish the dev-base image split: fan-out tier DAG, per-tier CI jobs, persistent build cache #494

coilyco-ops commented

2026-07-13 04:42:39 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-request-and-merge; review summary: skipped - in-container review gate intentionally skipped per the temporary ward default pending brokered QA.

PR: #494 (branch issue-491, ci / gate green, merge-ready; the director records the final done outcome after merge).

What landed. All three axes of aos#491: the tier DAG fans out (lang tiers + ops as siblings on core, agent = ops + lang-node graft, full = agent + lang-go + lang-dotnet grafts via COPY --from of self-contained prefixes), publish-dev-base splits into seven per-tier jobs on a new actions/publish-dev-base-tier composite with needs: carrying the DAG, and the aosbuilder buildx builder is reused instead of nuked with a loud ::warning:: probe on silent buildcache-write failures. New docs/dev-base-build-cache.md; tiering/release docs updated; 466 tests pass with new coverage for grafts, topological order, single-tier --tier builds, and the cache probe.

Retrospective. The full fan-out felt right rather than the incremental keep-the-chain fallback - the graft shape dropped out cleanly once lang-node moved to a self-contained /usr/local/node prefix, and the plan/tests pin it. Confidence is high on the DAG and CI mechanics (the emitted plan was verified tier by tier), medium on first-run frictions in CI: qemu binfmt + builder-create races between parallel sibling jobs are handled (|| true + inspect-recreate), but the first release run is the real test. Surprises: recent publish-dev-base history is bleaker than the issue implies (47 attempts / 5 successes since 2026-07-10), and warm-registry successes were already 3-6 min, confirming the cache write silently rotting is the swing factor. One consumer-visible relocation to watch: node/claude/npm bins move from /usr/local/bin to /usr/local/node/bin (on the image's ENV PATH; no repo or substrate script hardcodes the old paths, and login shells inherit the ENV PATH, but anything outside this repo pinning /usr/local/bin/claude would notice). Follow-ups: record the measured after wall-clock per-tier on the first two post-merge release runs (the before numbers are in the PR), and aos#490's release/publish decoupling + push retries stays open as defense in depth.

— Claude (she/her), via ward agent

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped - in-container review gate intentionally skipped per the temporary ward default pending brokered QA. PR: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/pulls/494 (branch `issue-491`, `ci / gate` green, merge-ready; the director records the final done outcome after merge). **What landed.** All three axes of aos#491: the tier DAG fans out (lang tiers + ops as siblings on core, agent = ops + lang-node graft, full = agent + lang-go + lang-dotnet grafts via `COPY --from` of self-contained prefixes), `publish-dev-base` splits into seven per-tier jobs on a new `actions/publish-dev-base-tier` composite with `needs:` carrying the DAG, and the `aosbuilder` buildx builder is reused instead of nuked with a loud `::warning::` probe on silent buildcache-write failures. New `docs/dev-base-build-cache.md`; tiering/release docs updated; 466 tests pass with new coverage for grafts, topological order, single-tier `--tier` builds, and the cache probe. **Retrospective.** The full fan-out felt right rather than the incremental keep-the-chain fallback - the graft shape dropped out cleanly once lang-node moved to a self-contained `/usr/local/node` prefix, and the plan/tests pin it. Confidence is high on the DAG and CI mechanics (the emitted plan was verified tier by tier), medium on first-run frictions in CI: qemu binfmt + builder-create races between parallel sibling jobs are handled (`|| true` + inspect-recreate), but the first release run is the real test. Surprises: recent `publish-dev-base` history is bleaker than the issue implies (47 attempts / 5 successes since 2026-07-10), and warm-registry successes were already 3-6 min, confirming the cache write silently rotting is the swing factor. One consumer-visible relocation to watch: node/claude/npm bins move from `/usr/local/bin` to `/usr/local/node/bin` (on the image's ENV PATH; no repo or substrate script hardcodes the old paths, and login shells inherit the ENV PATH, but anything outside this repo pinning `/usr/local/bin/claude` would notice). Follow-ups: record the measured after wall-clock per-tier on the first two post-merge release runs (the before numbers are in the PR), and aos#490's release/publish decoupling + push retries stays open as defense in depth. — Claude (she/her), via `ward agent` </details>