CI parity: run every app's CI inside the pinned dev-base container via ward verbs #328

New issue

Closed

opened 2026-07-05 02:10:40 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-05 02:10:40 +00:00

Owner

Proposal

Run every app's CI inside the pinned dev-base container, through the same ward exec verbs the agents run, so a green CI means "a headless agent can actually land this," not "it passed in some other environment."

Why (the motivating failure)

On coilyco-gaming/eco-app, CI was green while every dispatched warded run died. Cause: the two environments diverge.

CI runs runs-on: docker + astral-sh/setup-uv, which fetches its own Python into a writable location. Green.
The agent runs in the dev-base container, where UV_PYTHON_INSTALL_DIR=/opt/uv/python is root-owned and no 3.13 is present, so uv run cannot install the required Python and the test/smoke gate dies. Red, invisibly.

CI was validating a world no agent lives in. The dev-base image bug (companion image-fix issue) was the trigger, but the deeper problem is that CI and the agent runtime are not the same environment, so environment regressions are invisible until a dispatched run silently fails to land. Pinning CI to the same image the agents use makes those regressions fail loudly in CI, on a PR, before any container is dispatched.

Shape

Each app's CI job runs container: <registry>/dev-base:<pinned-tag> instead of a bare runner, and invokes ward exec test / ward exec lint / ward exec smoke (the agent's own gate) instead of hand-rolled uv run steps.
Pinned, not :latest - so CI is reproducible and adopting a new dev-base is a deliberate tag bump (which itself re-validates every app), not a silent drift.
Authored per-app (each repo's .forgejo/workflows/*.yml is the rollout unit), with the convention and the pinned-tag source of truth owned here in aos. An ansible/template rollout may fan it across repos per the authoring-vs-rollout law.

Blocking dependency

The dev-base image fix (companion issue) lands first, or CI-in-dev-base would inherit the same red uv gate. With the image fixed (or an app carrying the eco-app#56-style Makefile workaround), ward verbs pass inside dev-base and parity holds green.

Rollout

aos: fix the image (companion), publish a pinned tag, document the CI-in-dev-base convention.
Per app: switch the CI workflow to the pinned container + ward verbs. First mover / reference implementation: coilyco-gaming/eco-app (its own adoption issue).

Origin

coilyco-gaming/eco-app read-only director session, 2026-07-05. Incident: eco-app#56 (agents died on smoke while CI stayed green). Kai's call in that session: "run CI in all our apps from inside a pinned aos container."

## Proposal Run every app's CI **inside the pinned dev-base container, through the same `ward exec` verbs the agents run**, so a green CI means "a headless agent can actually land this," not "it passed in some other environment." ## Why (the motivating failure) On coilyco-gaming/eco-app, CI was green while **every** dispatched `warded` run died. Cause: the two environments diverge. * **CI** runs `runs-on: docker` + `astral-sh/setup-uv`, which fetches its own Python into a writable location. Green. * **The agent** runs in the dev-base container, where `UV_PYTHON_INSTALL_DIR=/opt/uv/python` is root-owned and no 3.13 is present, so `uv run` cannot install the required Python and the test/smoke gate dies. Red, invisibly. CI was validating a world no agent lives in. The dev-base image bug (companion image-fix issue) was the trigger, but the deeper problem is that **CI and the agent runtime are not the same environment**, so environment regressions are invisible until a dispatched run silently fails to land. Pinning CI to the same image the agents use makes those regressions fail loudly in CI, on a PR, before any container is dispatched. ## Shape * Each app's CI job runs `container: <registry>/dev-base:<pinned-tag>` instead of a bare runner, and invokes `ward exec test` / `ward exec lint` / `ward exec smoke` (the agent's own gate) instead of hand-rolled `uv run` steps. * **Pinned, not `:latest`** - so CI is reproducible and adopting a new dev-base is a deliberate tag bump (which itself re-validates every app), not a silent drift. * Authored per-app (each repo's `.forgejo/workflows/*.yml` is the rollout unit), with the convention and the pinned-tag source of truth owned here in aos. An ansible/template rollout may fan it across repos per the authoring-vs-rollout law. ## Blocking dependency The dev-base image fix (companion issue) lands first, or CI-in-dev-base would inherit the same red uv gate. With the image fixed (or an app carrying the eco-app#56-style Makefile workaround), ward verbs pass inside dev-base and parity holds green. ## Rollout * aos: fix the image (companion), publish a pinned tag, document the CI-in-dev-base convention. * Per app: switch the CI workflow to the pinned container + ward verbs. First mover / reference implementation: coilyco-gaming/eco-app (its own adoption issue). ## Origin coilyco-gaming/eco-app read-only director session, 2026-07-05. Incident: eco-app#56 (agents died on smoke while CI stayed green). Kai's call in that session: "run CI in all our apps from inside a pinned aos container."

coilysiren referenced this issue from coilyco-gaming/eco-app

2026-07-05 02:11:06 +00:00

Blocker: headless runs die on smoke - uv cannot install Python 3.13 into root-owned /opt/uv/python #56

coilysiren referenced this issue from coilyco-gaming/eco-app

2026-07-05 03:45:54 +00:00

CI: compile the C# mods (build-mods verb) so a mod typo can't ship to the server unseen #60

coilysiren referenced this issue

2026-07-05 03:46:36 +00:00

dev-base: ship the .NET SDK (+ ICU) so C# mods compile in-container (sibling of #327 uv fix) #329

coilyco-ops commented

2026-07-05 05:32:46 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agentic-os-328 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T05:32:38Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agentic-os-328` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T05:32:38Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-05 05:41:26 +00:00

docs(ci-parity): own the CI-in-dev-base convention + pinned-tag source of truth

coilysiren closed this issue

2026-07-05 05:41:26 +00:00

coilysiren referenced this issue from a commit

2026-07-05 05:41:26 +00:00

Merge issue-328: CI-in-dev-base convention + pinned-tag source of truth (closes #328)

coilysiren commented

2026-07-05 05:41:53 +00:00

Author

Owner

WARD-OUTCOME: done - aos now owns the CI-in-dev-base convention + a pinned-tag source of truth (docker/dev-base/ci-pinned-tag.txt @ v0.195.0), with a doc and a copy-paste reference workflow.

Honest read: this one was more docs-and-contract than code, and the real work was placement discipline, not typing. The trap was the config-placement law - it was tempting to have an app's CI reach up into aos for the tag at run time, which would have violated "never fetched downward." Landing it as an aos-authored value that an infra rollout templates down (authoring-vs-rollout) is the version that holds.

What fought back a little: the code-comments hook scans yaml, so a realistic example workflow with comment blocks trips the 2-contiguous-comment rule - had to add the example to the excludes list the way the existing skill-discipline examples are. FEATURES.md was also near its char cap, so the entry got trimmed twice.

Confidence: high on the aos-side deliverable (gate green, pinned to a tag whose published image already carries the uv + .NET fixes). The parity claim itself only gets proven when the first app (eco-app) actually adopts it - that's its own adoption issue and out of this repo's scope. Follow-ups worth filing: the infrastructure ansible/template rollout that fans the pinned tag into each app, and eco-app's reference adoption. Also note ci-pinned-tag.txt is bumped by hand - if that drifts stale it's silent, so a future freshness/lint tie-in might be worth it.

WARD-OUTCOME: done - aos now owns the CI-in-dev-base convention + a pinned-tag source of truth (docker/dev-base/ci-pinned-tag.txt @ v0.195.0), with a doc and a copy-paste reference workflow. Honest read: this one was more docs-and-contract than code, and the real work was placement discipline, not typing. The trap was the config-placement law - it was tempting to have an app's CI reach up into aos for the tag at run time, which would have violated "never fetched downward." Landing it as an aos-authored value that an infra rollout templates down (authoring-vs-rollout) is the version that holds. What fought back a little: the code-comments hook scans yaml, so a realistic example workflow with comment blocks trips the 2-contiguous-comment rule - had to add the example to the excludes list the way the existing skill-discipline examples are. FEATURES.md was also near its char cap, so the entry got trimmed twice. Confidence: high on the aos-side deliverable (gate green, pinned to a tag whose published image already carries the uv + .NET fixes). The parity claim itself only gets proven when the first app (eco-app) actually adopts it - that's its own adoption issue and out of this repo's scope. Follow-ups worth filing: the infrastructure ansible/template rollout that fans the pinned tag into each app, and eco-app's reference adoption. Also note ci-pinned-tag.txt is bumped by hand - if that drifts stale it's silent, so a future freshness/lint tie-in might be worth it.