coilyco-gaming/eco-app
2
0
Fork 0
Code Issues 16 Pull requests Projects Releases Packages Wiki Activity Actions

Blocker: headless runs die on smoke - uv cannot install Python 3.13 into root-owned /opt/uv/python #56

New issue
Closed
opened 2026-07-05 02:06:59 +00:00 by coilysiren · 5 comments
coilysiren commented 2026-07-05 02:06:59 +00:00
Owner
Copy link

Problem (blocks the entire trade wave and every headless run)

Every warded engineer run on this repo dies on the ward exec smoke / ward exec test gate before it can land, so nothing reaches main. The three dispatched foundation runs (#6, #53, #55) left zero commits, zero branches, and zero salvage - they never got past validation.

Root cause (environment, not code)

error: failed to create directory `/opt/uv/python`: Permission denied (os error 13)
make: *** [Makefile:32: smoke] Error 2

Reproduced live in the dev-base container:

  • pyproject.toml sets requires-python = ">=3.13", and the product Dockerfile is FROM python:3.13, so 3.13 is the deliberate target.
  • The dev-base agent image ships only system Python 3.12.3 (/usr/bin/python3.12), no 3.13.
  • So uv run must download a managed CPython 3.13, into UV_PYTHON_INSTALL_DIR, which the image env-sets to /opt/uv/python.
  • /opt/uv is root-owned (drwxr-xr-x root root) and the agent runs non-root, so the download dir cannot be created. uv run aborts, the MCP server never starts, smoke fails, the run cannot go green, and it dies (or the reaper salvages an empty tree).

CI is unaffected and stays green: .forgejo/workflows/build-publish.yml runs uv run directly via astral-sh/setup-uv, which bypasses this Makefile and this image entirely.

Fix (validated end to end)

Redirect uv's managed-Python download to a HOME-writable directory in the Makefile, unconditionally. It must be := / =, not ?=: the container env-sets UV_PYTHON_INSTALL_DIR=/opt/uv/python, and make treats an environment-origin variable as already-set, so ?= would not override it.

# dev-base ships Python 3.12 with a root-owned /opt/uv/python; this repo targets
# 3.13 (pyproject requires-python, product Dockerfile FROM python:3.13), so uv must
# download a managed 3.13. Point that download at a HOME-writable dir so `ward exec`
# (make -> uv) works in the non-root agent container. Unconditional on purpose: the
# image env-sets UV_PYTHON_INSTALL_DIR=/opt/uv/python, so `?=` would not override it.
export UV_PYTHON_INSTALL_DIR := $(HOME)/.cache/uv-python

Verified in the read-only surface clone: with the override in place, ward exec smoke downloads CPython 3.13.14, starts the server, and returns the full 12-tool tools/list round-trip with zero failure markers and exit 0 (up from immediate permission-denied death).

Acceptance

  • ward exec smoke, ward exec test, and ward exec lint pass in a fresh dev-base agent container.
  • The override is unconditional (overrides the image's root-owned default), documented with a one-line why.
  • No change to requires-python (3.13 stays the target) and no change to CI.

Priority and follow-up

P1 - hard blocker. Until this lands on main, re-dispatching #6 / #49-#55 is futile - they all inherit the same red smoke gate. A headless run that applies this Makefile edit first, then runs the gate, passes (the edit is active during validation), so this fix is self-healing.

The deeper cause is a dev-base image bug: a root-owned UV_PYTHON_INSTALL_DIR that non-root agents cannot write, on an image that ships a Python older than its consumers require. That warrants a systemic fix in the image (make the dir agent-writable, or pre-install common Pythons) so no other repo hits this wall. Tracked separately.

## Problem (blocks the entire trade wave and every headless run) Every `warded` engineer run on this repo dies on the `ward exec smoke` / `ward exec test` gate before it can land, so nothing reaches `main`. The three dispatched foundation runs ([#6](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/6), [#53](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/53), [#55](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/55)) left zero commits, zero branches, and zero salvage - they never got past validation. ## Root cause (environment, not code) ``` error: failed to create directory `/opt/uv/python`: Permission denied (os error 13) make: *** [Makefile:32: smoke] Error 2 ``` Reproduced live in the dev-base container: * `pyproject.toml` sets `requires-python = ">=3.13"`, and the product `Dockerfile` is `FROM python:3.13`, so **3.13 is the deliberate target**. * The dev-base agent image ships only **system Python 3.12.3** (`/usr/bin/python3.12`), no 3.13. * So `uv run` must download a managed CPython 3.13, into `UV_PYTHON_INSTALL_DIR`, which the image env-sets to **`/opt/uv/python`**. * `/opt/uv` is **root-owned** (`drwxr-xr-x root root`) and the agent runs non-root, so the download dir cannot be created. `uv run` aborts, the MCP server never starts, smoke fails, the run cannot go green, and it dies (or the reaper salvages an empty tree). CI is unaffected and stays green: `.forgejo/workflows/build-publish.yml` runs `uv run` directly via `astral-sh/setup-uv`, which bypasses this Makefile and this image entirely. ## Fix (validated end to end) Redirect uv's managed-Python download to a HOME-writable directory in the `Makefile`, **unconditionally**. It must be `:=` / `=`, not `?=`: the container env-sets `UV_PYTHON_INSTALL_DIR=/opt/uv/python`, and make treats an environment-origin variable as already-set, so `?=` would not override it. ```make # dev-base ships Python 3.12 with a root-owned /opt/uv/python; this repo targets # 3.13 (pyproject requires-python, product Dockerfile FROM python:3.13), so uv must # download a managed 3.13. Point that download at a HOME-writable dir so `ward exec` # (make -> uv) works in the non-root agent container. Unconditional on purpose: the # image env-sets UV_PYTHON_INSTALL_DIR=/opt/uv/python, so `?=` would not override it. export UV_PYTHON_INSTALL_DIR := $(HOME)/.cache/uv-python ``` Verified in the read-only surface clone: with the override in place, `ward exec smoke` downloads CPython 3.13.14, starts the server, and returns the full 12-tool `tools/list` round-trip with **zero failure markers** and exit 0 (up from immediate permission-denied death). ## Acceptance * `ward exec smoke`, `ward exec test`, and `ward exec lint` pass in a fresh dev-base agent container. * The override is unconditional (overrides the image's root-owned default), documented with a one-line why. * No change to `requires-python` (3.13 stays the target) and no change to CI. ## Priority and follow-up **P1 - hard blocker.** Until this lands on `main`, re-dispatching [#6](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/6) / [#49](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/49)-[#55](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/55) is futile - they all inherit the same red smoke gate. A headless run that applies this Makefile edit first, then runs the gate, passes (the edit is active during validation), so this fix is self-healing. The deeper cause is a **dev-base image bug**: a root-owned `UV_PYTHON_INSTALL_DIR` that non-root agents cannot write, on an image that ships a Python older than its consumers require. That warrants a systemic fix in the image (make the dir agent-writable, or pre-install common Pythons) so no other repo hits this wall. Tracked separately.
coilyco-ops commented 2026-07-05 02:08:20 +00:00
Member
Copy link

🔒 Reserved by ward agent --driver claude — container engineer-claude-eco-app-56 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T02:08:14Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-eco-app-56` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T02:08:14Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
coilysiren commented 2026-07-05 02:08:33 +00:00
Author
Owner
Copy link

🔓 Reservation released by ward container reap — container eco-app (--driver claude) exited without launching the agent (smoke-test death, ward#222/#264), so the hold it took is retracted. A plain ward agent retry no longer needs --force.

— Claude (she/her), via ward agent

<!-- ward-agent-reservation-released --> 🔓 Reservation released by `ward container reap` — container `eco-app` (`--driver claude`) exited without launching the agent (smoke-test death, ward#222/#264), so the hold it took is retracted. A plain `ward agent` retry no longer needs `--force`. <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
coilysiren commented 2026-07-05 02:11:06 +00:00
Author
Owner
Copy link

Cross-links for the full picture:

  • Root cause (fleet): coilyco-flight-deck/agentic-os#327 - the dev-base image bug this issue bandages per-repo (root-owned UV_PYTHON_INSTALL_DIR, no pre-installed 3.13).
  • Prevention (fleet): coilyco-flight-deck/agentic-os#328 - run every app's CI inside the pinned dev-base container so this class of breakage fails in CI, not in a silent dispatch.
  • Prevention (eco-app): #57 - eco-app's adoption of that CI parity, the first mover.

This issue is the immediate unblock: once it lands on main, the trade wave (#6, #53, #55 then #49/#50/#52 -> #51/#54) can re-dispatch and get past smoke.

Cross-links for the full picture: - **Root cause (fleet):** [coilyco-flight-deck/agentic-os#327](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/327) - the dev-base image bug this issue bandages per-repo (root-owned `UV_PYTHON_INSTALL_DIR`, no pre-installed 3.13). - **Prevention (fleet):** [coilyco-flight-deck/agentic-os#328](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/328) - run every app's CI inside the pinned dev-base container so this class of breakage fails in CI, not in a silent dispatch. - **Prevention (eco-app):** [#57](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/57) - eco-app's adoption of that CI parity, the first mover. This issue is the immediate unblock: once it lands on main, the trade wave (#6, #53, #55 then #49/#50/#52 -> #51/#54) can re-dispatch and get past smoke.
coilyco-ops commented 2026-07-05 02:39:15 +00:00
Member
Copy link

🔒 Reserved by ward agent --driver claude — container engineer-claude-eco-app-56 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T02:39:08Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-eco-app-56` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T02:39:08Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
coilysiren commented 2026-07-05 02:42:52 +00:00
Author
Owner
Copy link

WARD-OUTCOME: done - Makefile now redirects uv's managed-Python download to $(HOME)/.cache/uv-python, so smoke/test/lint pass in a fresh dev-base container; landed on main as bf6140d.

Honestly this one was almost anticlimactic - the issue body had already done the hard diagnostic work and even handed over the exact fix, so the job was mostly confirming it end to end rather than discovering anything. The one small surprise: uv resolved CPython 3.14.6, not the 3.13.14 the issue mentioned, because requires-python = ">=3.13" lets it grab the newest managed interpreter. The full suite (262 tests) is green under 3.14 anyway, so no concern, just worth noting for anyone expecting 3.13 exactly.

Confidence is high - all three gates verified green in this very container (smoke exit 0 with the full 12-tool round-trip, no permission-denied), which is the same environment every headless run inherits, so it's genuinely self-healing. The only rough edge is the one the issue already flagged: this is a per-repo bandage over a dev-base image bug (root-owned UV_PYTHON_INSTALL_DIR on an image shipping a too-old Python). That systemic fix is tracked in agentic-os#327/#328, so nothing new to file from here.

WARD-OUTCOME: done - Makefile now redirects uv's managed-Python download to $(HOME)/.cache/uv-python, so smoke/test/lint pass in a fresh dev-base container; landed on main as bf6140d. Honestly this one was almost anticlimactic - the issue body had already done the hard diagnostic work and even handed over the exact fix, so the job was mostly confirming it end to end rather than discovering anything. The one small surprise: uv resolved CPython **3.14.6**, not the 3.13.14 the issue mentioned, because `requires-python = ">=3.13"` lets it grab the newest managed interpreter. The full suite (262 tests) is green under 3.14 anyway, so no concern, just worth noting for anyone expecting 3.13 exactly. Confidence is high - all three gates verified green in this very container (smoke exit 0 with the full 12-tool round-trip, no permission-denied), which is the same environment every headless run inherits, so it's genuinely self-healing. The only rough edge is the one the issue already flagged: this is a per-repo bandage over a dev-base image bug (root-owned UV_PYTHON_INSTALL_DIR on an image shipping a too-old Python). That systemic fix is tracked in agentic-os#327/#328, so nothing new to file from here.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-109
issue-106
ward-salvage/eco-app-a58b14ad
issue-104
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
headless
Safe for autonomous warded engineer carry

  
consult
'A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
priority tier

  
P1
priority tier

  
P2
priority tier

  
P3
priority tier

  
P4
priority tier

No labels
burndown-2026-06
headless
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-gaming/eco-app#56
No description provided.