coilyco-flight-deck/agentic-os

Watch

Stop dev-base PYTHONPATH from overriding pinned Python tools #771

New issue

Closed

opened 2026-07-28 07:48:46 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-28 07:48:46 +00:00

Member

Problem

The full dev-base image exports PYTHONPATH=/opt/agentic-os/python globally from docker/dev-base/Dockerfile so embedded AOS operator modules are importable.

That environment leaks into every Python process in the container. Python console entry points created inside isolated environments, including pre-commit hook environments, resolve the image snapshot of agentic_os before the package installed from their pinned revision. A consumer pin therefore does not determine which validator implementation runs.

Evidence

coilyco-flight-deck/infrastructure pins the AOS hook catalog at v0.65.0, but lint inside the moving agentic-os:release image executed current catalog-trifecta and dead-cross-links behavior.
The same infrastructure checkout passes every hook with a fresh pre-commit cache when the image-level PYTHONPATH is absent.
Exporting the image-level path reproduces the same validator failures locally.
The local and remote v0.65.0 tag objects match. The tag and pre-commit cache are not the source of drift.
The consumer-side CI evidence is tracked in coilyco-flight-deck/infrastructure#656.

Required outcome

Remove the global runtime PYTHONPATH from the dev-base image.
Preserve AOS-owned Python operator modules through a scoped mechanism, such as an AOSguard-only wrapper or an installation boundary that does not enter isolated virtual environments.
Keep pre-commit and other isolated Python tools governed by their own installed package versions.
Add an owning image/runtime regression that proves an isolated environment imports its own sentinel package instead of the embedded AOS package. The test must exercise behavior rather than restating Dockerfile configuration.
Keep the existing AOSguard Python-backed operator leaves working in the built full image.

Acceptance

A consumer hook pinned to an older AOS revision executes that revision inside agentic-os:release.
Infrastructure lint no longer receives validators newer than its configured pin.
AOSguard Python-backed operations remain available without workflow-specific environment workarounds.
The full image verification, AOS tests, and canonical pre-commit suite pass.
A push to canonical main triggers the normal dev-base publication path for the corrected image.

coilyco-flight-deck/infrastructure#656
docker/dev-base/Dockerfile
docs/dev-base-image.md

## Problem The full dev-base image exports `PYTHONPATH=/opt/agentic-os/python` globally from `docker/dev-base/Dockerfile` so embedded AOS operator modules are importable. That environment leaks into every Python process in the container. Python console entry points created inside isolated environments, including pre-commit hook environments, resolve the image snapshot of `agentic_os` before the package installed from their pinned revision. A consumer pin therefore does not determine which validator implementation runs. ## Evidence * `coilyco-flight-deck/infrastructure` pins the AOS hook catalog at `v0.65.0`, but lint inside the moving `agentic-os:release` image executed current catalog-trifecta and dead-cross-links behavior. * The same infrastructure checkout passes every hook with a fresh pre-commit cache when the image-level `PYTHONPATH` is absent. * Exporting the image-level path reproduces the same validator failures locally. * The local and remote `v0.65.0` tag objects match. The tag and pre-commit cache are not the source of drift. * The consumer-side CI evidence is tracked in `coilyco-flight-deck/infrastructure#656`. ## Required outcome * Remove the global runtime `PYTHONPATH` from the dev-base image. * Preserve AOS-owned Python operator modules through a scoped mechanism, such as an AOSguard-only wrapper or an installation boundary that does not enter isolated virtual environments. * Keep pre-commit and other isolated Python tools governed by their own installed package versions. * Add an owning image/runtime regression that proves an isolated environment imports its own sentinel package instead of the embedded AOS package. The test must exercise behavior rather than restating Dockerfile configuration. * Keep the existing AOSguard Python-backed operator leaves working in the built full image. ## Acceptance * A consumer hook pinned to an older AOS revision executes that revision inside `agentic-os:release`. * Infrastructure lint no longer receives validators newer than its configured pin. * AOSguard Python-backed operations remain available without workflow-specific environment workarounds. * The full image verification, AOS tests, and canonical pre-commit suite pass. * A push to canonical main triggers the normal dev-base publication path for the corrected image. ## Related * `coilyco-flight-deck/infrastructure#656` * `docker/dev-base/Dockerfile` * `docs/dev-base-image.md`

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-29 03:45:21 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-08-05 10:25:51 +00:00

coilyco-ops referenced this issue from a commit

2026-08-10 00:40:46 +00:00

fix(dev-base): scope the embedded agentic_os to the system interpreter

coilyco-ops closed this issue

2026-08-10 00:40:47 +00:00

coilyco-ops commented

2026-08-10 00:41:57 +00:00

Author

Member

Landed on main as 479924cf, which also closes #970 and #971 as the same root cause.

What changed

docker/dev-base/full/Dockerfile drops the global ENV PYTHONPATH=/opt/agentic-os/python and installs the operator modules where the system interpreter already looks:

COPY --from=aosguard-python . /usr/lib/python3/dist-packages/agentic_os/

That keeps python3 -m agentic_os.* working for the two surfaces that need it inside the image, the aosguard ops actions leaves in .specgen/guardfiles/aosguard/actions.kdl and the prod-install-ref verb, while a virtualenv, which excludes dist-packages unless it opts in, is left to whatever pre-commit installed from the pinned rev.

This takes the first of the two directions #971 offered. rev: stays meaningful rather than becoming a documented lie.

The regression

verify-common.sh now builds a throwaway venv, drops a sentinel agentic_os into its purelib, and asserts the sentinel is what imports. Behavior, not an assertion that PYTHONPATH is unset, per the required outcome here. It runs during the image build, so a regression fails the build instead of shipping.

I confirmed the check is load-bearing rather than vacuous, by running the same mechanism under both conditions:

--- no PYTHONPATH (post-fix) ---      PASS: venv copy wins
--- with PYTHONPATH (today's image) --- resolved: image
FAIL as expected: the regression catches the shadowing

On the stranded-fix half

The comment on #971 is the sharper half: the image was the real source of validator truth in CI, but dev-base-publish keys off docker/**, so a fix under agentic_os/ had no path to a release. Scoping the interpreter dissolves that. The image stops deciding which validators a consumer runs, so an agentic_os/-only fix reaches consumers through a normal rev: bump instead of waiting for an unrelated docker/** change to carry it.

Verification state

Green: ward exec repo-test-gate (511 passed, full pre-commit suite), and the sentinel mechanism proven above on a host.

Pending: the image build itself. dist-packages being on the system interpreter's sys.path is the assumption the change rests on, and I could not run a container on this host to confirm it directly. It is self-guarding: verify-common.sh already runs python3 -m agentic_os.forgejo_actions_list --help at build time, so if the assumption is wrong the build fails and no image publishes. I am watching the dev-base-publish run and will follow up here with the result.

Landed on `main` as `479924cf`, which also closes #970 and #971 as the same root cause. ## What changed `docker/dev-base/full/Dockerfile` drops the global `ENV PYTHONPATH=/opt/agentic-os/python` and installs the operator modules where the system interpreter already looks: ``` COPY --from=aosguard-python . /usr/lib/python3/dist-packages/agentic_os/ ``` That keeps `python3 -m agentic_os.*` working for the two surfaces that need it inside the image, the `aosguard ops actions` leaves in `.specgen/guardfiles/aosguard/actions.kdl` and the `prod-install-ref` verb, while a virtualenv, which excludes `dist-packages` unless it opts in, is left to whatever pre-commit installed from the pinned `rev`. This takes the first of the two directions #971 offered. `rev:` stays meaningful rather than becoming a documented lie. ## The regression `verify-common.sh` now builds a throwaway venv, drops a sentinel `agentic_os` into its purelib, and asserts the sentinel is what imports. Behavior, not an assertion that `PYTHONPATH` is unset, per the required outcome here. It runs during the image build, so a regression fails the build instead of shipping. I confirmed the check is load-bearing rather than vacuous, by running the same mechanism under both conditions: ``` --- no PYTHONPATH (post-fix) --- PASS: venv copy wins --- with PYTHONPATH (today's image) --- resolved: image FAIL as expected: the regression catches the shadowing ``` ## On the stranded-fix half The comment on #971 is the sharper half: the image was the real source of validator truth in CI, but `dev-base-publish` keys off `docker/**`, so a fix under `agentic_os/` had no path to a release. Scoping the interpreter dissolves that. The image stops deciding which validators a consumer runs, so an `agentic_os/`-only fix reaches consumers through a normal `rev:` bump instead of waiting for an unrelated `docker/**` change to carry it. ## Verification state Green: `ward exec repo-test-gate` (511 passed, full pre-commit suite), and the sentinel mechanism proven above on a host. Pending: the image build itself. `dist-packages` being on the system interpreter's `sys.path` is the assumption the change rests on, and I could not run a container on this host to confirm it directly. It is self-guarding: `verify-common.sh` already runs `python3 -m agentic_os.forgejo_actions_list --help` at build time, so if the assumption is wrong the build fails and no image publishes. I am watching the `dev-base-publish` run and will follow up here with the result.

coilyco-ops referenced this issue

2026-08-10 00:42:09 +00:00

CI image PYTHONPATH shadows rev-pinned pre-commit hook clones #970

coilyco-ops referenced this issue

2026-08-10 00:42:12 +00:00

The release image's PYTHONPATH shadows the pinned hook rev in CI #971

coilyco-ops referenced this issue from a commit

2026-08-10 01:05:17 +00:00

fix(dev-base): resolve both python paths from the interpreter, not by hand

coilyco-ops commented

2026-08-10 01:28:38 +00:00

Author

Member

Verified. publish-full is green on 123eff46, so the image built and its own in-image verification passed.

That is the proof the earlier comment was waiting on, and it covers both halves at once:

python3 -m agentic_os.forgejo_actions_list --help runs inside the built image, so the operator modules are importable from the system interpreter without any PYTHONPATH.
The sentinel venv imports its own agentic_os, so an isolated environment is no longer shadowed. A pinned rev: decides which validator a consumer runs.

What the first attempt cost, and what it taught

479924cf failed at publish-full while all five language payloads passed. Forgejo 15 exposes no job-log route, so the error text was unreachable and there was no way to tell which of the two new steps broke. Rather than guess, 123eff46 removed both hardcoded assumptions:

The modules now land wherever site.getsitepackages() reports instead of an assumed /usr/lib/python3/dist-packages, and the same layer proves the import.
The isolation check globs the venv's own lib/python*/site-packages instead of asking sysconfig.

The second one was the more serious defect, and it is worth recording. sysconfig's default scheme is distribution patched on Debian. Had it reported the system scheme from inside the venv, the check would have written its sentinel agentic_os into the image rather than into the throwaway venv, corrupting the thing it was verifying. The glob is anchored under the temp dir, so that is now impossible by construction rather than merely unlikely.

I cannot say which of the two caused the original failure, only that neither can now. #987 tracks the log gap that made the distinction unavailable.

Verified. `publish-full` is green on `123eff46`, so the image built and its own in-image verification passed. That is the proof the earlier comment was waiting on, and it covers both halves at once: * `python3 -m agentic_os.forgejo_actions_list --help` runs inside the built image, so the operator modules are importable from the system interpreter without any `PYTHONPATH`. * The sentinel venv imports its own `agentic_os`, so an isolated environment is no longer shadowed. A pinned `rev:` decides which validator a consumer runs. ## What the first attempt cost, and what it taught `479924cf` failed at `publish-full` while all five language payloads passed. Forgejo 15 exposes no job-log route, so the error text was unreachable and there was no way to tell which of the two new steps broke. Rather than guess, `123eff46` removed both hardcoded assumptions: * The modules now land wherever `site.getsitepackages()` reports instead of an assumed `/usr/lib/python3/dist-packages`, and the same layer proves the import. * The isolation check globs the venv's own `lib/python*/site-packages` instead of asking `sysconfig`. The second one was the more serious defect, and it is worth recording. `sysconfig`'s default scheme is distribution patched on Debian. Had it reported the system scheme from inside the venv, the check would have written its sentinel `agentic_os` into the **image** rather than into the throwaway venv, corrupting the thing it was verifying. The glob is anchored under the temp dir, so that is now impossible by construction rather than merely unlikely. I cannot say which of the two caused the original failure, only that neither can now. #987 tracks the log gap that made the distinction unavailable.

coilyco-ops referenced this issue

2026-08-14 04:30:59 +00:00

Pin agent-compose 2.20.0 for the seat identity override #1052

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#771

No description provided.

Rows
Columns