coilyco-flight-deck/agent-proxy

Fork 0

agent-proxy container crash-loops: pydantic-settings + prometheus-client undeclared, [project.dependencies] is a malformed table #22

New issue

Closed

opened 2026-07-02 07:59:54 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-02 07:59:54 +00:00

Owner

Symptom

The agent-proxy container built from main crash-loops on startup:

File "/app/app/config.py", line 19, in <module>
    from pydantic_settings import BaseSettings, SettingsConfigDict
ModuleNotFoundError: No module named 'pydantic_settings'

docker compose up restarts it forever, so it never binds a port and every healthcheck against it fails. Seen on Kai's mac after the infrastructure agent-proxy ansible rollout (infrastructure#425).

Root cause - the dependency metadata is malformed

pyproject.toml declares runtime deps as a table:

[project.dependencies]
fastapi = ">=0.115.0"
pydantic = ">=2.9.0"
hypercorn = ">=0.17.0"

That is not valid PEP 621 - dependencies must be an array of strings under [project], not a subtable (the table form is Poetry-flavoured and belongs under [tool.poetry.dependencies]). Because it is malformed, the resolver does not treat these as project deps: the committed uv.lock contains only the dev tools (black, mypy, pytest, ruff) and their transitives - fastapi, pydantic, and hypercorn are absent from the lock entirely.

On top of that, two genuinely-runtime packages are declared nowhere, and two more are stranded in the dev extra. The actual third-party runtime imports across app/ are:

fastapi - declared (table)
pydantic - declared (table)
pydantic_settings (app/config.py) - undeclared -> the crash
prometheus_client (app/obs.py) - undeclared -> the next crash once pydantic-settings is fixed
httpx (app/upstream.py) - only in dev extra
structlog (app/obs.py) - only in dev extra
hypercorn (lazy import in app/__init__.py) - declared (table)

The image only limps as far as it does because the Dockerfile runs uv sync --extra dev (so httpx/structlog happen to be present) and uv.lock is not copied into the build, so uv re-resolves loosely from the malformed pyproject.

Fix

Rewrite [project.dependencies] as a proper PEP 621 array under [project], listing every runtime import:

[project]
dependencies = [
  "fastapi>=0.115.0",
  "pydantic>=2.9.0",
  "pydantic-settings>=2.4.0",
  "hypercorn>=0.17.0",
  "httpx>=0.27.0",
  "structlog>=24.0.0",
  "prometheus-client>=0.20.0",
]

Leave only true dev tools (pytest, black, ruff, mypy, and uvicorn if still wanted) in [project.optional-dependencies].dev.

Regenerate uv.lock so it actually contains the runtime deps, and commit it.
Make the Dockerfile reproducible and runtime-only: COPY pyproject.toml uv.lock ., then uv sync --frozen --no-dev (drop --extra dev from the runtime image). Confirm python -m app.main imports clean.

Acceptance

docker build from main produces an image where python -c "import app" (and python -m app.main up to the bind) imports with no ModuleNotFoundError.
uv.lock contains fastapi, pydantic, pydantic-settings, hypercorn, httpx, structlog, prometheus-client.
The container stays up instead of restart-looping.

Refs

Container scaffolding added under infrastructure#425.
Pairs with infrastructure#430 (a latent host-port mapping bug that only surfaces once the container actually boots) and the infra image-rebuild gap (filed separately).

## Symptom The `agent-proxy` container built from `main` crash-loops on startup: ``` File "/app/app/config.py", line 19, in <module> from pydantic_settings import BaseSettings, SettingsConfigDict ModuleNotFoundError: No module named 'pydantic_settings' ``` `docker compose up` restarts it forever, so it never binds a port and every healthcheck against it fails. Seen on Kai's mac after the infrastructure `agent-proxy` ansible rollout (infrastructure#425). ## Root cause - the dependency metadata is malformed `pyproject.toml` declares runtime deps as a **table**: ```toml [project.dependencies] fastapi = ">=0.115.0" pydantic = ">=2.9.0" hypercorn = ">=0.17.0" ``` That is not valid PEP 621 - `dependencies` must be an **array of strings under `[project]`**, not a subtable (the table form is Poetry-flavoured and belongs under `[tool.poetry.dependencies]`). Because it is malformed, the resolver does not treat these as project deps: the committed `uv.lock` contains only the dev tools (black, mypy, pytest, ruff) and their transitives - `fastapi`, `pydantic`, and `hypercorn` are absent from the lock entirely. On top of that, two genuinely-runtime packages are declared **nowhere**, and two more are stranded in the `dev` extra. The actual third-party runtime imports across `app/` are: - `fastapi` - declared (table) - `pydantic` - declared (table) - `pydantic_settings` (`app/config.py`) - **undeclared** -> the crash - `prometheus_client` (`app/obs.py`) - **undeclared** -> the next crash once pydantic-settings is fixed - `httpx` (`app/upstream.py`) - only in `dev` extra - `structlog` (`app/obs.py`) - only in `dev` extra - `hypercorn` (lazy import in `app/__init__.py`) - declared (table) The image only limps as far as it does because the Dockerfile runs `uv sync --extra dev` (so httpx/structlog happen to be present) and `uv.lock` is not copied into the build, so uv re-resolves loosely from the malformed pyproject. ## Fix 1. Rewrite `[project.dependencies]` as a proper PEP 621 array under `[project]`, listing every runtime import: ```toml [project] dependencies = [ "fastapi>=0.115.0", "pydantic>=2.9.0", "pydantic-settings>=2.4.0", "hypercorn>=0.17.0", "httpx>=0.27.0", "structlog>=24.0.0", "prometheus-client>=0.20.0", ] ``` Leave only true dev tools (pytest, black, ruff, mypy, and uvicorn if still wanted) in `[project.optional-dependencies].dev`. 2. Regenerate `uv.lock` so it actually contains the runtime deps, and commit it. 3. Make the Dockerfile reproducible and runtime-only: `COPY pyproject.toml uv.lock .`, then `uv sync --frozen --no-dev` (drop `--extra dev` from the runtime image). Confirm `python -m app.main` imports clean. ## Acceptance - `docker build` from `main` produces an image where `python -c "import app"` (and `python -m app.main` up to the bind) imports with no `ModuleNotFoundError`. - `uv.lock` contains fastapi, pydantic, pydantic-settings, hypercorn, httpx, structlog, prometheus-client. - The container stays up instead of restart-looping. ## Refs - Container scaffolding added under infrastructure#425. - Pairs with infrastructure#430 (a latent host-port mapping bug that only surfaces once the container actually boots) and the infra image-rebuild gap (filed separately).

coilyco-ops commented

2026-07-02 08:00:12 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-22 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T08:00:09Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-22` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T08:00:09Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from coilyco-flight-deck/infrastructure

2026-07-02 08:00:24 +00:00

agent-proxy: container binds PROXY_PORT=8082 but compose maps to container :8080 - proxy never serves on 127.0.0.1:8082 #430

coilysiren referenced this issue from a commit

2026-07-02 08:10:53 +00:00

Fix crash-loop: correct dependency metadata and restore the real OpenAI surface

coilysiren closed this issue

2026-07-02 08:10:54 +00:00

coilysiren commented

2026-07-02 08:11:39 +00:00

Author

Owner

WARD-OUTCOME: done - PEP 621 deps + regenerated lock + runtime-only Dockerfile, and restored the real OpenAI surface a scaffold commit had stubbed out; container now boots, binds, and serves on main.

The dependency half was exactly as diagnosed and went quickly: rewrite the malformed table as a PEP 621 array, add the two undeclared runtime imports, regenerate the lock, make the Dockerfile frozen and runtime-only. What fought back was everything hiding behind the crash the issue reported. Once pydantic-settings and prometheus-client were declared, the import chain kept failing past the point the issue's static analysis reached: app/__init__.py imported a run_queue_worker that does not exist, app/main.py did from app.main import run_app (a self-import that never resolves), and the bind used a hypercorn run API that is not real. Chasing those down, I found the container-scaffold commit (9535587) had clobbered the real, tested OpenAI surface in app/main.py down to a stub app/api.py returning hardcoded responses, and every test_api case was red against it. The right fix was to restore the real surface from cb0c5f7 (module-level app, lifespan-managed queue, real /metrics and model->404 validation) and re-wire python -m app.main to serve it under hypercorn.

I also added tiktoken as a runtime dep. It is a lazy import with no fallback sitting on the request path via apply_context_budget, so a container without it boots but 500s every chat request. The issue's list missed it.

Confidence is high on the runtime path: verified under runtime-only deps that import is clean, the server binds and stays up, and /healthz, /metrics, /v1/models and unknown-model->404 all respond correctly. The full suite is 22 passed, 5 skipped, 0 failed. Two rough edges worth a follow-up: I could not run an actual docker build here (no docker daemon in the container), though I validated the exact uv sync --frozen --no-dev path the Dockerfile uses; and the 5 skipped tests are async cases that need pytest-asyncio wired into the dev deps, a pre-existing harness gap unrelated to this crash. Both are small and I would file them rather than stretch this fix further.

WARD-OUTCOME: done - PEP 621 deps + regenerated lock + runtime-only Dockerfile, and restored the real OpenAI surface a scaffold commit had stubbed out; container now boots, binds, and serves on main. The dependency half was exactly as diagnosed and went quickly: rewrite the malformed table as a PEP 621 array, add the two undeclared runtime imports, regenerate the lock, make the Dockerfile frozen and runtime-only. What fought back was everything hiding behind the crash the issue reported. Once pydantic-settings and prometheus-client were declared, the import chain kept failing past the point the issue's static analysis reached: `app/__init__.py` imported a `run_queue_worker` that does not exist, `app/main.py` did `from app.main import run_app` (a self-import that never resolves), and the bind used a hypercorn `run` API that is not real. Chasing those down, I found the container-scaffold commit (9535587) had clobbered the real, tested OpenAI surface in `app/main.py` down to a stub `app/api.py` returning hardcoded responses, and every `test_api` case was red against it. The right fix was to restore the real surface from cb0c5f7 (module-level `app`, lifespan-managed queue, real /metrics and model->404 validation) and re-wire `python -m app.main` to serve it under hypercorn. I also added tiktoken as a runtime dep. It is a lazy import with no fallback sitting on the request path via apply_context_budget, so a container without it boots but 500s every chat request. The issue's list missed it. Confidence is high on the runtime path: verified under runtime-only deps that import is clean, the server binds and stays up, and /healthz, /metrics, /v1/models and unknown-model->404 all respond correctly. The full suite is 22 passed, 5 skipped, 0 failed. Two rough edges worth a follow-up: I could not run an actual `docker build` here (no docker daemon in the container), though I validated the exact `uv sync --frozen --no-dev` path the Dockerfile uses; and the 5 skipped tests are async cases that need pytest-asyncio wired into the dev deps, a pre-existing harness gap unrelated to this crash. Both are small and I would file them rather than stretch this fix further.

coilysiren referenced this issue

2026-07-02 08:19:22 +00:00

[ward-salvage] agent-proxy: unmerged container work on ward-salvage/agent-proxy-a5458e6e #23

coilysiren referenced this issue

2026-07-02 08:19:22 +00:00

container image starts with missing runtime dependencies and wrong Python path #21