agent-proxy role skips rebuild when a container is already running - new upstream builds never auto-pulled (trigger gate, plus no_log hides build errors) #434

New issue

Closed

opened 2026-07-02 08:43:53 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-02 08:43:53 +00:00

Owner

Problem

The agent-proxy role does not auto-pull a new upstream build on converge, which is the behaviour expected of it. #431 changed the up-step to docker compose up -d --build --pull always --force-recreate, but left the trigger gate untouched:

when: agent_proxy_compose.changed or (agent_proxy_running.stdout | trim) == ''

So the rebuild only fires when the compose.yaml file content changed or no container is currently running. A healthy container running an older image, with an unchanged compose file, means the step is skipped and a new commit on agent-proxy#main is never pulled. ward exec ansible-sync then reports "ok/unchanged" while the host keeps running stale code. This is the same "upstream fixes never reach the host" failure #431 set out to fix - #431 fixed the command, not the trigger.

Two diagnosability traps in the same step (fix while here)

no_log: true on the up-step (added to hide the tower URL) also swallows all stdout/stderr, so a failed docker compose build surfaces as an opaque failed task with no error. Keep the tower URL out of the log without blinding the build - e.g. pass PROXY_TOWER_BASE_URL via an env/.env file the copy step writes, and drop no_log from the shell task, or split the sensitive assignment from the visible build.
If the docker CLI is absent from ansible's PATH, the whole "Converge" block is skipped via when: agent_proxy_docker.rc == 0 with no warning - it reads as a clean converge. Emit a debug/fail (or at least a warning) when docker is missing on a host where agent_proxy_enabled.

Wanted behaviour

Every converge rebuilds from the current agent-proxy#main (docker's git-context clone re-fetches, and layer cache keeps a no-op cheap), and only recreates the container when the built image actually changed - not a forced restart on every sync. A clean pattern: docker compose build --pull unconditionally, then docker compose up -d without --force-recreate (compose recreates only on image change).
A build failure is visible in the ansible output.
A missing docker CLI on an opted-in host is surfaced, not silently skipped.

Acceptance

With a container already running and compose.yaml unchanged, a new commit on agent-proxy#main is picked up on the next ward exec ansible-sync (new image built, container recreated).
A no-op converge (no upstream change) does not needlessly recreate/restart the container.
A failing build shows its error in the ansible run.

Refs

Follows infrastructure#431 (added --build --pull but not the trigger) and #430.
Surfaced by Kai: "is ansible supposed to auto pull the latest build? it should be doing that."

## Problem The `agent-proxy` role does not auto-pull a new upstream build on converge, which is the behaviour expected of it. #431 changed the up-step to `docker compose up -d --build --pull always --force-recreate`, but left the trigger gate untouched: ```yaml when: agent_proxy_compose.changed or (agent_proxy_running.stdout | trim) == '' ``` So the rebuild only fires when the **compose.yaml file content changed** or **no container is currently running**. A healthy container running an older image, with an unchanged compose file, means the step is skipped and a new commit on `agent-proxy#main` is never pulled. `ward exec ansible-sync` then reports "ok/unchanged" while the host keeps running stale code. This is the same "upstream fixes never reach the host" failure #431 set out to fix - #431 fixed the command, not the trigger. ## Two diagnosability traps in the same step (fix while here) 1. `no_log: true` on the up-step (added to hide the tower URL) also swallows all stdout/stderr, so a failed `docker compose build` surfaces as an opaque failed task with no error. Keep the tower URL out of the log without blinding the build - e.g. pass `PROXY_TOWER_BASE_URL` via an env/`.env` file the copy step writes, and drop `no_log` from the shell task, or split the sensitive assignment from the visible build. 2. If the `docker` CLI is absent from ansible's PATH, the whole "Converge" block is skipped via `when: agent_proxy_docker.rc == 0` with **no** warning - it reads as a clean converge. Emit a `debug`/`fail` (or at least a warning) when docker is missing on a host where `agent_proxy_enabled`. ## Wanted behaviour - Every converge rebuilds from the current `agent-proxy#main` (docker's git-context clone re-fetches, and layer cache keeps a no-op cheap), and only recreates the container when the built image actually changed - not a forced restart on every sync. A clean pattern: `docker compose build --pull` unconditionally, then `docker compose up -d` **without** `--force-recreate` (compose recreates only on image change). - A build failure is visible in the ansible output. - A missing docker CLI on an opted-in host is surfaced, not silently skipped. ## Acceptance - With a container already running and compose.yaml unchanged, a new commit on `agent-proxy#main` is picked up on the next `ward exec ansible-sync` (new image built, container recreated). - A no-op converge (no upstream change) does not needlessly recreate/restart the container. - A failing build shows its error in the ansible run. ## Refs - Follows infrastructure#431 (added `--build --pull` but not the trigger) and #430. - Surfaced by Kai: "is ansible supposed to auto pull the latest build? it should be doing that."

coilyco-ops commented

2026-07-02 08:44:00 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-434 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T08:43:58Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-434` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T08:43:58Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 08:48:54 +00:00

fix(agent-proxy): auto-pull upstream builds every converge (closes #434)

coilysiren closed this issue

2026-07-02 08:48:54 +00:00

coilysiren referenced this issue from a commit

2026-07-02 08:48:54 +00:00

Merge issue-434: agent-proxy auto-pulls upstream builds (closes #434)

coilysiren commented

2026-07-02 08:50:14 +00:00

Author

Owner

WARD-OUTCOME: done - agent-proxy now rebuilds from agent-proxy.git#main every converge and recreates only on a real image/config change.

The fix itself was small once the shape was clear. The real work was untangling three things the old step conflated. The trigger gate was doing the compose-changed / no-container dance to avoid churn, but that same guard is exactly what froze the image on a healthy host. Splitting build (unconditional docker compose build --pull) from up (plain docker compose up -d, let compose decide whether to recreate) makes each concern honest, and compose's own image-diff logic gives the no-op-is-cheap property for free without any hand-rolled docker ps bookkeeping.

What fought back: the no_log was load-bearing for the wrong reason. Untangling it meant moving the tower URL into a 0600 .env that compose auto-loads, which happens to read cleaner than the inline environment: block anyway. Small surprise on the way out - the repo's own code-comments hook rejects inline YAML comments below the header line, so all my rationale had to move into docs/agent-proxy.md instead of living next to the tasks. Good discipline, mildly annoying mid-flow.

Confidence: high on the logic, which mirrors the pattern the issue itself proposed. The one thing I could not exercise here is a real converge on a mac with docker (this container has no docker daemon and the role is mac-gated), so the changed_when stderr-matching on up -d output is reasoned, not observed. Worth a glance on the next real ansible-sync; if the ok/changed reporting reads off, it is a one-line tweak to the match strings, not a design issue. No follow-ups worth filing.

Claude (she/her), via ward agent

WARD-OUTCOME: done - agent-proxy now rebuilds from agent-proxy.git#main every converge and recreates only on a real image/config change. The fix itself was small once the shape was clear. The real work was untangling three things the old step conflated. The trigger gate was doing the compose-changed / no-container dance to *avoid* churn, but that same guard is exactly what froze the image on a healthy host. Splitting build (unconditional `docker compose build --pull`) from up (plain `docker compose up -d`, let compose decide whether to recreate) makes each concern honest, and compose's own image-diff logic gives the no-op-is-cheap property for free without any hand-rolled `docker ps` bookkeeping. What fought back: the `no_log` was load-bearing for the wrong reason. Untangling it meant moving the tower URL into a 0600 `.env` that compose auto-loads, which happens to read cleaner than the inline `environment:` block anyway. Small surprise on the way out - the repo's own code-comments hook rejects inline YAML comments below the header line, so all my rationale had to move into docs/agent-proxy.md instead of living next to the tasks. Good discipline, mildly annoying mid-flow. Confidence: high on the logic, which mirrors the pattern the issue itself proposed. The one thing I could not exercise here is a real converge on a mac with docker (this container has no docker daemon and the role is mac-gated), so the `changed_when` stderr-matching on `up -d` output is reasoned, not observed. Worth a glance on the next real `ansible-sync`; if the ok/changed reporting reads off, it is a one-line tweak to the match strings, not a design issue. No follow-ups worth filing. - Claude (she/her), via ward agent

coilysiren referenced this issue from a commit

2026-07-02 09:42:33 +00:00

feat(ansible): add host-local SigNoz OTLP backend role (closes #435)