Director-surface dispatches inherit stale WARD_VERSION pins #700

Closed
opened 2026-07-08 21:17:58 +00:00 by coilyco-ops · 10 comments
Member

Bug

Director-surface forwarded engineer runs are downloading an older ward release because the surface exports WARD_VERSION=v0.463.0, even though the latest ward tag is v0.465.0.

Observed

A newly dispatched engineer container logged:

ward-container: downloading ward v0.463.0 for linux-arm64

But Forgejo tags show latest is v0.465.0:

refs/tags/v0.463.0
refs/tags/v0.464.0
refs/tags/v0.465.0

The director surface environment contains:

WARD_VERSION=v0.463.0
WARD_VERSION source appears as pinned via WARD_VERSION/--ward-version in reaper diagnostics

So child engineers inherit the stale version pin and do not get fixes that already landed in newer ward releases. This is especially visible while testing freshly landed agent/bootstrap fixes.

Root cause from code read

cmd/ward/container.go defaults the child container ward version to the host binary version unless --ward-version / WARD_AGENT_VERSION is set. The entrypoint only resolves latest when WARD_VERSION is empty.

A director surface, however, is itself an older container with WARD_VERSION set. Forwarded dispatches from that surface should not accidentally treat the surface's own bootstrap pin as an operator-requested pin for future sibling engineers.

Relevant paths to inspect:

  • cmd/ward/container.go (buildUpPlan, wardVersion := Version, --ward-version override)
  • cmd/ward/agent_director.go / agent_director_heartbeat.go (propagating wardVersion)
  • cmd/ward/containerassets/entrypoint.sh (downloads WARD_VERSION, resolves latest only when empty)
  • dispatch broker forwarding of --ward-version

Expected behavior

A director surface should distinguish:

  • explicit operator pin - forward --ward-version and keep using that exact release
  • surface bootstrap version - do not freeze future engineers to this stale value; let the host/current ward or latest resolution win

At minimum, a normal director-surface dispatch with no explicit --ward-version should not download v0.463.0 after v0.465.0 is latest.

Do

  • Identify where the stale WARD_VERSION from the surface is converted into an engineer pin.
  • Ensure only an explicit --ward-version / WARD_AGENT_VERSION from the operator is forwarded as a pin.
  • Let unpinned engineer dispatches use the host ward default or in-container latest resolution, according to ward's intended current policy.
  • Keep downgrade protection intact when an explicit pin is used.
  • Add tests for director-surface forwarding from an older container: no explicit pin must not forward the stale surface WARD_VERSION.
  • Make startup output clear about whether the version came from an explicit pin, host ward, or latest resolution.

Acceptance

  • From an older director surface with WARD_VERSION=v0.463.0, warded <issue> --harness codex starts a child engineer on current/latest ward rather than v0.463.0, unless --ward-version v0.463.0 was explicitly passed.
  • Explicit --ward-version still pins and still obeys downgrade guard semantics.
  • ward exec test passes.

Filed from the read-only director surface after a test engineer downloaded v0.463.0 despite latest being v0.465.0.

## Bug Director-surface forwarded engineer runs are downloading an older ward release because the surface exports `WARD_VERSION=v0.463.0`, even though the latest ward tag is `v0.465.0`. ## Observed A newly dispatched engineer container logged: ```text ward-container: downloading ward v0.463.0 for linux-arm64 ``` But Forgejo tags show latest is `v0.465.0`: ```text refs/tags/v0.463.0 refs/tags/v0.464.0 refs/tags/v0.465.0 ``` The director surface environment contains: ```text WARD_VERSION=v0.463.0 WARD_VERSION source appears as pinned via WARD_VERSION/--ward-version in reaper diagnostics ``` So child engineers inherit the stale version pin and do not get fixes that already landed in newer ward releases. This is especially visible while testing freshly landed agent/bootstrap fixes. ## Root cause from code read `cmd/ward/container.go` defaults the child container ward version to the host binary version unless `--ward-version` / `WARD_AGENT_VERSION` is set. The entrypoint only resolves latest when `WARD_VERSION` is empty. A director surface, however, is itself an older container with `WARD_VERSION` set. Forwarded dispatches from that surface should not accidentally treat the surface's own bootstrap pin as an operator-requested pin for future sibling engineers. Relevant paths to inspect: * `cmd/ward/container.go` (`buildUpPlan`, `wardVersion := Version`, `--ward-version` override) * `cmd/ward/agent_director.go` / `agent_director_heartbeat.go` (propagating `wardVersion`) * `cmd/ward/containerassets/entrypoint.sh` (downloads `WARD_VERSION`, resolves latest only when empty) * dispatch broker forwarding of `--ward-version` ## Expected behavior A director surface should distinguish: * **explicit operator pin** - forward `--ward-version` and keep using that exact release * **surface bootstrap version** - do not freeze future engineers to this stale value; let the host/current ward or latest resolution win At minimum, a normal director-surface dispatch with no explicit `--ward-version` should not download `v0.463.0` after `v0.465.0` is latest. ## Do * Identify where the stale `WARD_VERSION` from the surface is converted into an engineer pin. * Ensure only an explicit `--ward-version` / `WARD_AGENT_VERSION` from the operator is forwarded as a pin. * Let unpinned engineer dispatches use the host ward default or in-container latest resolution, according to ward's intended current policy. * Keep downgrade protection intact when an explicit pin is used. * Add tests for director-surface forwarding from an older container: no explicit pin must not forward the stale surface `WARD_VERSION`. * Make startup output clear about whether the version came from an explicit pin, host ward, or latest resolution. ## Acceptance * From an older director surface with `WARD_VERSION=v0.463.0`, `warded <issue> --harness codex` starts a child engineer on current/latest ward rather than `v0.463.0`, unless `--ward-version v0.463.0` was explicitly passed. * Explicit `--ward-version` still pins and still obeys downgrade guard semantics. * `ward exec test` passes. Filed from the read-only director surface after a test engineer downloaded `v0.463.0` despite latest being `v0.465.0`.
Author
Member

🔒 Reserved by ward agent --harness codex — container engineer-codex-ward-700 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-08T21:18:05Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/ward#700 · branch issue-700 · harness codex · workflow direct-main
  • Run: engineer-codex-ward-700 · ward v0.465.0 · dispatched 2026-07-08T21:18:05Z
  • Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.465.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --harness codex` — container `engineer-codex-ward-700` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-08T21:18:05Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#700` · branch `issue-700` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-ward-700` · ward `v0.465.0` · dispatched `2026-07-08T21:18:05Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.465.0). </details> <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

Correction: this issue was filed from a wrong premise. The engineer downloading v0.463.0 was not the root cause of the Codex adapter still failing after “latest”; the Codex fix commit de9e61abcc211b716e519e529b9ac673ab707d0e is not on main or tag v0.465.0.

Remote refs checked from the director surface:

22ded39465250c6da99731287e9291a170cd73f4 refs/heads/main
de9e61abcc211b716e519e529b9ac673ab707d0e refs/heads/issue-691
de9e61abcc211b716e519e529b9ac673ab707d0e refs/heads/ward-salvage/ward-b8ccdc84
22ded39465250c6da99731287e9291a170cd73f4 refs/tags/v0.465.0

So the real active issue is still coilyco-flight-deck/ward#699: the reaper rejected a valid Fixes #691 closing reference, salvaged the Codex fix, and left it off main. This ticket can be closed as invalid unless a separate stale-version inheritance problem reproduces after the issue-691 fix actually lands.

Correction: this issue was filed from a wrong premise. The engineer downloading `v0.463.0` was not the root cause of the Codex adapter still failing after “latest”; the Codex fix commit `de9e61abcc211b716e519e529b9ac673ab707d0e` is not on `main` or tag `v0.465.0`. Remote refs checked from the director surface: ```text 22ded39465250c6da99731287e9291a170cd73f4 refs/heads/main de9e61abcc211b716e519e529b9ac673ab707d0e refs/heads/issue-691 de9e61abcc211b716e519e529b9ac673ab707d0e refs/heads/ward-salvage/ward-b8ccdc84 22ded39465250c6da99731287e9291a170cd73f4 refs/tags/v0.465.0 ``` So the real active issue is still coilyco-flight-deck/ward#699: the reaper rejected a valid `Fixes #691` closing reference, salvaged the Codex fix, and left it off `main`. This ticket can be closed as invalid unless a separate stale-version inheritance problem reproduces after the `issue-691` fix actually lands.
Author
Member

Reopened after Kai pointed out the stronger repro: the explicit version override appears to be ignored or overwritten.

Observed dispatch from this surface:

warded coilyco-flight-deck/ward#700 --harness codex --force --ward-version v0.465.0
ward dispatch broker: forwarded `ward agent engineer coilyco-flight-deck/ward#700 --harness codex --ward-version v0.465.0 --force` ...

The reservation comment also records:

- **Run:** `engineer-codex-ward-700` · ward `v0.465.0` · dispatched `2026-07-08T21:18:05Z`

But Kai reports the engineer container log still shows:

ward-container: downloading ward v0.463.0 for linux-arm64

So this is not just accidental inheritance from an unpinned director surface. The dispatch argv and reservation say v0.465.0, while container bootstrap downloads the surface/default v0.463.0. Please trace where the explicit --ward-version is lost between broker forwarding, Docker env assembly, and containerassets/entrypoint.sh reading WARD_VERSION.

Reopened after Kai pointed out the stronger repro: the explicit version override appears to be ignored or overwritten. Observed dispatch from this surface: ```text warded coilyco-flight-deck/ward#700 --harness codex --force --ward-version v0.465.0 ward dispatch broker: forwarded `ward agent engineer coilyco-flight-deck/ward#700 --harness codex --ward-version v0.465.0 --force` ... ``` The reservation comment also records: ```text - **Run:** `engineer-codex-ward-700` · ward `v0.465.0` · dispatched `2026-07-08T21:18:05Z` ``` But Kai reports the engineer container log still shows: ```text ward-container: downloading ward v0.463.0 for linux-arm64 ``` So this is not just accidental inheritance from an unpinned director surface. The dispatch argv and reservation say `v0.465.0`, while container bootstrap downloads the surface/default `v0.463.0`. Please trace where the explicit `--ward-version` is lost between broker forwarding, Docker env assembly, and `containerassets/entrypoint.sh` reading `WARD_VERSION`.
Owner

this may be an ops issue rather than a code issue - dont chase ghosts if it doesnt seem like theres a real bug here

this may be an ops issue rather than a code issue - dont chase ghosts if it doesnt seem like theres a real bug here
Author
Member

Reproduced again during the 2026-07-09 recovery wave, so this is no longer just a possible ops ghost.

Current facts:

  • latest ward release is v0.480.0 (published 2026-07-09T06:50:56Z).
  • this read-only director surface is still running ward v0.474.0.
  • fresh dispatch for ward#735 at 2026-07-09T07:27:21Z recorded ward v0.474.0 in the reservation comment.
  • the final worker comment for ward#735 says workflow: pr; review summary: skipped ... and treats PR open as done.
  • PR #773 was opened from that run, not merged, and CI failed after the worker already posted WARD-OUTCOME: done.

That means the child engineer did not receive the latest v0.480.0 behavior from the repaired lane. The immediate cause is still the version propagation described in this issue: unpinned surface dispatches inherit the stale surface ward version instead of resolving current/latest for the child.

This also intersects with ward#706: plain workflow pr still means branch + PR, not autonomous merge. But the stale-version propagation is independently real because the child run itself was pinned to v0.474.0 after v0.480.0 was published.

Next recovery dispatch should run this issue with an explicit --ward-version v0.480.0 so the fix attempt itself does not reproduce the stale launcher problem.

— Codex, via ward agent

Reproduced again during the 2026-07-09 recovery wave, so this is no longer just a possible ops ghost. Current facts: * latest ward release is `v0.480.0` (published 2026-07-09T06:50:56Z). * this read-only director surface is still running `ward v0.474.0`. * fresh dispatch for `ward#735` at 2026-07-09T07:27:21Z recorded `ward v0.474.0` in the reservation comment. * the final worker comment for `ward#735` says `workflow: pr; review summary: skipped ...` and treats PR open as done. * PR `#773` was opened from that run, not merged, and CI failed after the worker already posted `WARD-OUTCOME: done`. That means the child engineer did not receive the latest `v0.480.0` behavior from the repaired lane. The immediate cause is still the version propagation described in this issue: unpinned surface dispatches inherit the stale surface ward version instead of resolving current/latest for the child. This also intersects with `ward#706`: plain `workflow pr` still means branch + PR, not autonomous merge. But the stale-version propagation is independently real because the child run itself was pinned to `v0.474.0` after `v0.480.0` was published. Next recovery dispatch should run this issue with an explicit `--ward-version v0.480.0` so the fix attempt itself does not reproduce the stale launcher problem. <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

🔒 Reserved by ward agent --harness codex — container engineer-codex-ward-700 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-09T07:38:32Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/ward#700 · branch issue-700 · harness codex · workflow pr
  • Run: engineer-codex-ward-700 · ward v0.480.0 · dispatched 2026-07-09T07:38:32Z
  • Comment thread: 4 included in the pre-flight read, 1 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.480.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --harness codex` — container `engineer-codex-ward-700` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-09T07:38:32Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#700` · branch `issue-700` · harness `codex` · workflow `pr` - **Run:** `engineer-codex-ward-700` · ward `v0.480.0` · dispatched `2026-07-09T07:38:32Z` - **Comment thread:** 4 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-08T21:20:22Z), @coilyco-ops (2026-07-08T21:22:21Z), @coilysiren (2026-07-08T21:24:51Z), @coilyco-ops (2026-07-09T07:38:13Z) - stripped: @coilyco-ops (2026-07-08T21:18:06Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.480.0). </details> <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

WARD-OUTCOME: blocked - review summary: blocked: codex: exit status 1: Reading additional input from stdin... OpenAI Codex v0.142.5 -------- workdir: /workspace/ward mod...

Review panel verdicts:

  • codex: ERROR: codex: exit status 1: Reading additional input from stdin... OpenAI Codex v0.142.5 -------- workdir: /workspace/ward model: gpt-5.5 provider: openai approval: never sandbox: read-only reasoning... (conf 0.00)
  • claude: ERROR: claude: exit status 1: {"type":"system","subtype":"hook_started","hook_id":"e8c9a974-7c51-4841-bb22-4616fdff56ba","hook_name":"SessionStart:startup","hook_event":"SessionStart","uuid":"c2b89316... (conf 0.00)
  • goose: ERROR: goose: exit status 1: error: No provider configured. Run 'goose configure' first. (conf 0.00)

— Codex, via ward agent

WARD-OUTCOME: blocked - review summary: blocked: codex: exit status 1: Reading additional input from stdin... OpenAI Codex v0.142.5 -------- workdir: /workspace/ward mod... Review panel verdicts: - codex: ERROR: codex: exit status 1: Reading additional input from stdin... OpenAI Codex v0.142.5 -------- workdir: /workspace/ward model: gpt-5.5 provider: openai approval: never sandbox: read-only reasoning... (conf 0.00) - claude: ERROR: claude: exit status 1: {"type":"system","subtype":"hook_started","hook_id":"e8c9a974-7c51-4841-bb22-4616fdff56ba","hook_name":"SessionStart:startup","hook_event":"SessionStart","uuid":"c2b89316... (conf 0.00) - goose: ERROR: goose: exit status 1: error: No provider configured. Run 'goose configure' first. (conf 0.00) <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

WARD-OUTCOME: blocked - review gate could not run
workflow: pr; review summary: blocked: codex: exit status 1: Reading additional input from stdin... OpenAI Codex v0.142.5 -------- workdir: /workspace/ward mod...

Review gate blocked. Codex failed to authenticate, Claude failed to run inside the harness, and Goose had no provider configured, so the panel had 0/2 passing. The implementation stayed green under ward exec test, but the review gate did not produce a runnable pass.

WARD-OUTCOME: blocked - review gate could not run workflow: pr; review summary: blocked: codex: exit status 1: Reading additional input from stdin... OpenAI Codex v0.142.5 -------- workdir: /workspace/ward mod... Review gate blocked. Codex failed to authenticate, Claude failed to run inside the harness, and Goose had no provider configured, so the panel had 0/2 passing. The implementation stayed green under `ward exec test`, but the review gate did not produce a runnable pass.
Author
Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-700 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T18:01:51Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/ward#700 · branch issue-700 · harness codex · workflow pull-request-and-merge
  • Run: engineer-codex-ward-700 · ward v0.493.0 · dispatched 2026-07-09T18:01:51Z
  • Comment thread: 6 included in the pre-flight read, 2 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.493.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-700` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T18:01:51Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#700` · branch `issue-700` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-700` · ward `v0.493.0` · dispatched `2026-07-09T18:01:51Z` - **Comment thread:** 6 included in the pre-flight read, 2 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-08T21:20:22Z), @coilyco-ops (2026-07-08T21:22:21Z), @coilysiren (2026-07-08T21:24:51Z), @coilyco-ops (2026-07-09T07:38:13Z), @coilyco-ops (2026-07-09T07:50:00Z), @coilyco-ops (2026-07-09T07:50:49Z) - stripped: @coilyco-ops (2026-07-08T21:18:06Z), @coilyco-ops (2026-07-09T07:38:33Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.493.0). </details> </details> <!-- ward-agent-signature --> — Codex, via `ward agent`
Author
Member

WARD-OUTCOME: done

details

workflow: pull-request-and-merge; review summary: skipped (review gate skipped by --skip-review / --no-review-gate)
retrospective: the fix was localized, but golangci-lint forced a second pass to split version-source resolution out of buildUpPlan.
confidence: high
surprises: the first pass was functionally correct but missed a gocyclo limit.
follow-ups: none.

WARD-OUTCOME: done ✅ <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped (review gate skipped by --skip-review / --no-review-gate) retrospective: the fix was localized, but golangci-lint forced a second pass to split version-source resolution out of buildUpPlan. confidence: high surprises: the first pass was functionally correct but missed a gocyclo limit. follow-ups: none. </details>
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/ward#700
No description provided.