coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

Make the Ward director evaluation executable, not a blueprint #523

New issue
Closed
opened 2026-07-10 09:09:19 +00:00 by coilyco-ops · 5 comments
coilyco-ops commented 2026-07-10 09:09:19 +00:00
Member
Copy link

Problem

The first Goose slice for infrastructure#520 started creating a Kubernetes job manifest, docs, and a validation script, but the observed run log describes the result as an infrastructure blueprint that will be ready "when the time comes". That is not enough for infrastructure#520. The target is a repeatable command that launches a fresh k3s Job and produces an independently verified PASS or a concrete FAIL evidence bundle.

This issue exists because the active infrastructure#520 engineer will not re-read comments while reserved. A correction needs a fresh run.

Required correction

Turn the evaluation from a placeholder/design artifact into an executable harness. The implementation must not claim success by documenting intent.

The harness must:

  • Create a real evaluation issue in coilysiren/inbox with a unique ward-eval:<run-id> title, P4, and headless.
  • Launch a disposable k3s Job that starts from no Ward install or Ward cache.
  • Run an outer Goose one-shot that installs Ward through the documented Homebrew path, runs setup, and launches a Ward director for the exact evaluation issue.
  • Use Goose/Ollama for every model-backed boundary: bootstrap operator, director decision, and engineer.
  • Prove the director makes a real Goose-backed DISPATCH decision for only that issue, not a ranked fallback or another backlog item.
  • Let the director-dispatched Goose engineer create only runs/<run-id>.txt in coilysiren/inbox, write exactly one RFC3339 UTC timestamp plus newline, commit it, land it, post WARD-OUTCOME: done, and close the issue only after the commit is on default branch.
  • Collect logs, Docker events, Ward audit records, the Goose transcript, and Forgejo issue/commit evidence.
  • Run an independent verifier that checks every infrastructure#520 verification bullet, including no Claude/Codex invocation and no Forgejo token leakage in collected logs.
  • Fail closed with preserved evidence and cleanup comments if the Job times out or dies.

Acceptance

  • A single committed command or script launches the fresh k3s Job.
  • Running that command either produces a verified PASS for a real coilysiren/inbox run, or a precise FAIL with preserved evidence and no false success.
  • The verifier rejects documentation-only, dry-run-only, blueprint-only, or "ready when implemented" outputs.
  • The implementation does not require a human to manually create the evaluation issue, manually inspect logs, or manually decide whether it passed.
  • ward exec test or the repo-appropriate validation passes, or the issue comment names the exact failing check.

Related: coilyco-flight-deck/infrastructure#520 and coilyco-flight-deck/ward#980.

## Problem The first Goose slice for infrastructure#520 started creating a Kubernetes job manifest, docs, and a validation script, but the observed run log describes the result as an infrastructure blueprint that will be ready "when the time comes". That is not enough for infrastructure#520. The target is a repeatable command that launches a fresh k3s Job and produces an independently verified PASS or a concrete FAIL evidence bundle. This issue exists because the active infrastructure#520 engineer will not re-read comments while reserved. A correction needs a fresh run. ## Required correction Turn the evaluation from a placeholder/design artifact into an executable harness. The implementation must not claim success by documenting intent. The harness must: * Create a real evaluation issue in `coilysiren/inbox` with a unique `ward-eval:<run-id>` title, `P4`, and `headless`. * Launch a disposable k3s Job that starts from no Ward install or Ward cache. * Run an outer Goose one-shot that installs Ward through the documented Homebrew path, runs setup, and launches a Ward director for the exact evaluation issue. * Use Goose/Ollama for every model-backed boundary: bootstrap operator, director decision, and engineer. * Prove the director makes a real Goose-backed `DISPATCH` decision for only that issue, not a ranked fallback or another backlog item. * Let the director-dispatched Goose engineer create only `runs/<run-id>.txt` in `coilysiren/inbox`, write exactly one RFC3339 UTC timestamp plus newline, commit it, land it, post `WARD-OUTCOME: done`, and close the issue only after the commit is on default branch. * Collect logs, Docker events, Ward audit records, the Goose transcript, and Forgejo issue/commit evidence. * Run an independent verifier that checks every infrastructure#520 verification bullet, including no Claude/Codex invocation and no Forgejo token leakage in collected logs. * Fail closed with preserved evidence and cleanup comments if the Job times out or dies. ## Acceptance * A single committed command or script launches the fresh k3s Job. * Running that command either produces a verified PASS for a real `coilysiren/inbox` run, or a precise FAIL with preserved evidence and no false success. * The verifier rejects documentation-only, dry-run-only, blueprint-only, or "ready when implemented" outputs. * The implementation does not require a human to manually create the evaluation issue, manually inspect logs, or manually decide whether it passed. * `ward exec test` or the repo-appropriate validation passes, or the issue comment names the exact failing check. Related: coilyco-flight-deck/infrastructure#520 and coilyco-flight-deck/ward#980.
coilyco-ops commented 2026-07-10 09:09:30 +00:00
Author
Member
Copy link

WARD-DISPATCH: deferred ⏸

deferred details

This forwarded dispatch was deferred after the issue was already reserved.

Attempted harness: goose
Attempted run: ward agent engineer coilyco-flight-deck/infrastructure#523 --harness goose --ward-version v0.573.0
Container: engineer-goose-infrastructure-523
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T090927Z-director-codex-vg55-coilyco-flight-deck-infrastructure-523.log
Capacity: ward agent engineer --harness goose: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run ward agent reap for stale engineers

Retry: the issue stays queued and the director will try again when a slot opens.

— Goose, via ward agent

<!-- ward-agent-reservation-released --> <!-- ward-needs-redispatch --> WARD-DISPATCH: deferred ⏸ <details><summary>deferred details</summary> This forwarded dispatch was deferred after the issue was already reserved. Attempted harness: `goose` Attempted run: `ward agent engineer coilyco-flight-deck/infrastructure#523 --harness goose --ward-version v0.573.0` Container: `engineer-goose-infrastructure-523` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T090927Z-director-codex-vg55-coilyco-flight-deck-infrastructure-523.log` Capacity: `ward agent engineer --harness goose: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run `ward agent reap` for stale engineers` Retry: the issue stays queued and the director will try again when a slot opens. </details> <!-- ward-agent-signature --> — Goose, via `ward agent`
coilyco-ops commented 2026-07-10 09:10:41 +00:00
Author
Member
Copy link

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-goose-infrastructure-523 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness goose (reserved 2026-07-10T09:10:40Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/infrastructure#523 · branch issue-523 · harness goose · workflow direct-main
  • Run: engineer-goose-infrastructure-523 · ward v0.573.0 · dispatched 2026-07-10T09:10:40Z
  • Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0).

— Goose, via ward agent

<!-- ward-agent-reservation --> WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-goose-infrastructure-523` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness goose` (reserved 2026-07-10T09:10:40Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#523` · branch `issue-523` · harness `goose` · workflow `direct-main` - **Run:** `engineer-goose-infrastructure-523` · ward `v0.573.0` · dispatched `2026-07-10T09:10:40Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T09:09:30Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0). </details> </details> <!-- ward-agent-signature --> — Goose, via `ward agent`
coilyco-ops commented 2026-07-10 09:29:07 +00:00
Author
Member
Copy link

WARD-REAP: reopened 🛑

salvage details

An ephemeral ward container (goose mode) dispatched for this issue finished but its work was not merged to main, so cleanup preserved it on a branch before teardown and reopened the issue (a closing reference for #523 never reached main). Recover from the salvage branch below.

  • Repo: coilyco-flight-deck/infrastructure
  • Salvage branch: ward-salvage/infrastructure-b6103c7a
  • Pull request: not opened - PR creation failed: forgejo: parse created pull request: unexpected end of JSON input
  • Reason: missing same-repo closing reference
  • Container uptime at reap: 18m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token)

Cleanup diagnostics

--- reap diagnostics ---
ward version:      v0.573.0
version source:    pinned via WARD_VERSION/--ward-version (v0.573.0)
HEAD:              a0060d9d88a1
origin/main:       6a8b1787722a
ancestry:          HEAD is NOT yet on origin/main - residual work remains to land
decision gate:     missing same-repo closing reference
reason:            missing same-repo closing reference
provenance:        present
run-owned landed:  no
working tree:      5 dirty path(s)
container uptime:  18m (baked Forgejo PAT age proxy)
--- end reap diagnostics ---

Recover

git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure.git ward-salvage/infrastructure-b6103c7a
git checkout -b ward-salvage/infrastructure-b6103c7a FETCH_HEAD

This salvage was blocked by a missing closing reference. To recover, amend or cherry-pick the salvaged work so the landing commit message includes closes #523, or add a small empty trailer commit with closes #523, then land the branch.

Working tree at reap time

 M Makefile
 M README.md
?? scripts/launch-ward-eval.sh
?? scripts/test-ward-eval.sh
?? scripts/ward-eval-executable.sh

— Goose, via ward agent

WARD-REAP: reopened 🛑 <details><summary>salvage details</summary> An ephemeral `ward container` (goose mode) dispatched for this issue finished but its work was **not merged to `main`**, so cleanup preserved it on a branch before teardown and reopened the issue (a closing reference for #523 never reached `main`). Recover from the salvage branch below. - **Repo:** `coilyco-flight-deck/infrastructure` - **Salvage branch:** `ward-salvage/infrastructure-b6103c7a` - **Pull request:** not opened - PR creation failed: forgejo: parse created pull request: unexpected end of JSON input - **Reason:** missing same-repo closing reference - **Container uptime at reap:** 18m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token) ## Cleanup diagnostics ``` --- reap diagnostics --- ward version: v0.573.0 version source: pinned via WARD_VERSION/--ward-version (v0.573.0) HEAD: a0060d9d88a1 origin/main: 6a8b1787722a ancestry: HEAD is NOT yet on origin/main - residual work remains to land decision gate: missing same-repo closing reference reason: missing same-repo closing reference provenance: present run-owned landed: no working tree: 5 dirty path(s) container uptime: 18m (baked Forgejo PAT age proxy) --- end reap diagnostics --- ``` ## Recover ```bash git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure.git ward-salvage/infrastructure-b6103c7a git checkout -b ward-salvage/infrastructure-b6103c7a FETCH_HEAD ``` This salvage was blocked by a missing closing reference. To recover, amend or cherry-pick the salvaged work so the landing commit message includes `closes #523`, or add a small empty trailer commit with `closes #523`, then land the branch. ## Working tree at reap time ``` M Makefile M README.md ?? scripts/launch-ward-eval.sh ?? scripts/test-ward-eval.sh ?? scripts/ward-eval-executable.sh ``` </details> <!-- ward-agent-signature --> — Goose, via `ward agent`
coilyco-ops commented 2026-07-10 10:45:45 +00:00
Author
Member
Copy link

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-523 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:45:44Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/infrastructure#523 · branch issue-523 · harness codex · workflow direct-main
  • Run: engineer-codex-infrastructure-523 · ward v0.580.0 · dispatched 2026-07-10T10:45:44Z
  • Comment thread: 2 included in the pre-flight read, 1 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-523` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:45:44Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#523` · branch `issue-523` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-523` · ward `v0.580.0` · dispatched `2026-07-10T10:45:44Z` - **Comment thread:** 2 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T09:09:30Z), @coilyco-ops (2026-07-10T09:29:07Z) - stripped: @coilyco-ops (2026-07-10T09:10:41Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details> <!-- ward-agent-signature --> — Codex, via `ward agent`
coilyco-ops commented 2026-07-10 10:53:08 +00:00
Author
Member
Copy link

WARD-OUTCOME: done

details workflow: direct-main; review summary: review gate skipped by ~/.ward/config.yaml default felt: steady salvage integration with remote tip churn and one doc conflict confidence: high surprises: origin/main advanced twice during the run follow-ups: none
WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-main; review summary: review gate skipped by ~/.ward/config.yaml default felt: steady salvage integration with remote tip churn and one doc conflict confidence: high surprises: origin/main advanced twice during the run follow-ups: none </details>
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#523
No description provided.