coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

Reject simulated Ward director evaluation runs #524

New issue
Closed
opened 2026-07-10 09:17:11 +00:00 by coilyco-ops · 5 comments
coilyco-ops commented 2026-07-10 09:17:11 +00:00
Member
Copy link

Problem

The active infrastructure#523 Goose run is building a launch path that says it would normally run kubectl apply, but is "simulating here" and then sleeps before running a local verifier. That still does not satisfy infrastructure#520.

The acceptance loop must prove a real k3s disposable Job, a real cold Ward install, a real Goose director decision, a real Goose engineer container, and a real commit in coilysiren/inbox. A simulated launcher can make the harness look executable while bypassing the exact boundary the issue exists to test.

This issue exists because infrastructure#523 is reserved and will not re-read comments.

Do

  • Remove any simulated kubectl, simulated Job completion, simulated verifier pass, or placeholder sleep path from the Ward director evaluation harness.
  • Make the committed launch command apply a real k3s Job manifest and wait on the real Job status.
  • Collect logs, Docker events, Ward audit records, the Goose transcript, and Forgejo issue/commit evidence from the real Job pod.
  • Make the verifier fail closed if the Job was not created, did not run, did not emit the expected evidence, or only exercised local shell stubs.
  • Preserve evidence on failure and post a cleanup comment to the generated coilysiren/inbox evaluation issue if one was created.

Acceptance

  • Grepping the committed evaluation harness finds no simulated kubectl, fake Job completion, placeholder sleep-based success, or "would normally" launch path.
  • Running the command creates a real Kubernetes Job in k3s and waits for its completion or timeout.
  • A PASS is possible only after the independent verifier observes the full infrastructure#520 chain in real evidence.
  • A missing cluster, missing kubectl, missing Job, or failed Job produces a precise FAIL with preserved evidence, not a successful dry run.
  • ward exec test or the repo-appropriate validation passes, or the issue comment names the exact failing check.

Related: coilyco-flight-deck/infrastructure#520 and coilyco-flight-deck/infrastructure#523.

## Problem The active infrastructure#523 Goose run is building a launch path that says it would normally run `kubectl apply`, but is "simulating here" and then sleeps before running a local verifier. That still does not satisfy infrastructure#520. The acceptance loop must prove a real k3s disposable Job, a real cold Ward install, a real Goose director decision, a real Goose engineer container, and a real commit in `coilysiren/inbox`. A simulated launcher can make the harness look executable while bypassing the exact boundary the issue exists to test. This issue exists because infrastructure#523 is reserved and will not re-read comments. ## Do * Remove any simulated `kubectl`, simulated Job completion, simulated verifier pass, or placeholder sleep path from the Ward director evaluation harness. * Make the committed launch command apply a real k3s Job manifest and wait on the real Job status. * Collect logs, Docker events, Ward audit records, the Goose transcript, and Forgejo issue/commit evidence from the real Job pod. * Make the verifier fail closed if the Job was not created, did not run, did not emit the expected evidence, or only exercised local shell stubs. * Preserve evidence on failure and post a cleanup comment to the generated `coilysiren/inbox` evaluation issue if one was created. ## Acceptance * Grepping the committed evaluation harness finds no simulated `kubectl`, fake Job completion, placeholder sleep-based success, or "would normally" launch path. * Running the command creates a real Kubernetes Job in k3s and waits for its completion or timeout. * A PASS is possible only after the independent verifier observes the full infrastructure#520 chain in real evidence. * A missing cluster, missing `kubectl`, missing Job, or failed Job produces a precise FAIL with preserved evidence, not a successful dry run. * `ward exec test` or the repo-appropriate validation passes, or the issue comment names the exact failing check. Related: coilyco-flight-deck/infrastructure#520 and coilyco-flight-deck/infrastructure#523.
coilyco-ops commented 2026-07-10 09:17:18 +00:00
Author
Member
Copy link

WARD-DISPATCH: deferred ⏸

deferred details

This forwarded dispatch was deferred after the issue was already reserved.

Attempted harness: goose
Attempted run: ward agent engineer coilyco-flight-deck/infrastructure#524 --harness goose --ward-version v0.573.0
Container: engineer-goose-infrastructure-524
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T091716Z-director-codex-vg55-coilyco-flight-deck-infrastructure-524.log
Capacity: ward agent engineer --harness goose: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run ward agent reap for stale engineers

Retry: the issue stays queued and the director will try again when a slot opens.

— Goose, via ward agent

<!-- ward-agent-reservation-released --> <!-- ward-needs-redispatch --> WARD-DISPATCH: deferred ⏸ <details><summary>deferred details</summary> This forwarded dispatch was deferred after the issue was already reserved. Attempted harness: `goose` Attempted run: `ward agent engineer coilyco-flight-deck/infrastructure#524 --harness goose --ward-version v0.573.0` Container: `engineer-goose-infrastructure-524` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T091716Z-director-codex-vg55-coilyco-flight-deck-infrastructure-524.log` Capacity: `ward agent engineer --harness goose: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run `ward agent reap` for stale engineers` Retry: the issue stays queued and the director will try again when a slot opens. </details> <!-- ward-agent-signature --> — Goose, via `ward agent`
coilyco-ops commented 2026-07-10 09:21:35 +00:00
Author
Member
Copy link

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-goose-infrastructure-524 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness goose (reserved 2026-07-10T09:21:34Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/infrastructure#524 · branch issue-524 · harness goose · workflow direct-main
  • Run: engineer-goose-infrastructure-524 · ward v0.573.0 · dispatched 2026-07-10T09:21:34Z
  • Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0).

— Goose, via ward agent

<!-- ward-agent-reservation --> WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-goose-infrastructure-524` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness goose` (reserved 2026-07-10T09:21:34Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#524` · branch `issue-524` · harness `goose` · workflow `direct-main` - **Run:** `engineer-goose-infrastructure-524` · ward `v0.573.0` · dispatched `2026-07-10T09:21:34Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T09:17:18Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0). </details> </details> <!-- ward-agent-signature --> — Goose, via `ward agent`
coilyco-ops commented 2026-07-10 09:43:21 +00:00
Author
Member
Copy link

WARD-REAP: reopened 🛑

salvage details

An ephemeral ward container (goose mode) dispatched for this issue finished but its work was not merged to main, so cleanup preserved it on a branch before teardown and reopened the issue (a closing reference for #524 never reached main). Recover from the salvage branch below.

  • Repo: coilyco-flight-deck/infrastructure
  • Salvage branch: ward-salvage/infrastructure-979430ef
  • Pull request: not opened - PR creation failed: forgejo: parse created pull request: unexpected end of JSON input
  • Reason: missing same-repo closing reference
  • Container uptime at reap: 21m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token)

Cleanup diagnostics

--- reap diagnostics ---
ward version:      v0.573.0
version source:    pinned via WARD_VERSION/--ward-version (v0.573.0)
HEAD:              0f2de5fb5dfa
origin/main:       ba45a0ac6a3f
ancestry:          HEAD is NOT yet on origin/main - residual work remains to land
decision gate:     missing same-repo closing reference
reason:            missing same-repo closing reference
provenance:        present
run-owned landed:  no
working tree:      2 dirty path(s)
container uptime:  21m (baked Forgejo PAT age proxy)
--- end reap diagnostics ---

Recover

git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure.git ward-salvage/infrastructure-979430ef
git checkout -b ward-salvage/infrastructure-979430ef FETCH_HEAD

This salvage was blocked by a missing closing reference. To recover, amend or cherry-pick the salvaged work so the landing commit message includes closes #524, or add a small empty trailer commit with closes #524, then land the branch.

Working tree at reap time

?? docs/ward-director-evaluation-harness.md
?? scripts/ward_director_evaluation_test.py

— Goose, via ward agent

WARD-REAP: reopened 🛑 <details><summary>salvage details</summary> An ephemeral `ward container` (goose mode) dispatched for this issue finished but its work was **not merged to `main`**, so cleanup preserved it on a branch before teardown and reopened the issue (a closing reference for #524 never reached `main`). Recover from the salvage branch below. - **Repo:** `coilyco-flight-deck/infrastructure` - **Salvage branch:** `ward-salvage/infrastructure-979430ef` - **Pull request:** not opened - PR creation failed: forgejo: parse created pull request: unexpected end of JSON input - **Reason:** missing same-repo closing reference - **Container uptime at reap:** 21m (age of the baked Forgejo PAT snapshot; a long-lived container is likelier to carry a rotated token) ## Cleanup diagnostics ``` --- reap diagnostics --- ward version: v0.573.0 version source: pinned via WARD_VERSION/--ward-version (v0.573.0) HEAD: 0f2de5fb5dfa origin/main: ba45a0ac6a3f ancestry: HEAD is NOT yet on origin/main - residual work remains to land decision gate: missing same-repo closing reference reason: missing same-repo closing reference provenance: present run-owned landed: no working tree: 2 dirty path(s) container uptime: 21m (baked Forgejo PAT age proxy) --- end reap diagnostics --- ``` ## Recover ```bash git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure.git ward-salvage/infrastructure-979430ef git checkout -b ward-salvage/infrastructure-979430ef FETCH_HEAD ``` This salvage was blocked by a missing closing reference. To recover, amend or cherry-pick the salvaged work so the landing commit message includes `closes #524`, or add a small empty trailer commit with `closes #524`, then land the branch. ## Working tree at reap time ``` ?? docs/ward-director-evaluation-harness.md ?? scripts/ward_director_evaluation_test.py ``` </details> <!-- ward-agent-signature --> — Goose, via `ward agent`
coilyco-ops commented 2026-07-10 10:45:54 +00:00
Author
Member
Copy link

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-524 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:45:53Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)
  • Resolved: coilyco-flight-deck/infrastructure#524 · branch issue-524 · harness codex · workflow direct-main
  • Run: engineer-codex-infrastructure-524 · ward v0.580.0 · dispatched 2026-07-10T10:45:53Z
  • Comment thread: 2 included in the pre-flight read, 1 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-524` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:45:53Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#524` · branch `issue-524` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-524` · ward `v0.580.0` · dispatched `2026-07-10T10:45:53Z` - **Comment thread:** 2 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T09:17:18Z), @coilyco-ops (2026-07-10T09:43:21Z) - stripped: @coilyco-ops (2026-07-10T09:21:35Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details> <!-- ward-agent-signature --> — Codex, via `ward agent`
coilyco-ops commented 2026-07-10 10:51:09 +00:00
Author
Member
Copy link

WARD-OUTCOME: done

details workflow: direct-main; review summary: skipped because review gate skipped by ~/.ward/config.yaml default felt: the salvage branch was stale smoke-test drift, so I kept the real helper path and added a regression test for simulated launch text. confidence: high surprises: `ward ops forgejo` was unavailable in this container, so the comment went through the Forgejo API directly. follow-ups: none.
WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-main; review summary: skipped because review gate skipped by ~/.ward/config.yaml default felt: the salvage branch was stale smoke-test drift, so I kept the real helper path and added a regression test for simulated launch text. confidence: high surprises: `ward ops forgejo` was unavailable in this container, so the comment went through the Forgejo API directly. follow-ups: none. </details>
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#524
No description provided.