Make ward-eval execute the #520 chain instead of echoing placeholder evidence #534

New issue

Closed

opened 2026-07-10 10:04:10 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-10 10:04:10 +00:00

Member

Problem

infrastructure#529 closed with WARD-OUTCOME: done, but current main still does not prove the infrastructure#520 chain.

Evidence from current coilyco-flight-deck/infrastructure main after #529 closed:

scripts/ward-eval-executable.sh writes evidence.json values like "goose_director_call": "pending", "dispatch_decision": "pending", "engineer_container": "pending", "docker_events": "pending", "ward_audit": "pending", "commit_content": "pending", and "attribution": "pending".
The same script sets "issue_closed": true and prints WARD-OUTCOME: done itself.
The script only echoes Goose bootstrap, install Ward cold, run issue-scoped Goose director, and let Ward dispatch the Goose engineer; it does not actually run those actions.
scripts/ward_eval.py verify consumes environment variables, not collected Job/Forgejo/Docker/Ward audit evidence.
scripts/ward_director_evaluation_test.py mostly checks that strings like Goose bootstrap and WARD-OUTCOME: done appear, which allows the echo-only implementation to pass.

This is another false-pass risk for infrastructure#520. The harness now has the right subject names, but it still accepts fabricated or placeholder evidence.

Do

Replace placeholder evidence in the Job entrypoint with real collection from the disposable Job, Docker, Ward logs/audit, and Forgejo.
The Job must actually run the outer Goose bootstrap, install Ward cold, launch the issue-scoped Goose director, and let Ward dispatch the Goose engineer.
The verifier must independently query Forgejo for the evaluation issue state, default-branch commit, touched files, timestamp file content, attribution, and touched issue set.
The verifier must inspect collected Job logs/audit/Docker events for Goose director call, parsed DISPATCH decision, no ranked fallback, and engineer-goose-* container existence.
A PASS must be impossible when evidence fields are pending, hard-coded, echoed by the entrypoint, supplied only via environment variables, or otherwise not backed by collected external evidence.
Keep the rejection of generic smoke tests from #529, but make the positive path real.

Acceptance

Grepping the committed code finds no "pending" evidence defaults in the verifier success path.
scripts/ward-eval-executable.sh invokes the real tools needed for the chain, rather than only echoing the required step names.
A unit or fixture test proves placeholder/hard-coded evidence fails.
A unit or fixture test proves a PASS requires evidence from Job logs, Docker events, Ward audit/logs, and Forgejo API or git state.
The launch command still creates a unique ward-eval:<run-id> issue in coilysiren/inbox and applies a disposable k3s Job.
The implementation does not claim full infrastructure#520 completion until a real run has produced the independent verifier PASS.

## Problem `infrastructure#529` closed with `WARD-OUTCOME: done`, but current `main` still does not prove the infrastructure#520 chain. Evidence from current `coilyco-flight-deck/infrastructure` main after #529 closed: * `scripts/ward-eval-executable.sh` writes `evidence.json` values like `"goose_director_call": "pending"`, `"dispatch_decision": "pending"`, `"engineer_container": "pending"`, `"docker_events": "pending"`, `"ward_audit": "pending"`, `"commit_content": "pending"`, and `"attribution": "pending"`. * The same script sets `"issue_closed": true` and prints `WARD-OUTCOME: done` itself. * The script only echoes `Goose bootstrap`, `install Ward cold`, `run issue-scoped Goose director`, and `let Ward dispatch the Goose engineer`; it does not actually run those actions. * `scripts/ward_eval.py verify` consumes environment variables, not collected Job/Forgejo/Docker/Ward audit evidence. * `scripts/ward_director_evaluation_test.py` mostly checks that strings like `Goose bootstrap` and `WARD-OUTCOME: done` appear, which allows the echo-only implementation to pass. This is another false-pass risk for infrastructure#520. The harness now has the right subject names, but it still accepts fabricated or placeholder evidence. ## Do * Replace placeholder evidence in the Job entrypoint with real collection from the disposable Job, Docker, Ward logs/audit, and Forgejo. * The Job must actually run the outer Goose bootstrap, install Ward cold, launch the issue-scoped Goose director, and let Ward dispatch the Goose engineer. * The verifier must independently query Forgejo for the evaluation issue state, default-branch commit, touched files, timestamp file content, attribution, and touched issue set. * The verifier must inspect collected Job logs/audit/Docker events for Goose director call, parsed DISPATCH decision, no ranked fallback, and `engineer-goose-*` container existence. * A PASS must be impossible when evidence fields are `pending`, hard-coded, echoed by the entrypoint, supplied only via environment variables, or otherwise not backed by collected external evidence. * Keep the rejection of generic smoke tests from #529, but make the positive path real. ## Acceptance * Grepping the committed code finds no `"pending"` evidence defaults in the verifier success path. * `scripts/ward-eval-executable.sh` invokes the real tools needed for the chain, rather than only echoing the required step names. * A unit or fixture test proves placeholder/hard-coded evidence fails. * A unit or fixture test proves a PASS requires evidence from Job logs, Docker events, Ward audit/logs, and Forgejo API or git state. * The launch command still creates a unique `ward-eval:<run-id>` issue in `coilysiren/inbox` and applies a disposable k3s Job. * The implementation does not claim full infrastructure#520 completion until a real run has produced the independent verifier PASS. Related: coilyco-flight-deck/infrastructure#520 and coilyco-flight-deck/infrastructure#529.

coilyco-ops added the

headless

labels

2026-07-10 10:04:10 +00:00

coilyco-ops commented

2026-07-10 10:04:19 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-534 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:04:18Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#534 · branch issue-534 · harness codex · workflow direct-main
Run: engineer-codex-infrastructure-534 · ward v0.573.0 · dispatched 2026-07-10T10:04:18Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-534` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:04:18Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#534` · branch `issue-534` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-534` · ward `v0.573.0` · dispatched `2026-07-10T10:04:18Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 10:21:03 +00:00

feat(ward): replace ward-eval placeholders with real evidence

coilyco-ops closed this issue

2026-07-10 10:21:03 +00:00

coilyco-ops commented

2026-07-10 10:22:27 +00:00

Author

Member

WARD-OUTCOME: done

details

workflow: direct-main; review summary: skipped by ~/.ward/config.yaml default

Retrospective: the implementation felt like replacing a false-pass seam with real evidence plumbing.

Confidence: high.

Surprises: origin/main had advanced, so I merged it before pushing.

Follow-ups: none.

WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-main; review summary: skipped by ~/.ward/config.yaml default Retrospective: the implementation felt like replacing a false-pass seam with real evidence plumbing. Confidence: high. Surprises: origin/main had advanced, so I merged it before pushing. Follow-ups: none. </details>

coilyco-ops referenced this issue

2026-07-10 10:25:11 +00:00

Close remaining ward-eval gaps after infrastructure#534 #535