Close remaining ward-eval gaps after infrastructure#534 #535

New issue

Closed

opened 2026-07-10 10:25:11 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-10 10:25:11 +00:00

Member

Problem

infrastructure#534 landed at d5596ac, but the committed evaluator still cannot prove the full infrastructure#520 chain. It is better than the previous echo-only harness, but it can still pass without several required #520 facts.

Evidence from current main after #534:

scripts/ward-eval-executable.sh still writes "ward_outcome": "WARD-OUTCOME: done" into evidence.json and prints WARD-OUTCOME: done itself. The verifier checks Forgejo issue closure separately, but the Ward outcome evidence is still generated by the Job script rather than collected from the dispatched engineer issue thread or Ward audit.
The default timestamp file path in scripts/ward_eval.py is ward-eval-timestamp.txt, while #520 requires the engineer to create only runs/<run-id>.txt.
The Job entrypoint defaults to goose bootstrap, ward install --cold, goose director <issue-url>, and ward dispatch ci <issue-ref>. It does not use the required issue-scoped director invocation:

warded director coilysiren/inbox#123 \
  --harness goose \
  --engineer-harness goose \
  --max-parallel 1 \
  --no-triage \
  --poll-interval 5s

collect_forgejo_evidence inspects the latest default-branch commit, but does not prove the commit happened after the evaluation issue was created.
The verifier records touched files from Forgejo, but does not require the touched file set to be exactly runs/<run-id>.txt.
The verifier does not inspect logs/audit for the absence of Claude/Codex invocations, except indirectly through placeholder/ranked-fallback checks.
The tests still assert string presence for ward dispatch ci and WARD-OUTCOME: done in the entrypoint, which lets the hard-coded outcome path stay green.

This means ward exec ward-eval could still report a PASS with evidence that falls short of the #520 independent-verifier requirements.

Do

Replace the hard-coded Ward outcome in the Job evidence with collected issue-thread or Ward audit evidence from the actual dispatched Goose engineer.
Make the evaluator create and verify exactly runs/<run-id>.txt, not ward-eval-timestamp.txt.
Make the Job launch the exact issue-scoped Goose director path, with --harness goose, --engineer-harness goose, --max-parallel 1, --no-triage, and --poll-interval 5s.
Verify the default-branch commit happened after the evaluation issue was created.
Verify the commit changes only runs/<run-id>.txt and the file contains exactly one RFC3339 UTC timestamp plus a newline.
Inspect collected logs/audit for no Claude/Codex invocation, not just no placeholder evidence.
Update tests so a hard-coded WARD-OUTCOME: done, ward dispatch ci, wrong timestamp path, wrong touched files, stale commit, or Claude/Codex evidence all fail.

Acceptance

Grepping the success path no longer finds a Job-script-assigned "ward_outcome": "WARD-OUTCOME: done" or an entrypoint echo that fabricates the engineer outcome.
DEFAULT_TIMESTAMP_PATH and generated evaluation issue instructions use runs/<run-id>.txt.
The rendered Job/director command contains the exact #520 issue-scoped warded director shape and rejects the old ward dispatch ci shortcut in tests.
Unit tests prove PASS fails when the latest commit predates issue creation, touches any file outside runs/<run-id>.txt, lacks Goose attribution, contains Claude/Codex invocation evidence, or supplies a fabricated Ward outcome.
The launch command still creates a unique ward-eval:<run-id> issue in coilysiren/inbox and applies a disposable k3s Job.
The implementation still does not claim full infrastructure#520 completion until a real run has produced independent verifier PASS.

## Problem `infrastructure#534` landed at `d5596ac`, but the committed evaluator still cannot prove the full `infrastructure#520` chain. It is better than the previous echo-only harness, but it can still pass without several required #520 facts. Evidence from current `main` after #534: * `scripts/ward-eval-executable.sh` still writes `"ward_outcome": "WARD-OUTCOME: done"` into `evidence.json` and prints `WARD-OUTCOME: done` itself. The verifier checks Forgejo issue closure separately, but the Ward outcome evidence is still generated by the Job script rather than collected from the dispatched engineer issue thread or Ward audit. * The default timestamp file path in `scripts/ward_eval.py` is `ward-eval-timestamp.txt`, while #520 requires the engineer to create only `runs/<run-id>.txt`. * The Job entrypoint defaults to `goose bootstrap`, `ward install --cold`, `goose director <issue-url>`, and `ward dispatch ci <issue-ref>`. It does not use the required issue-scoped director invocation: ```bash warded director coilysiren/inbox#123 \ --harness goose \ --engineer-harness goose \ --max-parallel 1 \ --no-triage \ --poll-interval 5s ``` * `collect_forgejo_evidence` inspects the latest default-branch commit, but does not prove the commit happened after the evaluation issue was created. * The verifier records touched files from Forgejo, but does not require the touched file set to be exactly `runs/<run-id>.txt`. * The verifier does not inspect logs/audit for the absence of Claude/Codex invocations, except indirectly through placeholder/ranked-fallback checks. * The tests still assert string presence for `ward dispatch ci` and `WARD-OUTCOME: done` in the entrypoint, which lets the hard-coded outcome path stay green. This means `ward exec ward-eval` could still report a PASS with evidence that falls short of the #520 independent-verifier requirements. ## Do * Replace the hard-coded Ward outcome in the Job evidence with collected issue-thread or Ward audit evidence from the actual dispatched Goose engineer. * Make the evaluator create and verify exactly `runs/<run-id>.txt`, not `ward-eval-timestamp.txt`. * Make the Job launch the exact issue-scoped Goose director path, with `--harness goose`, `--engineer-harness goose`, `--max-parallel 1`, `--no-triage`, and `--poll-interval 5s`. * Verify the default-branch commit happened after the evaluation issue was created. * Verify the commit changes only `runs/<run-id>.txt` and the file contains exactly one RFC3339 UTC timestamp plus a newline. * Inspect collected logs/audit for no Claude/Codex invocation, not just no placeholder evidence. * Update tests so a hard-coded `WARD-OUTCOME: done`, `ward dispatch ci`, wrong timestamp path, wrong touched files, stale commit, or Claude/Codex evidence all fail. ## Acceptance * Grepping the success path no longer finds a Job-script-assigned `"ward_outcome": "WARD-OUTCOME: done"` or an entrypoint echo that fabricates the engineer outcome. * `DEFAULT_TIMESTAMP_PATH` and generated evaluation issue instructions use `runs/<run-id>.txt`. * The rendered Job/director command contains the exact #520 issue-scoped `warded director` shape and rejects the old `ward dispatch ci` shortcut in tests. * Unit tests prove PASS fails when the latest commit predates issue creation, touches any file outside `runs/<run-id>.txt`, lacks Goose attribution, contains Claude/Codex invocation evidence, or supplies a fabricated Ward outcome. * The launch command still creates a unique `ward-eval:<run-id>` issue in `coilysiren/inbox` and applies a disposable k3s Job. * The implementation still does not claim full `infrastructure#520` completion until a real run has produced independent verifier PASS. Related: coilyco-flight-deck/infrastructure#520 and coilyco-flight-deck/infrastructure#534.

coilyco-ops added the

headless

labels

2026-07-10 10:25:11 +00:00

coilyco-ops commented

2026-07-10 10:25:31 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-535 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:25:30Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#535 · branch issue-535 · harness codex · workflow direct-main
Run: engineer-codex-infrastructure-535 · ward v0.573.0 · dispatched 2026-07-10T10:25:30Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-535` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:25:30Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#535` · branch `issue-535` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-535` · ward `v0.573.0` · dispatched `2026-07-10T10:25:30Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 10:38:27 +00:00

fix: tighten ward-eval evidence chain

coilyco-ops closed this issue

2026-07-10 10:38:27 +00:00

coilyco-ops commented

2026-07-10 10:41:00 +00:00

Author

Member

WARD-OUTCOME: done

details

workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default
felt: tighter than the previous harness. the outcome now comes from real logs and issue evidence
confidence: high
surprises: none
follow-ups: none

WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default felt: tighter than the previous harness. the outcome now comes from real logs and issue evidence confidence: high surprises: none follow-ups: none </details>