feat(evalkit): project a committed run into a display payload, one way #290

Merged

coilysiren merged 1 commit from aos/claude/xu59-eval-export into main

2026-08-16 05:33:59 +00:00

coilyco-ops commented

2026-08-16 05:32:32 +00:00

Member

The exporter from coilyco-bridge/deploy #572. My seat owns this half; the Phoenix deployment is ops'.

One way, and that is the whole design

Committed records stay canonical. The payload goes out, nothing comes back, and the display surface is a rebuildable projection rather than a second home for evidence.

That shape is what makes this cheap. #213 step 4 — returning label, critique, evidence, reviewer identity, and provenance to the owning records without manual reconstruction — is the hard requirement for a review UI. This is not one. evalkit.annotate stays the grading surface and nothing is authored downstream.

The grader's notes are withheld by default

critique and evidence are free text a human writes for herself, mid-flow, in a keystroke-driven TUI. One committed critique in ops-board-2026-08-12-regraded trails off into a note about the terminal rendering badly.

Nothing unsafe about it. But it was not written for an audience, and the display target is a permanent public recording. So those two fields need --include-private, and the payload records which way it went in includes_private_fields.

Labels, prompts, targets, outputs, and the pair structure carry by default. That is what makes a board legible — the pass/fail shape and the pairing, not the margin notes.

It refuses rather than redacts

A scrubber that misses a pattern ships the secret. This stops instead, names every reason in one pass so a fix is not a loop, and exits non-zero.

Recognized: AWS key ids, bearer and API tokens, JWTs, private key blocks, SSM parameter paths, Discord snowflakes, tailnet hosts, email addresses.

Withheld text is not scanned, because text that never leaves cannot leak, and refusing on it would block an export that is safe. There is a test for exactly that asymmetry.

Two things worth reviewing

A test caught a real bug in the gate. My SSM pattern required the keyword at the start of the path segment, so /sirens-echo/postgres-password — a real parameter in this estate — did not match, because the keyword has a prefix. Fixed, and the parametrized case is what found it.

Verified against real data. Both committed pilot runs export without refusal, so the gate does not cry wolf on legitimate records. A gate that fires on real data gets turned off, which would be worse than not having one.

Pairs travel whole

The pair is the scoring unit, never the half, so the payload carries complete and passed as their own structure. A renderer rebuilding that from case rows would report a half-graded pair as a pass, which is the degenerate reading the board's pairing exists to prevent.

Verified

ward exec test exits 0: ruff, ruff format, mypy, and 57 tests including 20 new ones.

Not in this PR

The Phoenix client mapping. There is no instance yet, and this payload is the stable thing to map from. ward exec evalkit-export is usable now for screenshots of the payload itself.

🤖 Generated with Claude Code

The exporter from `coilyco-bridge/deploy` #572. My seat owns this half; the Phoenix deployment is ops'. ## One way, and that is the whole design Committed records stay canonical. The payload goes out, nothing comes back, and the display surface is a rebuildable projection rather than a second home for evidence. That shape is what makes this cheap. **#213 step 4** — returning label, critique, evidence, reviewer identity, and provenance to the owning records without manual reconstruction — is the hard requirement for a review UI. This is not one. `evalkit.annotate` stays the grading surface and nothing is authored downstream. ## The grader's notes are withheld by default `critique` and `evidence` are free text a human writes **for herself**, mid-flow, in a keystroke-driven TUI. One committed critique in `ops-board-2026-08-12-regraded` trails off into a note about the terminal rendering badly. Nothing unsafe about it. But it was not written for an audience, and the display target is a permanent public recording. So those two fields need `--include-private`, and the payload records which way it went in `includes_private_fields`. Labels, prompts, targets, outputs, and the pair structure carry by default. **That is what makes a board legible** — the pass/fail shape and the pairing, not the margin notes. ## It refuses rather than redacts A scrubber that misses a pattern ships the secret. This stops instead, names **every** reason in one pass so a fix is not a loop, and exits non-zero. Recognized: AWS key ids, bearer and API tokens, JWTs, private key blocks, SSM parameter paths, Discord snowflakes, tailnet hosts, email addresses. **Withheld text is not scanned**, because text that never leaves cannot leak, and refusing on it would block an export that is safe. There is a test for exactly that asymmetry. ## Two things worth reviewing **A test caught a real bug in the gate.** My SSM pattern required the keyword at the start of the path segment, so `/sirens-echo/postgres-password` — a real parameter in this estate — did not match, because the keyword has a prefix. Fixed, and the parametrized case is what found it. **Verified against real data.** Both committed pilot runs export without refusal, so the gate does not cry wolf on legitimate records. A gate that fires on real data gets turned off, which would be worse than not having one. ## Pairs travel whole The pair is the scoring unit, never the half, so the payload carries `complete` and `passed` as their own structure. A renderer rebuilding that from case rows would report a half-graded pair as a pass, which is the degenerate reading the board's pairing exists to prevent. ## Verified `ward exec test` exits 0: ruff, ruff format, mypy, and 57 tests including 20 new ones. ## Not in this PR The Phoenix client mapping. There is no instance yet, and this payload is the stable thing to map from. `ward exec evalkit-export` is usable now for screenshots of the payload itself. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-16 05:32:32 +00:00

feat(evalkit): project a committed run into a display payload, one way

ci / test (pull_request) Successful in 54s

Details

755b9631a1

The eval display surface reads this. It goes out and nothing comes back,
so committed records stay canonical and the surface is a rebuildable
projection rather than a second home for evidence. Deployment reasoning
in coilyco-bridge/deploy#572.

That one-way shape is what makes it cheap. agent-compose#213 step 4, the
round trip of label, critique, evidence, reviewer identity and
provenance back to the owning records, is the hard requirement for a
review UI. This is not one: evalkit.annotate stays the grading surface
and nothing is authored downstream.

THE GRADER'S NOTES ARE WITHHELD

critique and evidence are free text a human writes for herself, mid-flow,
in a keystroke-driven TUI. One committed critique trails off into a note
about the terminal rendering badly. Useful to the grader, not written for
an audience, and the display target is a permanent public recording.

So they need --include-private, and the payload records which way it went.
Labels, prompts, targets, outputs and the pair structure carry by default,
which is what makes a board legible without the margin notes.

IT REFUSES RATHER THAN REDACTS

A scrubber that misses a pattern ships the secret. This stops, names every
reason in one pass so a fix is not a loop, and exits non-zero. Withheld
text is not scanned, because text that never leaves cannot leak and
refusing on it would block a safe export.

Verified against both committed pilot runs: neither is refused, so the
gate does not cry wolf on real records. One pattern was wrong until a
test caught it - an SSM path like /sirens-echo/postgres-password puts a
prefix before the keyword, which the first expression missed.

Pairs travel as their own structure, since the pair is the scoring unit
and a renderer that rebuilds it from case rows gets a half-graded pair
wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-16 05:32:32 +00:00