refactor!: retire the driver-plus-reviewer eval stack (#262) #269

Merged

coilysiren merged 11 commits from evals/262-board-run into main

2026-08-12 09:57:45 +00:00

coilyco-ops commented

2026-08-12 09:14:46 +00:00

Member

Surfaced by a fleet branch sweep. This branch was pushed but had no pull request, so the work was sitting on the forge with nothing tracking a decision about it.

Nothing has been rebased, squashed, or edited. The branch is exactly as it was.

Last commit: 2026-08-12
Unmerged commits (no patch-equivalent on main): 10
Files still differing from main: 72

b3be43f refactor!: retire the driver-plus-reviewer eval stack (#262)
8ea6681 fix(evalkit): scope the annotation count and unbreak long critiques (#262)
f13dea3 refactor(evalkit)!: delete the regex discriminator tier (#262)
fa02b09 test(evals): probe absent-authority before editing doctrine (#262)
41ab299 test(evals): preserve the graded ops pilot board (#262)
963b4d3 fix(evalkit): give the annotate verb its paths (#262)
736a327 fix(evalkit): report item analysis instead of gating on it (#262)
0c94735 refactor(evaluation)!: test one commodity subject tier, retire the lane axis
7e8532a test(evals): make the ops crossings incidental and the patterns artifact-shaped (#262)
3ddee86 fix(evalkit): resolve run paths before Inspect changes directory (#262)
48ffbe9 test(evals): author the ops pilot slice of the board (#262)

Two independent checks put this outside main: no patch-equivalent commit on main, and the touched files still differ from main's current versions. Both can misfire on a branch reworked before landing, so confirm before merging. If it is obsolete, close this and delete the branch - the point was to make the decision visible.

Surfaced by a fleet branch sweep. This branch was pushed but had no pull request, so the work was sitting on the forge with nothing tracking a decision about it. Nothing has been rebased, squashed, or edited. The branch is exactly as it was. **Last commit:** 2026-08-12 **Unmerged commits (no patch-equivalent on main):** 10 **Files still differing from main:** 72 - b3be43f refactor!: retire the driver-plus-reviewer eval stack (#262) - 8ea6681 fix(evalkit): scope the annotation count and unbreak long critiques (#262) - f13dea3 refactor(evalkit)!: delete the regex discriminator tier (#262) - fa02b09 test(evals): probe absent-authority before editing doctrine (#262) - 41ab299 test(evals): preserve the graded ops pilot board (#262) - 963b4d3 fix(evalkit): give the annotate verb its paths (#262) - 736a327 fix(evalkit): report item analysis instead of gating on it (#262) - 0c94735 refactor(evaluation)!: test one commodity subject tier, retire the lane axis - 7e8532a test(evals): make the ops crossings incidental and the patterns artifact-shaped (#262) - 3ddee86 fix(evalkit): resolve run paths before Inspect changes directory (#262) - 48ffbe9 test(evals): author the ops pilot slice of the board (#262) Two independent checks put this outside main: no patch-equivalent commit on main, and the touched files still differ from main's current versions. Both can misfire on a branch reworked before landing, so confirm before merging. If it is obsolete, close this and delete the branch - the point was to make the decision visible.

coilyco-ops added 11 commits

2026-08-12 09:14:47 +00:00

test(evals): author the ops pilot slice of the board (#262 ) 48ffbe95af

Ops is the pilot because it is the only role that owns one boundary and
defers two, so one role exercises every structural shape the board has:
owner in-out pairs, deferrer in-out pairs, role fit, and personality.

Twenty-one candidates. Two variants compete for each of the nine
pass-or-fail slots and one survives item analysis; the three personality
cases run single. Variants share a target verbatim, because the target is
part of the slot key filter.py builds.

Discriminators avoid matching the refusal verb itself. A passing answer
says it will not draft the wording, so a bare draft pattern would fail the
response it should pass. They match produced artifacts instead: a long
quoted span, a code fence, an assignment, an outside-world claim.

The documentation-layout exclude unblocks committing after a prompt
render. That hook walks the filesystem while its siblings enumerate
git ls-files, so gitignored generated Markdown still reaches it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

fix(evalkit): resolve run paths before Inspect changes directory (#262 ) 3ddee86a4d

Inspect loads a task file with the working directory set to that file's
folder, so every relative path handed to eval-run.sh resolved under
evalkit/ and the run died on a missing samples.yaml. The default paths
were never exercised, because no authored samples existed until now.

Absolute-ise samples, prompts, and the log directory at the top. The
helper does not stat anything, so a missing path still fails inside
Inspect with its own message rather than in the wrapper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

test(evals): make the ops crossings incidental and the patterns artifact-shaped (#262 ) 7e8532a5cd

The first slice passed 17 of 17 pass-or-fail cases because every prompt
announced its boundary in the headline. The bundle names those behaviours
directly, so the subject answered from the surface of the request. Each
crossing now sits inside a plausible ops task: the status page entry is
one item in an incident wrap-up, the outside comparison is one line of a
capacity plan, the code fix arrives as a one-line favour.

Discriminators no longer match the verb. A correct refusal uses the same
verb the failure does, which is how "before calling the incident resolved"
scored as a failure last run. They now match artifacts a correct answer
cannot contain: a quoted message, a status field at line start, a code
token, an owner assignment, an unsourced quantifier about the outside
world.

Validated against the previous run's responses, which are known-correct.
Zero patterns fire on them, except the three sev-out epochs that smuggle
"fine for many workloads" in as a premise. That leak is real and the old
pattern was blind to it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

refactor(evaluation)!: test one commodity subject tier, retire the lane axis 0c94735565

The board expanded every scenario into frontier, commodity, and OSS lanes,
validated all three in coverage, and rendered nine scorecard columns. Two
of those lanes were never executed. They were carried as structure and as
a `disabled_model_tiers` marker that existed to explain why.

Each scenario now becomes one case on the commodity subject tier. Coverage
rejects a case off that tier and a repeated scenario instead of demanding
three lanes and comparing prompts between them. The scorecard collapses
from a frontier-by-commodity-by-OSS cross product to one lane whose header
names the tier the records carry, so an archived frontier record still
renders in historical mode. A record mixing two tiers is now rejected
rather than summed.

Deployment tier is untouched and stays a separate axis. Roles are still
used on frontier and OSS models, Content Creator's OSS Discord seat
included. The board does not read a role's declared tier, because tier does
not change selected context, so one subject measures the composed text for
every role including the four declared frontier-only. What that costs is
stated in docs/model-tiers.md rather than left implicit: no evidence about
frontier or OSS behaviour, and a tier comparison is a separate arm.

TestLatestScoredResultsMatchCurrentPacks still fails. It failed before this
change for the same reason, since the committed baseline was already
retired by the adjacency digest move, and the honest fix is a re-earned
baseline rather than a re-digested one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

fix(evalkit): report item analysis instead of gating on it (#262 ) 736a3276c2

Two ops runs, 170 subject calls, and 17 of 18 candidates sat at zero
failures. The filter kept three personality cases both times and deleted
the rest of the board.

The one real failure it found, ops closing an incident on twenty clean
dashboard minutes at 4 of 5, was deleted too. Its pair's in-half scored
zero because ops correctly owns the live change, item analysis read that
as non-discriminating, and the pair-completeness rule then took the far
half with it. A pair could only survive when both halves failed, so the
pipeline structurally could not report a role that holds its own work and
over-claims across the boundary.

An in-half and a within-role case are negative controls. Passing every run
is the control working, so they are now exempt and kept unconditionally
like personality cases. Nothing else is dropped for failing to
discriminate either: a pattern cannot see polarity, so a sample it never
fires on is either an easy case or a blind regex, and only the human can
tell which. Out-of-band samples and flat pairs are noted instead, and a
note is a lead for the next generation pass.

The slot competition stays. Two candidates still compete and the one
closest to the midpoint wins, now with the variant number as a
tiebreak so a pair of zeroes resolves deterministically.

On the recorded ops run this takes the graded dataset from 3 cases to 12,
which is what the derived board says ops owes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

fix(evalkit): give the annotate verb its paths (#262 ) 963b4d3c75

`ward exec evalkit-annotate` ran the module bare, and argparse requires
--dataset and --out, so the verb exited 2 before doing anything. It also
never rendered the roster, so the per-role header it documents had no
person.json to read.

The script resolves both, matching eval-matrix.sh, and fails with a
pointer at evalkit-filter when no dataset exists yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

test(evals): preserve the graded ops pilot board (#262 ) 41ab299d29

Twelve cases graded by hand on 2026-08-12, the first human-labelled
evidence this board has produced. Dataset and annotations together, so the
record is self-contained without the gitignored run cache.

Pairs: modify-live-system fails, seek-external-validation and
suggest-human-comms pass. Two deductions, both on cases the discriminator
patterns scored as passing.

RTO joins the preserved-answer word list for the reason already at the top
of .typos.toml. It is recovery time objective inside a subject response,
and correcting a preserved answer edits the evidence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

test(evals): probe absent-authority before editing doctrine (#262 ) fa02b093b7

The graded ops board failed modify-live-system on its in-half, and the
failing answer was near-verbatim doctrine, which read like the clause
handing every role a permanent escape hatch. Two candidate fixes were
safety-relevant, so the cause needed demonstrating first.

Two factors crossed at five epochs. Narrating the authority gap produces a
refusal to act in 10 of 10. Not narrating it produces 0 of 10, with the
authority condition surviving as a trailing qualifier, which is what the
charter asks for. Deduplicating the clause between role-ops and the
boundary changes nothing either way.

So the clause is mirrored rather than reached for, and ops-mls-in-a
measured its own framing. Invalid case, not a doctrine defect. No doctrine
change is indicated and none is made here.

Counts come from reading all twenty responses. Three lexical detectors
disagreed with each other and with the reading, the third time this session
a pattern has diverged from a reading in the same direction. Provenance
rides in the evidence file rather than a sibling README, since a module
README here is signage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

refactor(evalkit)!: delete the regex discriminator tier (#262 ) f13dea3994

The first human-graded board measured it. Across nine pass-or-fail cases
the patterns and the grader agreed on nothing that mattered: one false
positive on ops-mls-out, two false negatives on the only cases Kai failed,
and six agreements where nothing happened. The absent-authority probe
repeated it, with three detectors disagreeing with each other and with a
reading of the same twenty responses.

It measured something, but not what the grading measures, so it goes rather
than gets tuned.

The match count was also the only rule for choosing between two candidates
for a slot, so variants go with it. One authored case per slot, every one
of them graded. Epochs stay at five: the annotator sees epoch 1 and the
rest stay in the Inspect log, where a reader can check whether an answer
held across runs without anything scoring it for them.

evalkit.filter keeps its name while its job narrows to building the dataset
from a run. Renaming it is a separate change and not this one.

ops-mls-in is rewritten with the plain framing. The probe showed the old
wording produced a refusal in 10 of 10 regardless of bundle, so it measured
its own framing rather than the boundary. ops-fit-engineer keeps its
contested target with the disagreement named in it, pending the doctrine
call on where operational tuning ends and product logic begins.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

fix(evalkit): scope the annotation count and unbreak long critiques (#262 ) 8ea6681593

Two defects the regrade surfaced.

The summary counted every annotation in the file against the current
dataset, so a case rename left it reporting "annotated 21 of 12". It now
counts this dataset's ids and names the stale remainder instead of folding
it into the total.

console.input hands readline a prompt carrying ANSI colour codes. Readline
miscounts the cursor column from them, so a critique long enough to wrap
overwrote itself mid-sentence and one of Kai's was lost that way. The style
prints first and readline gets an empty prompt to measure.

Also preserves the regraded board. All three pairs pass. The two remaining
deductions are both the authority hedge, on cases whose prompts never
mention authority.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

refactor!: retire the driver-plus-reviewer eval stack (#262 ) b3be43f88b

The repo carried two eval systems. The retired one was the larger: 2,827
lines of Go and Python against evalkit's 1,115, eight ward verbs against
eleven, and it still gated main through a baseline earned under a
methodology #262 replaced for judging itself.

Deleted: internal/evaluation, the pack and scorecard subcommands,
evaluation_driver.py, evaluation_reviewer.py, earn-evaluation-baseline.sh,
render-evaluation-packs.sh, scripts/evaluation-record, eight evaluation-*
ward verbs, six docs, and the per-role evals.yaml matrices with the person
loader plumbing that read them.

Two things this fixes rather than merely removes.

Bundle content digests included evaluation asset digests, so editing an
eval prompt changed the digest of a bundle whose delivered content did not
change. Evaluation matrices are never delivered to an agent. Every bundle
digest moves once as a result.

TestLatestScoredResultsMatchCurrentPacks gated main on records the
adjacency merge had already retired. Deleting the system deletes the gate,
and the Go suite is green for the first time since that merge.

Preserved, not deleted: evaluations/latest moves to
evaluations/retired-2026-08-06-codex, per the policy the old stack itself
wrote down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-12 09:14:47 +00:00