eval: publish one-pass v2 baseline snapshot #162

Merged

coilysiren merged 5 commits from issue-150-v2-baseline into main

2026-07-30 18:13:09 +00:00

coilyco-ops commented

2026-07-30 18:06:09 +00:00

Member

Outcome

Replaces the preserved v1 records with the eight-role v2 Core Roster evidence captured at exact source revision 562be1a95883e35d80866eab890e5477d020cd95.
Preserves all 96 model responses, exact model identities, pack digests, full source revision, criterion evidence, verdicts, and retry provenance.
Records one independent QA score set per case with no quality retry or prompt-tuning loop.
Extends historical scorecard parsing to understand v2 scenario IDs, including adjacent-role cases, without rebinding immutable results to a changed roster.

Result for the evaluated snapshot

Frontier: 48/48 pass, including every role, personality, and adjacent-role gate in that snapshot.
OSS: 0/48 pass and remains explicitly unsupported.
Aggregate: 48/96 cases, 530/768 points.
The initial OSS runner attempt failed before inference because its state database targeted a read-only directory. The preserved attempt-2 responses were generated once after isolating writable runner state.

Main advanced during the run with dd81035, which changed the Design briefing and scenarios. Per the one-pass instruction, this PR does not rerun or rescore them. The evidence lands honestly as an immutable snapshot and does not clear the final-current v2 release gate. Issue #150 remains open.

Validation

ward exec test

## Outcome * Replaces the preserved v1 records with the eight-role v2 Core Roster evidence captured at exact source revision `562be1a95883e35d80866eab890e5477d020cd95`. * Preserves all 96 model responses, exact model identities, pack digests, full source revision, criterion evidence, verdicts, and retry provenance. * Records one independent QA score set per case with no quality retry or prompt-tuning loop. * Extends historical scorecard parsing to understand v2 scenario IDs, including adjacent-role cases, without rebinding immutable results to a changed roster. ## Result for the evaluated snapshot * Frontier: 48/48 pass, including every role, personality, and adjacent-role gate in that snapshot. * OSS: 0/48 pass and remains explicitly unsupported. * Aggregate: 48/96 cases, 530/768 points. * The initial OSS runner attempt failed before inference because its state database targeted a read-only directory. The preserved attempt-2 responses were generated once after isolating writable runner state. Main advanced during the run with `dd81035`, which changed the Design briefing and scenarios. Per the one-pass instruction, this PR does not rerun or rescore them. The evidence lands honestly as an immutable snapshot and does not clear the final-current v2 release gate. Issue #150 remains open. ## Validation * `ward exec test`

coilyco-ops added 3 commits

2026-07-30 18:06:09 +00:00