Define the Core Roster v2 evaluation matrix #151

Merged

coilysiren merged 1 commit from issue-150-v2-evaluation into issue-148-remove-v1-integrations

2026-07-30 16:50:18 +00:00

coilyco-ops commented

2026-07-30 12:45:26 +00:00

Member

Part of #150.

Defines the deterministic v2 evaluation program without fabricating model or QA evidence.

48 authored scenarios expand to 96 paired frontier and OSS cases across all eight Core Roster roles.
Every role covers mission, personality, authority, completion, and one real-portfolio replay.
All approved adjacent-role boundaries are explicit and machine-validated.
evaluation --all writes eight self-contained packs plus a digest index to an empty directory.
V2 results require exact full source revision, prompt author, a different reviewer, pack digest, retry provenance, raw responses, criterion evidence, totals, and derived verdicts.
Scorecards surface frontier and OSS adjacent-role gates separately.

Validation:

ward --audit-override-dirty exec test
ward --audit-override-dirty exec smoke
ward --audit-override-dirty exec lint

This PR intentionally leaves #150 open. External frontier/OSS model runs and independent QA scoring remain required before the v2 baseline can land.

Part of #150. Defines the deterministic v2 evaluation program without fabricating model or QA evidence. * 48 authored scenarios expand to 96 paired frontier and OSS cases across all eight Core Roster roles. * Every role covers mission, personality, authority, completion, and one real-portfolio replay. * All approved adjacent-role boundaries are explicit and machine-validated. * `evaluation --all` writes eight self-contained packs plus a digest index to an empty directory. * V2 results require exact full source revision, prompt author, a different reviewer, pack digest, retry provenance, raw responses, criterion evidence, totals, and derived verdicts. * Scorecards surface frontier and OSS adjacent-role gates separately. Validation: * `ward --audit-override-dirty exec test` * `ward --audit-override-dirty exec smoke` * `ward --audit-override-dirty exec lint` This PR intentionally leaves #150 open. External frontier/OSS model runs and independent QA scoring remain required before the v2 baseline can land.

coilyco-ops added 1 commit

2026-07-30 12:45:26 +00:00