Refresh current Design evidence for the v2 release gate

coilyco-ops commented

2026-07-31 06:39:52 +00:00

Member

Parent: #142

Outcome

Refresh only the changed Core Roster Design evaluation pack against exact current canonical main. Reuse the other seven role results because their pack digests remain identical to the independently reviewed baseline.

Evidence

Current pack comparison shows that engineer, director, QA, ops, community, strats, and content still match their committed digests. Design changed from sha256:7895f52b...16b1 to sha256:40274892...9dfa after #160 changed the Design briefing and scenarios.

Work

Freeze and record the exact source revision and generated Design pack digest.
Run the six Design frontier cases first. Stop on a failed frontier release gate.
Run all six Design OSS cases after frontier passes. Preserve failures and retry provenance.
Have an independent reviewer score every case with one evidence sentence per criterion.
Replace only evaluations/latest/design-codex.yaml.
Restore the normal current-pack scorecard check by removing historical mode from the repository scorecard verbs and test gate.
Regenerate docs/evaluation-scores.md.

Acceptance

The refreshed Design result validates against the exact current pack and contains complete frontier and OSS lanes.
All six frontier Design cases pass, including the Content boundary case.
OSS support remains explicit and fail closed according to the recorded verdicts.
The other seven result files remain byte-unchanged.
The non-historical scorecard validates all eight current role packs.
ward exec test and ward exec smoke pass from clean branch state.

Parent: #142 ## Outcome Refresh only the changed Core Roster Design evaluation pack against exact current canonical main. Reuse the other seven role results because their pack digests remain identical to the independently reviewed baseline. ## Evidence Current pack comparison shows that engineer, director, QA, ops, community, strats, and content still match their committed digests. Design changed from `sha256:7895f52b...16b1` to `sha256:40274892...9dfa` after #160 changed the Design briefing and scenarios. ## Work * Freeze and record the exact source revision and generated Design pack digest. * Run the six Design frontier cases first. Stop on a failed frontier release gate. * Run all six Design OSS cases after frontier passes. Preserve failures and retry provenance. * Have an independent reviewer score every case with one evidence sentence per criterion. * Replace only `evaluations/latest/design-codex.yaml`. * Restore the normal current-pack scorecard check by removing historical mode from the repository scorecard verbs and test gate. * Regenerate `docs/evaluation-scores.md`. ## Acceptance * The refreshed Design result validates against the exact current pack and contains complete frontier and OSS lanes. * All six frontier Design cases pass, including the Content boundary case. * OSS support remains explicit and fail closed according to the recorded verdicts. * The other seven result files remain byte-unchanged. * The non-historical scorecard validates all eight current role packs. * `ward exec test` and `ward exec smoke` pass from clean branch state.

coilyco-ops referenced this issue

2026-07-31 07:52:10 +00:00

Record the direct Agent Proxy Design gate verdict #164

coilyco-ops referenced this issue

2026-07-31 07:53:21 +00:00

eval: record direct Design gate verdict #165

coilyco-ops commented

2026-07-31 07:53:32 +00:00

Author

Member

Direct harness-free evaluation is complete and preserved in PR #165.

Frontier route designer/product-shaping: 0 of 6 cases pass. Four responses scored below gate and two were rejected by Agent Proxy for validation:ungrounded_action_claim.
OSS route designer/design-production: 2 of 6 cases pass. Two responses pass, two score below gate, and two were rejected for the same governed failure.
Combined Design score: 40/96.
No evaluated response was retried.
ward exec test and ward exec smoke pass from clean commit 85508c8.

The required all-frontier-pass acceptance is not met, so this issue and the .release-major hold remain open.

Direct harness-free evaluation is complete and preserved in PR #165. * Frontier route `designer/product-shaping`: 0 of 6 cases pass. Four responses scored below gate and two were rejected by Agent Proxy for `validation:ungrounded_action_claim`. * OSS route `designer/design-production`: 2 of 6 cases pass. Two responses pass, two score below gate, and two were rejected for the same governed failure. * Combined Design score: 40/96. * No evaluated response was retried. * `ward exec test` and `ward exec smoke` pass from clean commit `85508c8`. The required all-frontier-pass acceptance is not met, so this issue and the `.release-major` hold remain open.

coilyco-ops referenced this issue

2026-07-31 08:13:25 +00:00

Replace routed Design evidence with pinned Ornith results #166

coilyco-ops referenced this issue

2026-07-31 08:14:49 +00:00

eval: pin Design field to Ornith 35B #167

coilyco-ops commented

2026-07-31 08:15:02 +00:00

Author

Member

Corrected direct-model evidence is now in PR #167.

All twelve Design cases used ornith:35b through Agent Proxy route community/knowledge-retrieval. The deployed route has Ornith 35B as its sole target and an empty fallback list. No response was retried.

Full-context lane: 2 of 6 pass, 35/48 points
Low-context lane: 4 of 6 pass, 38/48 points
Combined Design score: 73/96
ward exec test and ward exec smoke pass from clean commit bd2187a

This supersedes the role-route result from PR #165. The all-full-context-pass gate remains unmet, so this issue and .release-major stay open.

Corrected direct-model evidence is now in PR #167. All twelve Design cases used `ornith:35b` through Agent Proxy route `community/knowledge-retrieval`. The deployed route has Ornith 35B as its sole target and an empty fallback list. No response was retried. * Full-context lane: 2 of 6 pass, 35/48 points * Low-context lane: 4 of 6 pass, 38/48 points * Combined Design score: 73/96 * `ward exec test` and `ward exec smoke` pass from clean commit `bd2187a` This supersedes the role-route result from PR #165. The all-full-context-pass gate remains unmet, so this issue and `.release-major` stay open.

coilyco-ops referenced this issue

2026-08-02 00:06:20 +00:00

Add AI Engineer as a Core Roster role #175

coilyco-ops referenced this issue

2026-08-02 04:46:32 +00:00

Ops: authorize P0 Agent Compose v2.0.0 release override #176

coilyco-ops referenced this issue

2026-08-02 04:57:20 +00:00

Ops: authorize P0 Agent Compose v2.0.0 release override #176

coilyco-ops referenced this issue

2026-08-02 05:09:44 +00:00

Ops: authorize P0 Agent Compose v2.0.0 release override #176

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-02 11:38:34 +00:00

fix: make Designer frontier evaluation green #177

coilyco-ops commented

2026-08-02 11:39:16 +00:00

Author

Member

Current Designer gate is green in PR #177.

Exact current Design pack: sha256:244dfaad780638fb91f9a6aa7249c8ae7f1cb810a10199a817216d8a7d5eb33c
Frontier: all 6 cases pass, 47/48 points, including the Content boundary.
OSS: 5 of 6 cases pass. The completion case returned empty content with finish reason length and remains an explicit 0/8 failure, so support stays fail closed.
No candidate response required a retry.
Every criterion was scored by the independent evaluation/ornith-35b rubric reviewer.
The other seven role result files remain byte-unchanged.
The non-historical scorecard check, ward exec test, and ward exec smoke all pass at b97ca71.

Current Designer gate is green in PR #177. * Exact current Design pack: sha256:244dfaad780638fb91f9a6aa7249c8ae7f1cb810a10199a817216d8a7d5eb33c * Frontier: all 6 cases pass, 47/48 points, including the Content boundary. * OSS: 5 of 6 cases pass. The completion case returned empty content with finish reason length and remains an explicit 0/8 failure, so support stays fail closed. * No candidate response required a retry. * Every criterion was scored by the independent evaluation/ornith-35b rubric reviewer. * The other seven role result files remain byte-unchanged. * The non-historical scorecard check, ward exec test, and ward exec smoke all pass at b97ca71.

coilysiren closed this issue

2026-08-02 19:18:43 +00:00

Rows
Columns

Refresh current Design evidence for the v2 release gate #163

Outcome

Evidence

Work

Acceptance