Improve Claude commodity role behavior #234

New issue

Closed

opened 2026-08-06 16:22:08 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-06 16:22:08 +00:00

Member

Outcome

Improve the 12 failed Claude Sonnet commodity cases recorded by #232 by correcting the lowest owning role, scenario, or runner-contract layer, then rerun every affected role's complete commodity pack.

Baseline

Candidate: claude-sonnet-5, medium effort
Reviewer: gpt-5.6-sol, high effort
Commodity result: 55/67 passing, 507/560 points
Failed roles: Engineer, Director, Designer, Portfolio Strategist, AI Engineer

Acceptance

Classify every failure against preserved response and reviewer evidence.
Do not retry an unchanged behavioral failure.
Keep every role briefing at or below 400 authored body words.
Preserve role ownership and authority boundaries.
Make evaluation scenarios supply enough facts for a direct answer without repository or tool access.
Rerun the complete commodity pack for every changed role in fresh isolated sessions.
Preserve compact v3 results and regenerate the scorecard.
Keep the commodity lane disabled until the evidence supports a separate admission decision.
Run validation and land through workflow: direct-to-main.

## Outcome Improve the 12 failed Claude Sonnet commodity cases recorded by #232 by correcting the lowest owning role, scenario, or runner-contract layer, then rerun every affected role's complete commodity pack. ## Baseline * Candidate: `claude-sonnet-5`, medium effort * Reviewer: `gpt-5.6-sol`, high effort * Commodity result: 55/67 passing, 507/560 points * Failed roles: Engineer, Director, Designer, Portfolio Strategist, AI Engineer ## Acceptance * Classify every failure against preserved response and reviewer evidence. * Do not retry an unchanged behavioral failure. * Keep every role briefing at or below 400 authored body words. * Preserve role ownership and authority boundaries. * Make evaluation scenarios supply enough facts for a direct answer without repository or tool access. * Rerun the complete commodity pack for every changed role in fresh isolated sessions. * Preserve compact v3 results and regenerate the scorecard. * Keep the commodity lane disabled until the evidence supports a separate admission decision. * Run validation and land through `workflow: direct-to-main`.

coilyco-ops referenced this issue from a commit

2026-08-06 16:38:05 +00:00

test(evals): clarify commodity scenarios (#234)

coilyco-ops referenced this issue from a commit

2026-08-06 16:53:29 +00:00

test(evals): make completion cases executable (#234)

coilyco-ops referenced this issue from a commit

2026-08-06 17:03:03 +00:00

test(evals): retain fully reviewed AI pack (#234)

coilyco-ops referenced this issue

2026-08-06 17:06:29 +00:00

Resolve final Claude commodity evaluation misses #235

coilyco-ops referenced this issue from a commit

2026-08-06 17:06:39 +00:00

test(evals): record improved commodity evidence (#234)

coilyco-ops commented

2026-08-06 17:06:58 +00:00

Author

Member

Landed the score improvement on main.

Result

Scorecard: 122/134 -> 131/134 passing
Points: 1065/1120 -> 1091/1120
Improvement: +9 passing cases, +26 points
Claude commodity target resolved consistently to claude-sonnet-5 at medium effort.
Independent review used fresh gpt-5.6-sol high sessions.
Reviewer context now includes the same repository instructions available to the candidate.
Compact v3 records and the generated scorecard are current.

Landed commits

5fc064a - clarify commodity scenarios
f17e87e - make completion cases executable
61c7e3e - retain the fully reviewed AI pack after reviewer quota exhaustion
7fdee39 - record improved commodity evidence and refresh the scorecard

Validation

build: pass
lint: pass
pre-commit: pass
scorecard freshness: pass
result records match current pack digests: pass
full tests retain only the three accepted roster-refactor failures already tracked this round

Three evidence-backed commodity misses remain visible. Follow-up: #235.

Landed the score improvement on `main`. ## Result * Scorecard: **122/134 -> 131/134 passing** * Points: **1065/1120 -> 1091/1120** * Improvement: **+9 passing cases, +26 points** * Claude commodity target resolved consistently to `claude-sonnet-5` at medium effort. * Independent review used fresh `gpt-5.6-sol` high sessions. * Reviewer context now includes the same repository instructions available to the candidate. * Compact v3 records and the generated scorecard are current. ## Landed commits * `5fc064a` - clarify commodity scenarios * `f17e87e` - make completion cases executable * `61c7e3e` - retain the fully reviewed AI pack after reviewer quota exhaustion * `7fdee39` - record improved commodity evidence and refresh the scorecard ## Validation * build: pass * lint: pass * pre-commit: pass * scorecard freshness: pass * result records match current pack digests: pass * full tests retain only the three accepted roster-refactor failures already tracked this round Three evidence-backed commodity misses remain visible. Follow-up: #235.

coilyco-ops closed this issue

2026-08-06 17:07:03 +00:00

coilyco-ops referenced this issue

2026-08-07 04:30:38 +00:00

finish role melds: re-earn the eight evaluation records #240