coilyco-flight-deck/agent-compose
coilyco-flight-deck/agent-compose
Watch
2
0
Code Issues 20 Pull requests Projects Releases 161 Packages Wiki Activity Actions

test(evals): compare opus-5 high against sonnet-5 medium (#240) #245

Merged
coilysiren merged 1 commit from evals/240-model-arms into main 2026-08-07 05:15:33 +00:00
Conversation 0 Commits 1 Files changed 4 +3173
coilysiren commented 2026-08-07 05:14:30 +00:00
Owner
Copy link

Adds the reviewer that scores preserved responses in separate sessions,
blind to which arm produced an answer, with verdicts computed from the
pack review rule rather than requested from the reviewer.

Records the ai and engineer frontier pilot: opus-5 high passes 17 of 18
cases at 143/150 points, sonnet-5 medium passes 10 of 18 at 113/150. Both
arms fail the engineer implementation-checkpoint case by crossing the
communication-ownership boundary while over-deferring the record they own.

Driver sessions ran on the host home, so these records compare arms but
cannot serve as bundle-behavior evidence.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_012ZNXKvYTQxH3RH9ewrt4R3
Co-authored-by: Kai Siren coilysiren@gmail.com
Co-authored-by: Claude noreply@anthropic.com

Adds the reviewer that scores preserved responses in separate sessions, blind to which arm produced an answer, with verdicts computed from the pack review rule rather than requested from the reviewer. Records the ai and engineer frontier pilot: opus-5 high passes 17 of 18 cases at 143/150 points, sonnet-5 medium passes 10 of 18 at 113/150. Both arms fail the engineer implementation-checkpoint case by crossing the communication-ownership boundary while over-deferring the record they own. Driver sessions ran on the host home, so these records compare arms but cannot serve as bundle-behavior evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZNXKvYTQxH3RH9ewrt4R3 Co-authored-by: Kai Siren <coilysiren@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
Adds the reviewer that scores preserved responses in separate sessions,
blind to which arm produced an answer, with verdicts computed from the
pack review rule rather than requested from the reviewer.

Records the ai and engineer frontier pilot: opus-5 high passes 17 of 18
cases at 143/150 points, sonnet-5 medium passes 10 of 18 at 113/150. Both
arms fail the engineer implementation-checkpoint case by crossing the
communication-ownership boundary while over-deferring the record they own.

Driver sessions ran on the host home, so these records compare arms but
cannot serve as bundle-behavior evidence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZNXKvYTQxH3RH9ewrt4R3
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
qa-fixture
Disposable issue admitted to the bounded Ward QA verification lane.

  
role/ai
requires work from the AI Engineer role

  
role/creator
requires work from Content Creator role

  
role/design
requires work from the design role

  
role/director
requires work from the director role

  
role/engineer
requires work from the engineer role

  
role/exec
requires work from the exec role

  
role/human
requires a person, and specifically not an agent seat

  
role/ops
requires work from the ops role

  
role/qa
requires work from the QA role

No labels
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/ai
role/creator
role/design
role/director
role/engineer
role/exec
role/human
role/ops
role/qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/agent-compose!245
No description provided.