evals/240-baseline #247

Merged

coilysiren merged 3 commits from evals/240-baseline into main

2026-08-07 17:30:05 +00:00

coilysiren commented

2026-08-07 17:29:25 +00:00

Owner

No description provided.

coilysiren added 3 commits

2026-08-07 17:29:25 +00:00

test(evals): drive the Claude frontier lane with global memory relocated (#240 ) 8994cb6ee2

Adds --shuffle-config, which moves the host global instruction file aside
for the duration of a run and always restores it. The stored credential is
namespaced by the config directory path rather than its contents, so the
session stays authenticated on the host subscription while host doctrine
leaves the case context. Host hook configuration stays active, so a
residual identity line survives and the records say so.

Records the complete Core Roster frontier lane: 67 cases, all eight roles,
driver claude-sonnet-5 at medium per the pack policy, no retries, $15.35.
Review is pending.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZNXKvYTQxH3RH9ewrt4R3
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

docs: bring the size caps back under the limit (#240 ) 2f52f55e0a

FEATURES.md sat at 4020 chars after the meld merge. Role charter detail
about Designer and Content Creator ownership moves out of the coarse
inventory, which role briefings already own.

evaluation.md crossed the cap when the driver cross-link landed. Its
review-record section splits into docs/evaluation-records.md, which also
documents where unreviewed driver output lives and why it never gates a
release.

Adds .typos.toml. Evaluation records preserve raw model responses, so a
spelling correction inside a preserved answer would edit the evidence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZNXKvYTQxH3RH9ewrt4R3
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

fix(evals): seal driver sessions from real systems (#240 ) c862e46d41

During the first full Core Roster run, the director communication case
posted a decision record to live issue #240 and returned the comment URL
as its answer. The case authorizes a factual record, the session inherited
the host MCP inventory, so the model did what the case said and wrote to a
real tracker. Blast radius was that one comment.

Driver sessions now launch with no MCP servers and with shell, file-write,
and notebook tools denied. Read tools stay, which is what the replay cases
use. Re-running the same case under the seal produces the record in the
response and asks for the issue reference instead of writing.

Every record carries tool_policy. --allow-mutation restores the old
surface for anyone who needs it and marks the run as an experiment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZNXKvYTQxH3RH9ewrt4R3
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilysiren merged commit 976475621d into main

2026-08-07 17:30:05 +00:00

coilysiren referenced this pull request from a commit

2026-08-07 17:30:07 +00:00

Merge pull request 'evals/240-baseline' (#247) from evals/240-baseline into main

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-compose!247

No description provided.

Rows
Columns