Pilot compact Director evaluation review records #225

New issue

Closed

opened 2026-08-06 10:06:56 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-06 10:06:56 +00:00

Member

Outcome

Pilot one human-legible evaluation result for the Core Director role before the new eight-role baseline.

Director is the shortest current role briefing at 316 authored body words, so it is the closest existing example to the proposed 400-word hard ceiling.

Workflow

workflow: remote-branch-only

Kai will review the pilot artifact before the format or role-length policy rolls across the remaining roles.

Acceptance

Preserve the existing role-definition structure and editorial style.
Keep the Director briefing at or below 400 authored body words.
Add one concise durable result format with exact question and exact answer.
Store compact criterion scores and require explanatory notes only for deductions.
Retain model, reviewer, revision, pack digest, finish reason, and retry provenance.
Keep generated execution packs ephemeral.
Run every active Director frontier case in a fresh Codex harness session.
Review responses independently and commit one validated Director record.
Keep historical v2 results readable.
Push the pilot to a remote task branch for Kai's review.
Track global enforcement of the 400-word Core briefing cap after the pilot is accepted.

## Outcome Pilot one human-legible evaluation result for the Core Director role before the new eight-role baseline. Director is the shortest current role briefing at 316 authored body words, so it is the closest existing example to the proposed 400-word hard ceiling. ## Workflow workflow: remote-branch-only Kai will review the pilot artifact before the format or role-length policy rolls across the remaining roles. ## Acceptance * Preserve the existing role-definition structure and editorial style. * Keep the Director briefing at or below 400 authored body words. * Add one concise durable result format with exact question and exact answer. * Store compact criterion scores and require explanatory notes only for deductions. * Retain model, reviewer, revision, pack digest, finish reason, and retry provenance. * Keep generated execution packs ephemeral. * Run every active Director frontier case in a fresh Codex harness session. * Review responses independently and commit one validated Director record. * Keep historical v2 results readable. * Push the pilot to a remote task branch for Kai's review. * Track global enforcement of the 400-word Core briefing cap after the pilot is accepted.

coilyco-ops referenced this issue

2026-08-06 10:08:18 +00:00

Compress Core role briefings and enforce a 400-word ceiling #226

coilyco-ops referenced this issue from a commit

2026-08-06 10:14:15 +00:00

feat(evaluation): add compact review records, refs #225

coilyco-ops referenced this issue from a commit

2026-08-06 10:21:21 +00:00

test(evaluation): add compact Director review, refs #225

coilyco-ops commented

2026-08-06 10:21:39 +00:00

Author

Member

Pilot ready for Kai's review.

Branch - issue-225-compact-director-eval
Compact record - evaluations/pilot/director-codex.yaml
Director length - 316 authored-body words, 344 including frontmatter. It is already under the proposed 400-word ceiling.
Evaluation - eight fresh frontier cases on gpt-5.6-sol at medium reasoning, each independently reviewed in a fresh gpt-5.6-sol high-reasoning session.
Result - 8/8 pass. Seven cases scored full marks. The meeting-deferral case scored 9/10 because its answer used a heading despite an explicit no-heading constraint.
Durable shape - exact question, exact answer, criterion score map, deduction-only notes, and provenance. Execution packs and runner output remained temporary.
Validation - compact decode and frozen-pack validation pass. Build, lint, formatting, and the full pre-commit suite pass.
Known test state - the full repository test run still has only the three accepted identity-refactor failures in internal/converge, internal/evaluation, and internal/person.
400-word enforcement - tracked in #226 for the post-review role compression pass, since five current roles exceed the proposed ceiling.

Workflow remains remote-branch-only. I am leaving this issue open for review.

Pilot ready for Kai's review. * Branch - [issue-225-compact-director-eval](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/src/branch/issue-225-compact-director-eval) * Compact record - [evaluations/pilot/director-codex.yaml](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/src/branch/issue-225-compact-director-eval/evaluations/pilot/director-codex.yaml) * Director length - 316 authored-body words, 344 including frontmatter. It is already under the proposed 400-word ceiling. * Evaluation - eight fresh frontier cases on `gpt-5.6-sol` at medium reasoning, each independently reviewed in a fresh `gpt-5.6-sol` high-reasoning session. * Result - 8/8 pass. Seven cases scored full marks. The meeting-deferral case scored 9/10 because its answer used a heading despite an explicit no-heading constraint. * Durable shape - exact question, exact answer, criterion score map, deduction-only notes, and provenance. Execution packs and runner output remained temporary. * Validation - compact decode and frozen-pack validation pass. Build, lint, formatting, and the full pre-commit suite pass. * Known test state - the full repository test run still has only the three accepted identity-refactor failures in `internal/converge`, `internal/evaluation`, and `internal/person`. * 400-word enforcement - tracked in #226 for the post-review role compression pass, since five current roles exceed the proposed ceiling. Workflow remains `remote-branch-only`. I am leaving this issue open for review.

coilyco-ops referenced this issue

2026-08-06 10:30:30 +00:00

Refresh Core evaluation baseline in compact review format #229

coilyco-ops commented

2026-08-06 10:30:30 +00:00

Author

Member

Kai accepted the compact pilot. The full eight-role rollout continues in #228, including ASCII apostrophe normalization for Forgejo review.

coilyco-ops closed this issue

2026-08-06 10:30:30 +00:00

coilyco-ops commented

2026-08-06 10:30:38 +00:00