meld/231-shared-role-melds

coilysiren commented

2026-08-07 05:15:55 +00:00

Owner

No description provided.

coilysiren added 2 commits

2026-08-07 05:15:55 +00:00

docs(features): fit the merged inventory under the size cap (#231 ) cba79ac835

The meld entry plus the native-UI entries landing on main pushed
docs/FEATURES.md 20 chars past the 4000-char catalog cap. Tighten four
inventory lines without dropping a capability.

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

test(evals): re-earn a baseline end to end (#231 ) 24eb173599

The meld extraction retires every recorded score, and the repo could
render packs and validate records but had no way to produce them. Close
that gap.

evaluation_reviewer.py scores a preserved response in a separate session
with tools disabled, supplying judgement only. The evaluation-record
command joins a driver run with its review and writes the records through
the owning marshaller, deriving every total and verdict from the pack
review rule. earn-evaluation-baseline.sh runs the whole pass behind
ward exec evaluation-baseline, giving the driver and the reviewer
separate isolated homes so the reviewer never inherits the projected
role context.

Pack rendering now takes a seat instead of hardcoding codex.

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops referenced this pull request

2026-08-07 05:37:49 +00:00

finish role melds: re-earn the eight evaluation records #240

coilyco-ops added 1 commit

2026-08-07 18:14:41 +00:00

merge: reconcile the reviewer with main's arm lane (#231 )

release / release (push) Failing after 14s

Details

a96b5a4184

Two lanes wrote scripts/evaluation_reviewer.py in parallel. #240 built it for
a blind two-arm comparison, and #231 rebuilt it for an isolated baseline. Keep
the #231 base, which supplies judgement only and leaves totals and verdicts to
the pack review rule, and graft back what the arm lane needs: --runs takes one
file per arm, the skip for cases the driver could not complete, and the
--output-format json envelope that carries reviewer cost.

Fix three defects the merge exposed:

--tools is variadic, so passing it an empty string swallowed the prompt
argument that followed and every review would have failed with no input. Both
seal flags now take the equals form, matching the driver's --mcp-config idiom.

evaluation-record set the v3 format up front, which skips the branch in
MarshalResult that stamps the pack digest, so every record it wrote failed
validation against its own pack. It now binds the digest explicitly.

The driver records the attempt that worked with an empty reason, which
provenance validation rejects. The writer names the fact instead.

evaluation-record also filters reviews by arm so a two-arm file cannot
cross-join, and fails loudly when a succeeded case has no review rather than
writing it as a silent zero.

docs/FEATURES.md takes main's shorter wording, which is what fits the cap.

The evaluation gate stays red. Every pack digest changed when melds moved
doctrine out of the role bodies, so the recorded codex baseline no longer
applies. main is red on the same gate today. Re-earning it is separate work.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops commented

2026-08-07 18:18:23 +00:00

Member

Landed on main as a96b5a4. The branch is fully contained in main, so this PR's diff is now empty and it closes here.

What resolved. Both lanes had independently written scripts/evaluation_reviewer.py, so the conflict was add/add rather than a text overlap. The #231 base survives, because it supplies judgement only and leaves totals and verdicts to the pack review rule instead of restating that rule in Python. Grafted back from the #240 arm lane: --runs takes one file per arm, cases the driver could not complete are skipped, and the --output-format json envelope carries reviewer cost again.

docs/FEATURES.md took main's shorter wording, which is the one that fits the size cap.

Three defects the merge exposed, all fixed.

--tools is variadic, so "--tools", "", prompt swallowed the prompt and the harness exited with Input must be provided. Every review on this branch would have failed. Both seal flags now use the equals form, matching the driver's --mcp-config idiom.
evaluation-record set the v3 format up front, which skips the branch in MarshalResult that stamps the pack digest, so every record it wrote failed validation against its own pack. It now binds the digest explicitly.
The driver records the attempt that worked with an empty reason, which provenance validation rejects. The writer names the fact rather than dropping the attempt.

evaluation-record also filters reviews by arm so a two-arm file cannot cross-join, and fails loudly when a succeeded case has no review instead of writing it as a silent zero.

Verified. The reviewer ran green against two cases from the committed evaluations/baseline/claude-frontier-driver.json, returning valid scores, evidence, and cost. A synthetic full-coverage review then drove evaluation-record through all 67 cases and all eight roles, which is what surfaced the digest and provenance defects. Those runs used the host home and are plumbing only, not evidence. go build, go vet, and pre-commit run --all-files are clean.

Still open. TestLatestScoredResultsMatchCurrentPacks fails for all eight packs. Every pack digest changed when melds moved doctrine out of the role bodies, so the recorded codex baseline no longer applies. main was red on the same gate before this merge and the gate inputs are byte-identical, so this does not regress it. Re-earning still needs one interactive /login per isolated home.

Kai is reworking the evaluation subsystem, so the baseline is deliberately not re-earned here.

Landed on `main` as `a96b5a4`. The branch is fully contained in `main`, so this PR's diff is now empty and it closes here. **What resolved.** Both lanes had independently written `scripts/evaluation_reviewer.py`, so the conflict was add/add rather than a text overlap. The #231 base survives, because it supplies judgement only and leaves totals and verdicts to the pack review rule instead of restating that rule in Python. Grafted back from the #240 arm lane: `--runs` takes one file per arm, cases the driver could not complete are skipped, and the `--output-format json` envelope carries reviewer cost again. `docs/FEATURES.md` took main's shorter wording, which is the one that fits the size cap. **Three defects the merge exposed, all fixed.** * `--tools` is variadic, so `"--tools", "", prompt` swallowed the prompt and the harness exited with `Input must be provided`. Every review on this branch would have failed. Both seal flags now use the equals form, matching the driver's `--mcp-config` idiom. * `evaluation-record` set the v3 format up front, which skips the branch in `MarshalResult` that stamps the pack digest, so every record it wrote failed validation against its own pack. It now binds the digest explicitly. * The driver records the attempt that worked with an empty `reason`, which provenance validation rejects. The writer names the fact rather than dropping the attempt. `evaluation-record` also filters reviews by arm so a two-arm file cannot cross-join, and fails loudly when a succeeded case has no review instead of writing it as a silent zero. **Verified.** The reviewer ran green against two cases from the committed `evaluations/baseline/claude-frontier-driver.json`, returning valid scores, evidence, and cost. A synthetic full-coverage review then drove `evaluation-record` through all 67 cases and all eight roles, which is what surfaced the digest and provenance defects. Those runs used the host home and are plumbing only, not evidence. `go build`, `go vet`, and `pre-commit run --all-files` are clean. **Still open.** `TestLatestScoredResultsMatchCurrentPacks` fails for all eight packs. Every pack digest changed when melds moved doctrine out of the role bodies, so the recorded codex baseline no longer applies. `main` was red on the same gate before this merge and the gate inputs are byte-identical, so this does not regress it. Re-earning still needs one interactive `/login` per isolated home. Kai is reworking the evaluation subsystem, so the baseline is deliberately not re-earned here.

coilyco-ops closed this pull request

2026-08-07 18:18:55 +00:00

release / release (push) Failing after 14s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Rows
Columns

meld/231-shared-role-melds #246

Pull request closed