feat(evals): commit the personality grading anchors before grading begins (#262) #301

Merged

coilysiren merged 1 commit from aos/claude/personality-anchors into main

2026-08-17 18:30:35 +00:00

coilyco-ops commented

2026-08-17 18:26:53 +00:00

Member

Closes the "write the personality anchors before grading" item from the board specification comment on #262. Does not touch the board, the pipeline, or any case.

Why now

The personality tier is the only tier with no mechanical filter. Item analysis drops boundary and role-fit candidates that do not discriminate, but personality cases are kept unconditionally, which is why both ops runs surfaced three personality survivors and nothing else. With no filter, these anchors are the only thing holding the scale still across 24 cases in one sitting.

#262 also requires them to be observable behaviours rather than adjectives. "Was the response bold" is unanswerable. "Did it put a larger-scope option on the table and cost it" is answerable from the text.

Shape

18 personalities, each with three fields:

fit - the behaviour that must be present in the response
deduct - what earns a deduction
distinguish - the neighbour the trait must not collapse into

The distinguish field is the part worth arguing about. The separations are the ones the personality bodies already assert about themselves after #296, where I wrote explicit cross-references into each body ("You are not the one who widens the question"). Reusing them means a response that satisfies the neighbour instead is a real miss against what the bundle says, rather than a grader preference invented at grading time.

Two universal deduction patterns

Named explicitly rather than folded into a general impression, because both are the defect #296 targeted:

default-register - competent, balanced, no trait signal, reads as the model's own default voice rather than the composed one. This is precisely the outcome I argued the shared prose template was producing. It is the load-bearing observation of the whole tier, so it is does not fit rather than undecided.
named-not-enacted - the response names the trait instead of performing it. Naming a register rather than demonstrating one is exactly what the retired template did, so a subject doing it is reproducing the defect one layer down.

If the personality tier comes back clean on both patterns, the prose rewrite worked. If it comes back full of default-register, three merged PRs moved the spec and not the behaviour. Either result is worth having, which is the point.

Guarded, not asserted

TestEveryRosterPersonalityHasAGradingAnchor fails in both directions: a melded personality with no anchor, and an anchor matching no meld. I verified it catches a removed anchor rather than trusting it, by deleting empirical and confirming the failure names the role and trait.

This matters because the board is derived from the roster. A future personality swap moves the case list on its own, and without the guard it would silently leave a case ungradeable.

Also confirmed

The derived board already tracks #299 with no intervention: it emits exec-per-outward and ai-per-empirical. The 18 anchors cover its 18 traits exactly, checked mechanically in both directions.

Worth recording that I first ran ward evalkit-matrix from the canonical checkout, which sits behind origin/main, and briefly read the stale curious output as roster drift. It was not. Dev verbs also live in the root justfile now rather than .ward/ward.yaml, per inbox#366.

Verification

go build ./..., go test ./..., gofmt -l, and pre-commit run --all-files all pass.

Not in this PR

No cases authored. Per the ops grading comment, the absent-authority question and the operational-tuning-versus-product-logic question both need settling before more boundary and role-fit cases are worth writing. Neither blocks the personality tier, which is why this piece could move.

Closes the "write the personality anchors before grading" item from the board specification comment on #262. Does not touch the board, the pipeline, or any case. ## Why now The personality tier is the only tier with no mechanical filter. Item analysis drops boundary and role-fit candidates that do not discriminate, but personality cases are kept unconditionally, which is why both ops runs surfaced three personality survivors and nothing else. With no filter, these anchors are the only thing holding the scale still across 24 cases in one sitting. #262 also requires them to be observable behaviours rather than adjectives. "Was the response bold" is unanswerable. "Did it put a larger-scope option on the table and cost it" is answerable from the text. ## Shape 18 personalities, each with three fields: * `fit` - the behaviour that must be present in the response * `deduct` - what earns a deduction * `distinguish` - the neighbour the trait must not collapse into The `distinguish` field is the part worth arguing about. The separations are the ones the personality bodies already assert about themselves after #296, where I wrote explicit cross-references into each body ("You are not the one who widens the question"). Reusing them means a response that satisfies the neighbour instead is a real miss against what the bundle says, rather than a grader preference invented at grading time. ## Two universal deduction patterns Named explicitly rather than folded into a general impression, because both are the defect #296 targeted: * **default-register** - competent, balanced, no trait signal, reads as the model's own default voice rather than the composed one. This is precisely the outcome I argued the shared prose template was producing. It is the load-bearing observation of the whole tier, so it is `does not fit` rather than `undecided`. * **named-not-enacted** - the response names the trait instead of performing it. Naming a register rather than demonstrating one is exactly what the retired template did, so a subject doing it is reproducing the defect one layer down. If the personality tier comes back clean on both patterns, the prose rewrite worked. If it comes back full of default-register, three merged PRs moved the spec and not the behaviour. Either result is worth having, which is the point. ## Guarded, not asserted `TestEveryRosterPersonalityHasAGradingAnchor` fails in both directions: a melded personality with no anchor, and an anchor matching no meld. I verified it catches a removed anchor rather than trusting it, by deleting `empirical` and confirming the failure names the role and trait. This matters because the board is derived from the roster. A future personality swap moves the case list on its own, and without the guard it would silently leave a case ungradeable. ## Also confirmed The derived board already tracks #299 with no intervention: it emits `exec-per-outward` and `ai-per-empirical`. The 18 anchors cover its 18 traits exactly, checked mechanically in both directions. Worth recording that I first ran `ward evalkit-matrix` from the canonical checkout, which sits behind `origin/main`, and briefly read the stale `curious` output as roster drift. It was not. Dev verbs also live in the root `justfile` now rather than `.ward/ward.yaml`, per inbox#366. ## Verification `go build ./...`, `go test ./...`, `gofmt -l`, and `pre-commit run --all-files` all pass. ## Not in this PR No cases authored. Per the ops grading comment, the absent-authority question and the operational-tuning-versus-product-logic question both need settling before more boundary and role-fit cases are worth writing. Neither blocks the personality tier, which is why this piece could move.

coilyco-ops added 1 commit

2026-08-17 18:26:53 +00:00

feat(evals): commit the personality grading anchors before grading begins (#262 )

ci / test (pull_request) Successful in 57s

Details

2c333f63bb

The personality tier scores fit / undecided / does not fit and bypasses item
analysis, so it has no mechanical filter. #262 requires the anchors to exist
before grading and to be observable behaviours rather than adjectives. They did
not exist.

Anchors for all 18 personalities, each recording what fit looks like in a
response, what earns a deduction, and the neighbour the trait must not collapse
into. The separations are the ones the personality bodies assert about
themselves after #296, so a response that satisfies the neighbour instead is a
real miss rather than a grader preference.

Two deduction patterns apply to every trait and are named rather than left to
general impression:

* default-register - competent, balanced, no trait signal, reads as the model's
  own voice rather than the composed one. This is the exact outcome #296 argued
  the shared prose template was producing, so it is the tier's load-bearing
  observation and belongs as does not fit rather than undecided.
* named-not-enacted - the response names the trait instead of performing it.
  Naming a register rather than demonstrating one is what the retired template
  did, so a subject doing it is reproducing the defect.

Guarded rather than asserted. TestEveryRosterPersonalityHasAGradingAnchor fails
both ways: a melded personality with no anchor, and an anchor matching no meld.
Verified it catches a removed anchor rather than trusting it. The board is
derived from the roster, so a future personality swap moves the case list on its
own and would otherwise leave a case ungradeable.

Confirmed the derived board already tracks #299: it emits exec-per-outward and
ai-per-empirical, and the 18 anchors cover its 18 traits exactly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>