refactor(evalkit): adopt the reference vocabulary and add axial coding #265

Merged

coilysiren merged 1 commit from adjacency/262-role-adjacents into main

2026-08-11 00:10:45 +00:00

coilysiren commented

2026-08-11 00:10:29 +00:00

Owner

Terminology now follows the six references wherever they have a word for
something, so a reader arriving from any of them recognises the shape.

From Inspect: sample, dataset, and target, where target is defined as
"the ideal answer or grading guidance" and matches the old expected
exactly. From CheckList: test type, its column axis, replacing kind.
From Phoenix and Hamel: annotation, label, and critique, replacing grade,
verdict, and note. evalkit.board becomes evalkit.matrix, since what it
derives is CheckList's capability-by-test-type matrix, and that frees
dataset for the filtered artifact. evalkit.grade becomes
evalkit.annotate.

The role-fit target field became against, because Inspect's target
claimed the name. It holds "within" or the adjacent role whose work may
be absorbed.

Three practices adopted alongside the words:

Axial coding. Both practitioner references describe error analysis as
open coding, then axial coding, then a taxonomy, and only the first
existed here. evalkit.taxonomy groups deductions by structural axis and
shared critique terms, then ranks by frequency, so the output is a list
of failure modes rather than a score. Undecided counts as a deduction, so
a cluster surfaces bad samples rather than bad roles.

Evidence anchoring. A deduction now records a verbatim span from the
output, verified against it before acceptance, which is RULERS' rule for
making a judgment auditable rather than impressionistic.

Configured label sets. The keystroke-to-label map moves out of the
annotate loop into schema.LABEL_SETS, following Phoenix's treatment of an
annotation rubric as data.

TestType trips pytest's class collection, so python_classes is emptied
rather than surrendering CheckList's word.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: Kai Siren coilysiren@gmail.com
Co-authored-by: Claude noreply@anthropic.com

Terminology now follows the six references wherever they have a word for something, so a reader arriving from any of them recognises the shape. From Inspect: sample, dataset, and target, where target is defined as "the ideal answer or grading guidance" and matches the old `expected` exactly. From CheckList: test type, its column axis, replacing `kind`. From Phoenix and Hamel: annotation, label, and critique, replacing grade, verdict, and note. evalkit.board becomes evalkit.matrix, since what it derives is CheckList's capability-by-test-type matrix, and that frees `dataset` for the filtered artifact. evalkit.grade becomes evalkit.annotate. The role-fit `target` field became `against`, because Inspect's `target` claimed the name. It holds "within" or the adjacent role whose work may be absorbed. Three practices adopted alongside the words: Axial coding. Both practitioner references describe error analysis as open coding, then axial coding, then a taxonomy, and only the first existed here. evalkit.taxonomy groups deductions by structural axis and shared critique terms, then ranks by frequency, so the output is a list of failure modes rather than a score. Undecided counts as a deduction, so a cluster surfaces bad samples rather than bad roles. Evidence anchoring. A deduction now records a verbatim span from the output, verified against it before acceptance, which is RULERS' rule for making a judgment auditable rather than impressionistic. Configured label sets. The keystroke-to-label map moves out of the annotate loop into schema.LABEL_SETS, following Phoenix's treatment of an annotation rubric as data. TestType trips pytest's class collection, so python_classes is emptied rather than surrendering CheckList's word. Refs #262. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Kai Siren <coilysiren@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

coilysiren added 1 commit

2026-08-11 00:10:29 +00:00

refactor(evalkit): adopt the reference vocabulary and add axial coding 84ede51c4a

Terminology now follows the six references wherever they have a word for
something, so a reader arriving from any of them recognises the shape.

From Inspect: sample, dataset, and target, where target is defined as
"the ideal answer or grading guidance" and matches the old `expected`
exactly. From CheckList: test type, its column axis, replacing `kind`.
From Phoenix and Hamel: annotation, label, and critique, replacing grade,
verdict, and note. evalkit.board becomes evalkit.matrix, since what it
derives is CheckList's capability-by-test-type matrix, and that frees
`dataset` for the filtered artifact. evalkit.grade becomes
evalkit.annotate.

The role-fit `target` field became `against`, because Inspect's `target`
claimed the name. It holds "within" or the adjacent role whose work may
be absorbed.

Three practices adopted alongside the words:

Axial coding. Both practitioner references describe error analysis as
open coding, then axial coding, then a taxonomy, and only the first
existed here. evalkit.taxonomy groups deductions by structural axis and
shared critique terms, then ranks by frequency, so the output is a list
of failure modes rather than a score. Undecided counts as a deduction, so
a cluster surfaces bad samples rather than bad roles.

Evidence anchoring. A deduction now records a verbatim span from the
output, verified against it before acceptance, which is RULERS' rule for
making a judgment auditable rather than impressionistic.

Configured label sets. The keystroke-to-label map moves out of the
annotate loop into schema.LABEL_SETS, following Phoenix's treatment of an
annotation rubric as data.

TestType trips pytest's class collection, so python_classes is emptied
rather than surrendering CheckList's word.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilysiren merged commit 4035f10cf7 into main

2026-08-11 00:10:45 +00:00

coilysiren referenced this pull request from a commit

2026-08-11 00:10:47 +00:00

Merge pull request 'refactor(evalkit): adopt the reference vocabulary and add axial coding' (#265) from adjacency/262-role-adjacents into main

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-compose!265

No description provided.

Rows
Columns