coilyco-flight-deck/agent-compose

Watch

Role adjacency in KDL, and evalkit as the Python half of the eval system #263

Merged

coilysiren merged 8 commits from adjacency/262-role-adjacents into main

2026-08-10 19:43:07 +00:00

coilysiren commented

2026-08-10 18:01:26 +00:00

Owner

Eight commits in three groups. Refs #262 and #213.

Read this first: the failing test is pre-existing

TestLatestScoredResultsMatchCurrentPacks fails for all eight packs on this branch. It also fails on clean main at 67fd5bc with none of these commits applied.

I diffed the expected-versus-computed digest pairs between clean main and this branch: identical, all eight. Nothing here moves a pack digest. Adding Adjacents to the Role struct does not alter rendered pack content.

The baseline was retired by work that landed before this branch, all of it changing pack content:

d09e97d and fb94739, the meld-to-boundary rename and renaming boundaries for the behavior they remove, both breaking
09c9ec0, 43739fd, 9f8730d, 67518b7, freeform personality bodies, flat data layout, prose bounds, second-person rewrite

Re-earning the baseline is #262's job and is tracked there. This PR neither fixes nor worsens it.

Everything else is green: go build ./..., go vet ./..., every other Go package, 26 Python tests, ruff, ruff format, mypy --strict, and the full pre-commit suite.

1. Adjacency becomes a role property (`cdd293c`)

Adjacency lived in internal/person/data/role-*/evals.yaml as adjacent_role, so it existed only where someone had written a case. The graph was incomplete: five of thirteen edges were one-directional, and QA declared none while being named by AI.

It moves onto the role as a KDL node with a required reason, plus validateRoleAdjacents in the load chain, docs/role-adjacency.md, and four tests.

engineer -> qa, ai          director -> exec, engineer
qa       -> engineer, ai    exec     -> director, creator
ops      -> engineer, director  design -> creator, engineer
ai       -> engineer, qa    creator  -> design, exec

Directed on purpose, so do not add a symmetry check. Absorption risk is not symmetric: DevOps sequencing follow-up work after an incident is a live confusion, while the Director rarely reaches for a runbook. Out-degree is fixed at two so the roster picks its sharpest confusions instead of listing every plausible neighbour. A roster declaring no adjacency at all is skipped, so external person packages keep loading.

The reason is generator input rather than commentary. An adjacency case has to construct one specific confusion, and a bare pair leads a generator to invent the wrong one on exactly the non-obvious edges.

2. evalkit, the Python orchestration half (`9d22c27` through `533dbe8`)

The repo already ran its eval driver and reviewer in Python, but unmanaged: no project table, no declared dependencies, no lint, no types, no tests, no ward verbs. This promotes it to a real package pointed at the new board.

evalkit.schema - pipeline data model
evalkit.run - subject fan-out at n=5, transport recorded per response so a direct-provider fallback cannot be mistaken for a measured result
evalkit.filter - item analysis, drops candidates that pass or fail every run, reports every drop
evalkit.grade - one case per screen, one keystroke per decision, saves after every grade, reports boundary pairs rather than halves
evalkit.board - derives the case list from the roster Go exports

The seam: Go owns what a pack is and what a valid record is. Python owns running the subject, filtering, and putting a case in front of a human. Python consumes what Go emits and never restates it, because two parsers is the failure this split exists to avoid. Grades stay a dumb intermediate so the canonical record still comes from the owning marshaller.

The board is derived rather than maintained. Boundaries and owners produce the pairs, adjacency produces the role-fit targets, each role's meld produces the personality cases. 78 cases today. ward evalkit-board prints it.

Gates run from scripts/evalkit-check.sh rather than pre-commit, because that file is managed by agentic-os between BEGIN and END markers and a hand-added hook would be overwritten on the next sync. CI's agentic-os:release image ships uv, so ward exec test resolves.

3. Reference pages (`3aa9faf`)

Six external reference points, two papers, two platforms, two blogs, each with a page and an index at docs/eval-references.md. Records two things a reader would otherwise trip on: the Pragmatic Engineer guide is paywalled after section 3, and the Phoenix llm-evals page is the wrong citation for this work since it covers LLM-as-judge rather than human annotation.

Notes that RULERS' three recommendations were already in place here independently, so nobody reads the design as having followed the paper.

Deliberately not in this PR

The stale adjacent_role entries in role-*/evals.yaml. Removing them is a fork with real costs, discussed on #262, and that matrix retires anyway.
Any eval platform adoption. Both platforms are referenced, neither adopted, and #213 still owns that decision.
substring_matcher is a placeholder. A prose discriminator cannot be matched reliably by substring, and the two real options are documented in docs/eval-orchestration.md.

Size note

2782 insertions across 37 files, 520 of them generated uv.lock.

Eight commits in three groups. Refs #262 and #213. ## Read this first: the failing test is pre-existing `TestLatestScoredResultsMatchCurrentPacks` fails for all eight packs on this branch. **It also fails on clean `main` at `67fd5bc` with none of these commits applied.** I diffed the expected-versus-computed digest pairs between clean main and this branch: identical, all eight. Nothing here moves a pack digest. Adding `Adjacents` to the `Role` struct does not alter rendered pack content. The baseline was retired by work that landed before this branch, all of it changing pack content: * `d09e97d` and `fb94739`, the meld-to-boundary rename and renaming boundaries for the behavior they remove, both breaking * `09c9ec0`, `43739fd`, `9f8730d`, `67518b7`, freeform personality bodies, flat data layout, prose bounds, second-person rewrite Re-earning the baseline is #262's job and is tracked there. This PR neither fixes nor worsens it. Everything else is green: `go build ./...`, `go vet ./...`, every other Go package, 26 Python tests, ruff, ruff format, `mypy --strict`, and the full pre-commit suite. ## 1. Adjacency becomes a role property (`cdd293c`) Adjacency lived in `internal/person/data/role-*/evals.yaml` as `adjacent_role`, so it existed only where someone had written a case. The graph was incomplete: five of thirteen edges were one-directional, and QA declared none while being named by AI. It moves onto the role as a KDL node with a required `reason`, plus `validateRoleAdjacents` in the load chain, `docs/role-adjacency.md`, and four tests. ``` engineer -> qa, ai director -> exec, engineer qa -> engineer, ai exec -> director, creator ops -> engineer, director design -> creator, engineer ai -> engineer, qa creator -> design, exec ``` **Directed on purpose, so do not add a symmetry check.** Absorption risk is not symmetric: DevOps sequencing follow-up work after an incident is a live confusion, while the Director rarely reaches for a runbook. Out-degree is fixed at two so the roster picks its sharpest confusions instead of listing every plausible neighbour. A roster declaring no adjacency at all is skipped, so external person packages keep loading. The `reason` is generator input rather than commentary. An adjacency case has to construct one specific confusion, and a bare pair leads a generator to invent the wrong one on exactly the non-obvious edges. ## 2. evalkit, the Python orchestration half (`9d22c27` through `533dbe8`) The repo already ran its eval driver and reviewer in Python, but unmanaged: no project table, no declared dependencies, no lint, no types, no tests, no ward verbs. This promotes it to a real package pointed at the new board. * `evalkit.schema` - pipeline data model * `evalkit.run` - subject fan-out at n=5, transport recorded per response so a direct-provider fallback cannot be mistaken for a measured result * `evalkit.filter` - item analysis, drops candidates that pass or fail every run, reports every drop * `evalkit.grade` - one case per screen, one keystroke per decision, saves after every grade, reports boundary pairs rather than halves * `evalkit.board` - derives the case list from the roster Go exports **The seam:** Go owns what a pack is and what a valid record is. Python owns running the subject, filtering, and putting a case in front of a human. Python consumes what Go emits and never restates it, because two parsers is the failure this split exists to avoid. Grades stay a dumb intermediate so the canonical record still comes from the owning marshaller. The board is derived rather than maintained. Boundaries and owners produce the pairs, adjacency produces the role-fit targets, each role's meld produces the personality cases. 78 cases today. `ward evalkit-board` prints it. Gates run from `scripts/evalkit-check.sh` rather than pre-commit, because that file is managed by agentic-os between BEGIN and END markers and a hand-added hook would be overwritten on the next sync. CI's `agentic-os:release` image ships `uv`, so `ward exec test` resolves. ## 3. Reference pages (`3aa9faf`) Six external reference points, two papers, two platforms, two blogs, each with a page and an index at `docs/eval-references.md`. Records two things a reader would otherwise trip on: the Pragmatic Engineer guide is paywalled after section 3, and the Phoenix `llm-evals` page is the wrong citation for this work since it covers LLM-as-judge rather than human annotation. Notes that RULERS' three recommendations were already in place here independently, so nobody reads the design as having followed the paper. ## Deliberately not in this PR * The stale `adjacent_role` entries in `role-*/evals.yaml`. Removing them is a fork with real costs, discussed on #262, and that matrix retires anyway. * Any eval platform adoption. Both platforms are referenced, neither adopted, and #213 still owns that decision. * `substring_matcher` is a placeholder. A prose discriminator cannot be matched reliably by substring, and the two real options are documented in `docs/eval-orchestration.md`. ## Size note 2782 insertions across 37 files, 520 of them generated `uv.lock`.

coilysiren added 8 commits

2026-08-10 18:01:26 +00:00

feat(person): declare role adjacency in KDL with a fixed out-degree cdd293cc55

Adjacency was a property of an evaluation scenario, declared as
adjacent_role in internal/person/data/role-*/evals.yaml. It existed only
where someone had written a case, so the graph was incomplete: five of
thirteen edges were one-directional and QA declared none at all while
being named by AI.

Move it onto the role as a first-class KDL node. Each edge carries a
reason, which is generator input rather than commentary, because an
adjacency case has to construct one specific confusion and a bare pair
leads the generator to invent the wrong one.

The graph is deliberately directed. Absorption risk is not symmetric:
DevOps sequencing follow-up work after an incident is a live confusion
while the Director rarely reaches for a runbook. Out-degree is fixed at
two so the roster picks its sharpest confusions instead of listing every
plausible neighbour. A roster declaring no adjacency at all is left
alone, so external person packages keep loading.

TestLatestScoredResultsMatchCurrentPacks now fails for all eight packs.
That is the digest contract working: adding adjacency to the role KDL
moves every pack digest and retires the committed baseline. The records
are re-earned under the triple in #262, so this branch does not land on
main until that happens.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

feat(evalkit): add the Python orchestration half of the eval system 9d22c27719

The repo already ran its eval driver and reviewer in Python, but that
Python was unmanaged: no project table, no declared dependencies, no
lint, no type checking, no tests, and no ward verbs. This promotes it to
a real package and points it at the new board rather than the retired
one.

Go keeps what a pack is and what a valid record is. Python takes running
the subject, filtering candidates, and putting a case in front of a
human. The rule holding that seam is that Python consumes what Go emits
and never restates it, because two parsers is the failure the split
exists to avoid. Grades stay a dumb intermediate so the canonical record
still comes from the owning marshaller.

evalkit.run fans every candidate out at n=5 and records transport per
response, so a direct-provider fallback can never be mistaken for a
measured result. evalkit.filter drops candidates that pass or fail every
run, picks the survivor closest to the midpoint, and reports every drop
because silent truncation reads as full coverage. evalkit.grade is one
case per screen and one keystroke per decision, saving after every
grade, and it reports boundary pairs rather than halves because the pair
is the scoring unit.

Gates run from scripts/evalkit-check.sh rather than pre-commit. That
file is managed by agentic-os between its BEGIN and END markers, so a
hand-added hook there would be overwritten on the next sync.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

feat(evalkit): derive the eval case list from the roster 470fab29d8

The 86-case board was a hand-maintained list living in a chat log and an
issue comment. It is really a consequence of the roster: boundaries and
their owners decide the boundary pairs, adjacency decides the role-fit
targets, and each role's meld decides the personality cases.

evalkit.board derives it instead. Adding a boundary, changing an
adjacency edge, or swapping a personality changes the printed board on
its own, so the list cannot drift from the roster that produces it.

Go exports the roster and Python derives the board, keeping the seam the
package already documents. There is no KDL parser here, and adjacency
reaches the output through Role's json tags because SnapshotRole embeds
Role so new fields export automatically.

Adjacency reasons become the role-fit case descriptors directly, which
is the same text the generator needs to construct the right confusion.

scripts/eval-board.sh renders the roster to a temp dir and prints. It
does not override HOME, because doing so relocates the Go module cache
into that directory and the read-only cache then defeats cleanup.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

feat(evalkit): interpolate roster context into boundary case descriptors 5dd70829cc

Thirty boundary lines all read "must own its own side" or "must defer to
<owner>", which carried no more information than the id already did. The
roster has better material sitting unused: each boundary summary names
the owner's behaviour, and each role names its purpose.

An owning half now names the role's purpose and a deferring half names
the behaviour it hands over, so the deferring lines differ per boundary
and the owning lines differ per role.

The owner clause is quoted verbatim rather than negated into prose. It is
conjugated for the owner's display name, so "recommends communication"
becomes "must not recommends communication" under naive negation, and
fixing that would put English inflection rules in a board renderer.

A summary without the expected two-clause shape passes through intact
rather than being truncated at a comma that is not there.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

feat(evalkit): drop the composed-meld personality case 45e9bfed10

Personality goes from four cases per role to three, one per trait, so
the board falls from 86 to 78 and the personality tier from 32 to 24.

Nothing about composition is lost from execution. Every case on the board
already runs against the fully composed bundle with role, boundaries, and
all three personalities applied, so a trait case observes that trait as
it actually composes. What goes away is a separate score for whether the
three blend, which the descriptor now names instead: each trait case says
which peers it sits alongside.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

feat(evalkit): state what a within-role case tests, not whose work it is 54c1df8f05

"qa's own work" named the subject rather than the behaviour under test.
The adjacent cases in the same tier all name a failure mode, so the
within case was the only line a reader could not act on.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

feat(evalkit): group grading by role and print the charter once per group 533dbe816e

Grading now orders role-major and --roster prints purpose, owned and
deferred boundaries, adjacency reasons, and personalities at the top of
each role's group. --role grades a subset. evalkit.board groups the same
way by default, with --group tier for the old view.

Role context is the expensive thing for a grader to reload, and holding
one charter across a role's ten to twelve cases beats reloading it every
case. Kind-major degraded more gracefully, leaving every role partly
scored on an early stop, but grading saves after every decision and is
resumable, so speed wins.

Execution stays one case per session. Batching a role's cases into one
request would save background machine time and no human time, while
manufacturing the reflexive deferral the in-out pair exists to catch,
breaking the per-case independence n=5 assumes, and adding order effects.
Whether doctrine survives accumulated context is a real question, but it
is a second arm rather than a cheaper version of this one.

Splits the grading rules out to docs/eval-grading.md, since the doc had
grown past its cap by carrying both the pipeline and the scoring rules.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

docs: add reference pages for the six external eval reference points 3aa9faf294

Two papers, two platforms, two blogs, each with its own page and an
index at docs/eval-references.md. Every page states what the source
says and how it relates to this system, so a reader arriving from any
of the three directions can place the work.

The pairs are deliberate. CheckList names the board's shape and RULERS
prescribes rather than warns. Phoenix covers the grading half and
Inspect the execution half, both referenced rather than adopted. The two
blogs reach the working engineer who opens neither a paper nor a docs
site.

Records two things a reader would otherwise trip on. The Pragmatic
Engineer guide is paywalled after section 3, though the free portion
carries the parts cited. The Phoenix llm-evals page is explicitly not
the right citation for this work, since it covers code-based and
LLM-as-judge evaluators rather than human annotation.

Notes that RULERS' three recommendations were already in place here
independently, so a reader does not assume the design followed the
paper.

Refs #262, #213.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops changed title from ~~adjacency/262-role-adjacents~~ to Role adjacency in KDL, and evalkit as the Python half of the eval system

2026-08-10 19:36:31 +00:00

coilysiren merged commit 922469fb2c into main

2026-08-10 19:43:07 +00:00

coilysiren referenced this pull request from a commit

2026-08-10 19:43:08 +00:00

Merge pull request 'Role adjacency in KDL, and evalkit as the Python half of the eval system' (#263) from adjacency/262-role-adjacents into main

coilyco-ops referenced this pull request

2026-08-11 03:23:42 +00:00

evalkit: run through Inspect, pydantic wire types, pattern discriminators #266