evalkit: run through Inspect, pydantic wire types, pattern discriminators #266

Merged

coilysiren merged 2 commits from adjacency/262-role-adjacents into main

2026-08-11 03:27:17 +00:00

coilyco-ops commented

2026-08-11 03:23:42 +00:00

Member

The two commits that landed after the last merge. Follows #263 and its successors.

Without this, main still carries evalkit/run.py, the jsonl response sink, and substring_matcher, so a fresh eval session would get the pre-Inspect pipeline.

Inspect takes the run leg

evalkit/run.py is deleted. evalkit/task.py is an Inspect task: the composed role bundle is the system message, the solver is a single generate, and --epochs replaces the hand-rolled n=5 fan-out. Agent Proxy is reached through the openai-api provider, which reads AGENTPROXY_BASE_URL.

Deferred scoring is native rather than a workaround. Inspect documents --no-score for unscored logs and score editing for "applying manual review adjustments", so a human scorer is a supported shape. inspect view arrives as a browser log surface without hosting anything.

Verified live: two samples at two epochs through ser8 in eleven seconds, then filtered straight out of the .eval log.

Pydantic models the wire types

Sample, Response, DatasetEntry, and Annotation become BaseModel. __post_init__ becomes a model_validator and from_dict collapses into model_validate. Sample.to_inspect and from_inspect cross to Inspect's Sample, whose only fields are input, target, id, and metadata, so the domain fields ride in metadata.

Discriminators become machine-checkable patterns

A discriminator was prose matched by substring, which the code itself called a placeholder because prose cannot be matched that way. It is now a list of regexes: any match is a failure, matching is case-insensitive and multiline, and every pattern compiles at load so a bad regex fails before a 390-call run rather than during one.

That removes the last model from the loop. Item analysis is now fully deterministic, so the same log and the same samples always select the same dataset.

A pattern will miss failures a reader would catch. The cost is bounded, because the filter only decides whether a sample discriminates and the human annotation is the measurement, so a miss produces a slightly worse sample rather than a wrong label.

Consequences worth naming

httpx is gone as a direct dependency. openai arrives, because Inspect's OpenAI-compatible provider needs it.
evalkit becomes an installed package with a hatchling build, because Inspect loads a task by file path and evalkit was not importable that way.

Test state

41 Python tests, mypy --strict, ruff, ruff format, the full pre-commit suite, and go build ./... all green.

TestLatestScoredResultsMatchCurrentPacks still fails for all eight packs, unchanged from main and unrelated to this diff. Re-earning that baseline is tracked at #262.

Refs #262

The two commits that landed after the last merge. Follows https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/pulls/263 and its successors. Without this, `main` still carries `evalkit/run.py`, the jsonl response sink, and `substring_matcher`, so a fresh eval session would get the pre-Inspect pipeline. ## Inspect takes the run leg `evalkit/run.py` is deleted. `evalkit/task.py` is an Inspect task: the composed role bundle is the system message, the solver is a single generate, and `--epochs` replaces the hand-rolled n=5 fan-out. Agent Proxy is reached through the `openai-api` provider, which reads `AGENTPROXY_BASE_URL`. Deferred scoring is native rather than a workaround. Inspect documents `--no-score` for unscored logs and score editing for "applying manual review adjustments", so a human scorer is a supported shape. `inspect view` arrives as a browser log surface without hosting anything. Verified live: two samples at two epochs through ser8 in eleven seconds, then filtered straight out of the `.eval` log. ## Pydantic models the wire types `Sample`, `Response`, `DatasetEntry`, and `Annotation` become `BaseModel`. `__post_init__` becomes a `model_validator` and `from_dict` collapses into `model_validate`. `Sample.to_inspect` and `from_inspect` cross to Inspect's `Sample`, whose only fields are input, target, id, and metadata, so the domain fields ride in metadata. ## Discriminators become machine-checkable patterns A discriminator was prose matched by substring, which the code itself called a placeholder because prose cannot be matched that way. It is now a list of regexes: any match is a failure, matching is case-insensitive and multiline, and every pattern compiles at load so a bad regex fails before a 390-call run rather than during one. That removes the last model from the loop. Item analysis is now fully deterministic, so the same log and the same samples always select the same dataset. A pattern will miss failures a reader would catch. The cost is bounded, because the filter only decides whether a sample discriminates and the human annotation is the measurement, so a miss produces a slightly worse sample rather than a wrong label. ## Consequences worth naming * `httpx` is gone as a direct dependency. `openai` arrives, because Inspect's OpenAI-compatible provider needs it. * `evalkit` becomes an installed package with a hatchling build, because Inspect loads a task by file path and `evalkit` was not importable that way. ## Test state 41 Python tests, `mypy --strict`, ruff, ruff format, the full pre-commit suite, and `go build ./...` all green. `TestLatestScoredResultsMatchCurrentPacks` still fails for all eight packs, unchanged from `main` and unrelated to this diff. Re-earning that baseline is tracked at https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/issues/262. Refs https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/issues/262

coilyco-ops added 2 commits

2026-08-11 03:23:42 +00:00

feat(evalkit): run through Inspect and model the wire types with pydantic dc426cec6f

Two dependencies replace hand-rolled code rather than adding surface.

Inspect takes the run leg. evalkit/task.py is an Inspect task where the
composed role bundle is the system message and the solver is a single
generate. --epochs replaces the n=5 fan-out, the .eval log replaces a
jsonl sink, and evalkit/run.py is deleted. Agent Proxy is reached through
the openai-api provider, which reads AGENTPROXY_BASE_URL. Verified live:
two samples at two epochs through ser8 in eleven seconds, then filtered
straight out of the log.

Deferred scoring turned out to be native rather than a workaround.
Inspect documents --no-score for unscored logs and score editing for
"applying manual review adjustments", so a human scorer is a supported
shape rather than a detour. inspect view comes along free, which is a
browser surface without hosting one.

Pydantic models the wire types. Sample, Response, DatasetEntry, and
Annotation become BaseModel, __post_init__ becomes a model_validator,
and from_dict collapses into model_validate. Sample.to_inspect and
from_inspect cross to Inspect's Sample, whose only fields are input,
target, id, and metadata, so the domain fields ride in metadata.

Two consequences worth naming. httpx is gone as a direct dependency, and
openai arrives because Inspect's OpenAI-compatible provider needs it.
evalkit becomes an installed package with a hatchling build, because
Inspect loads a task by file path and evalkit was not importable that
way.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

feat(evalkit): make discriminators machine-checkable patterns e6bbfb9260

A discriminator was prose describing the failing behaviour, matched by
substring, which the code called a placeholder because prose cannot be
matched that way. It is now a list of regexes. Any match is a failure,
matching is case-insensitive and multiline, and every pattern compiles at
load so a bad regex fails before a 390-call run rather than during one.

This removes the last model from the loop. Item analysis is now fully
deterministic: the same log and the same samples always select the same
dataset.

A pattern will miss failures a reader would catch. That cost is bounded,
because the filter only decides whether a sample discriminates and the
human annotation is the measurement, so a miss produces a slightly worse
sample rather than a wrong label.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-11 03:23:42 +00:00