coilyco-flight-deck/agent-compose

Watch

evalkit derives the board from the data, with no Go in the path #338

New issue

Open

opened 2026-08-25 17:09:13 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-25 17:09:13 +00:00

Member

Parent

#329

What to build

Remove the Go binary from the evaluation pipeline. This is the slice where the inversion in #329 is actually achieved rather than merely enabled.

Today both eval entry points open by shelling out:

scripts/eval-matrix.sh runs go run ./cmd/agent-compose roster --out "$render_dir", reads person.json from a mktemp -d, then deletes the directory.
scripts/eval-prompts.sh runs go run ./cmd/agent-compose roster and then compose once per role, extracting each compiled.md.

After this slice, evalkit.matrix reads the YAML roster directly and evalkit composes the per-role system prompts through the compositor. No go run, no temp-directory round trip, no person.json intermediary on the eval path.

The rule in docs/evaluation.md that "Python consumes what Go emits and never restates it" was written under the old direction and has to be restated. It becomes a statement about the eval pack schema and coverage rules, which still have exactly one home, rather than about roster composition, which has moved.

Check whether person.json retains any consumer once the eval path stops reading it. If its only remaining readers are launch-side and machine-to-machine, it stays JSON. If a human reads it as a debugging surface, it becomes YAML under #329 decision 2.

Acceptance criteria

scripts/eval-matrix.sh and scripts/eval-prompts.sh contain no go run and no Go dependency.
just evalkit-matrix and just evalkit-prompts work on a machine with no Go toolchain installed.
The derived board is unchanged from before this slice, case for case.
Composed per-role prompts are byte-identical to what the Go path produced.
docs/evaluation.md restates the one-parser rule in terms of the pack schema rather than roster composition.
person.json's remaining consumers are enumerated, and its format is confirmed or changed on that evidence.

Blocked by

Blocked by #335

## Parent #329 ## What to build Remove the Go binary from the evaluation pipeline. This is the slice where the inversion in #329 is actually achieved rather than merely enabled. Today both eval entry points open by shelling out: * `scripts/eval-matrix.sh` runs `go run ./cmd/agent-compose roster --out "$render_dir"`, reads `person.json` from a `mktemp -d`, then deletes the directory. * `scripts/eval-prompts.sh` runs `go run ./cmd/agent-compose roster` and then `compose` once per role, extracting each `compiled.md`. After this slice, `evalkit.matrix` reads the YAML roster directly and `evalkit` composes the per-role system prompts through the compositor. No `go run`, no temp-directory round trip, no `person.json` intermediary on the eval path. The rule in `docs/evaluation.md` that "Python consumes what Go emits and never restates it" was written under the old direction and has to be restated. It becomes a statement about the eval pack schema and coverage rules, which still have exactly one home, rather than about roster composition, which has moved. Check whether `person.json` retains any consumer once the eval path stops reading it. If its only remaining readers are launch-side and machine-to-machine, it stays JSON. If a human reads it as a debugging surface, it becomes YAML under #329 decision 2. ## Acceptance criteria - [ ] `scripts/eval-matrix.sh` and `scripts/eval-prompts.sh` contain no `go run` and no Go dependency. - [ ] `just evalkit-matrix` and `just evalkit-prompts` work on a machine with no Go toolchain installed. - [ ] The derived board is unchanged from before this slice, case for case. - [ ] Composed per-role prompts are byte-identical to what the Go path produced. - [ ] `docs/evaluation.md` restates the one-parser rule in terms of the pack schema rather than roster composition. - [ ] `person.json`'s remaining consumers are enumerated, and its format is confirmed or changed on that evidence. ## Blocked by - Blocked by #335

coilyco-ops referenced this issue

2026-08-25 17:09:31 +00:00

Delete the Go semantic layer and drop the KDL dependency #339

coilyco-ops referenced this issue

2026-08-25 17:09:40 +00:00

Coverage check: report unauthored and ungraded cases, with a switch to make it blocking #340

coilyco-ops referenced this issue

2026-08-25 17:10:24 +00:00

Invert the stack: data, compositor, bundle, then Go. Drop KDL, move composition out of the engine, mount the roster #329

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-25 17:13:48 +00:00

coilyco-ops referenced this issue

2026-08-25 17:14:40 +00:00

Invert the stack: data, compositor, bundle, then Go. Drop KDL, move composition out of the engine, mount the roster #329

coilyco-ops referenced this issue

2026-08-26 01:45:30 +00:00

Coverage check: report unauthored and ungraded cases, with a switch to make it blocking #340

coilyco-ops referenced this issue

2026-08-26 02:45:37 +00:00

Eval prep: the measured gap on the seven-seat board, and every eval, prose, and content ticket folded into one ordering #357

coilyco-ops referenced this issue

2026-08-27 03:25:25 +00:00

Audience-as-grader segment: the hands-on reopen created it and nothing tracks it #367

coilyco-ops commented

2026-08-27 11:16:23 +00:00

Author

Member

Go is already out of the eval path, as a side effect of the #337 move

Not closing this: what landed is the mechanical half, and the acceptance criteria here are about where the board comes from, not only about which binary runs.

evalkit moved to housecast, where no Go toolchain exists, so the three scripts that shelled out to go run ./cmd/agent-compose could not survive the move unchanged:

eval-prompts.sh composed one compiled bundle per role through Go. It calls python -m housecast compose --delivery compiled now.
eval-matrix.sh and eval-annotate.sh exported the roster through agent-compose roster. They call python -m housecast roster now.

housecast gained housecast/snapshot.py, which emits the same person.json shape evalkit already read. That kept evalkit itself untouched, which is what made this a swap rather than a rewrite.

Evidence it is the same board, not a similar one. Feeding evalkit.roster.to_entity_roster the Go snapshot and the housecast snapshot produces identical output, and evalkit.matrix prints an identical 94-line case list from either. Both were diffed rather than eyeballed.

housecast CI runs just evalkit-matrix as a step, so the board deriving from the roster with no Go in the path is now gated rather than incidental.

What this issue still owns

The snapshot is a JSON handoff in the person.json shape, which is a compatibility shim rather than the board reading the roster directly. housecast/snapshot.py says so in its own docstring and points here. If the intent is that evalkit consume housecast.roster.Roster and drop the JSON hop entirely, that is still to do and this is where it lives.

Filed from the Agentic Platform Engineer seat while landing #337.

## Go is already out of the eval path, as a side effect of the #337 move Not closing this: what landed is the mechanical half, and the acceptance criteria here are about where the board comes from, not only about which binary runs. evalkit moved to housecast, where no Go toolchain exists, so the three scripts that shelled out to `go run ./cmd/agent-compose` could not survive the move unchanged: * `eval-prompts.sh` composed one compiled bundle per role through Go. It calls `python -m housecast compose --delivery compiled` now. * `eval-matrix.sh` and `eval-annotate.sh` exported the roster through `agent-compose roster`. They call `python -m housecast roster` now. housecast gained `housecast/snapshot.py`, which emits the same `person.json` shape evalkit already read. That kept evalkit itself untouched, which is what made this a swap rather than a rewrite. **Evidence it is the same board, not a similar one.** Feeding `evalkit.roster.to_entity_roster` the Go snapshot and the housecast snapshot produces identical output, and `evalkit.matrix` prints an identical 94-line case list from either. Both were diffed rather than eyeballed. housecast CI runs `just evalkit-matrix` as a step, so the board deriving from the roster with no Go in the path is now gated rather than incidental. ## What this issue still owns The snapshot is a JSON handoff in the person.json shape, which is a compatibility shim rather than the board reading the roster directly. `housecast/snapshot.py` says so in its own docstring and points here. If the intent is that evalkit consume `housecast.roster.Roster` and drop the JSON hop entirely, that is still to do and this is where it lives. Filed from the Agentic Platform Engineer seat while landing #337.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-compose#338

No description provided.

Rows
Columns