coilyco-flight-deck/agent-compose

Watch

Evaluate a replaceable review UI for behavioral evals #213

New issue

Open

opened 2026-08-06 03:12:40 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-06 03:12:40 +00:00

Member

Outcome

Decide whether a replaceable off-the-shelf review UI materially reduces the time and friction of independently reviewing Agent Compose behavioral evaluations, without moving the canonical evaluation contract or evidence out of Git.

A valid outcome is an explicit decision to keep the current repository-native review path.

Current evidence

#142 established the v2 evaluation pack, raw-response, scored-result, provenance, and independent-QA contracts.
#183 records a current frontier baseline of 54/56 passing cases and 458/464 points.
#191 owns commodity and OSS evaluation lanes.
coilysiren/inbox#329 owns the cost-controlled inference program.

The evaluation system exists. This issue isolates review ergonomics from inference cost and model routing.

Tracer comparison

Measure the current time and steps needed to review one already-committed, public-safe frontier result set.
Import that same frozen result set into at most two candidate surfaces. Start with Langfuse's bounded annotation queue and Promptfoo's local web viewer unless current evidence identifies a clearly better fit.
Review the same bounded case slice through each candidate.
Test whether criterion scores, comments, reviewer identity, completion state, and source provenance can return to the owning Agent Compose records without loss or manual reconstruction.
Record setup cost, recurring review time, data placement, export quality, dependency burden, and deletion or exit behavior.
Choose adopt, view-only, watch, or keep-current.

Adoption gate

Adopt a surface only when all of these are true:

It materially reduces recurring independent-review friction.
Agent Compose YAML, raw responses, pack digests, and generated scorecards remain canonical.
The integration is a thin adapter or derived view, not a second evaluation framework.
A reviewer can export and leave without losing evidence.
The tool does not require model calls merely to display or record human review.
The recurring cost and maintenance burden are proportionate to a small portfolio.

Boundaries

Use committed public-safe evaluation evidence only.
Run no new model matrix for this comparison.
Upload no Voice Corpus records or other private material.
Do not replace the current pack, result, scorer, scorecard, or release-gate contracts.
Do not add production tracing, prompt management, or generalized observability.
Do not purchase a paid tier before the tracer demonstrates value.
Do not use career-signature value as the adoption criterion. Any later case-study value must come from a tool that first improves the real workflow.

Complete when

The current path and candidate paths have comparable time and friction evidence.
Round-trip and exit behavior are proven or rejected.
One explicit adopt, view-only, watch, or keep-current decision is recorded with its revisit trigger.
Any adopted adapter remains bounded, documented, validated, and committed through the repository's declared Ward workflow.

## Outcome Decide whether a replaceable off-the-shelf review UI materially reduces the time and friction of independently reviewing Agent Compose behavioral evaluations, without moving the canonical evaluation contract or evidence out of Git. A valid outcome is an explicit decision to keep the current repository-native review path. ## Current evidence * https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/issues/142 established the v2 evaluation pack, raw-response, scored-result, provenance, and independent-QA contracts. * https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/issues/183 records a current frontier baseline of 54/56 passing cases and 458/464 points. * https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/issues/191 owns commodity and OSS evaluation lanes. * https://forgejo.coilysiren.me/coilysiren/inbox/issues/329 owns the cost-controlled inference program. The evaluation system exists. This issue isolates review ergonomics from inference cost and model routing. ## Tracer comparison 1. Measure the current time and steps needed to review one already-committed, public-safe frontier result set. 2. Import that same frozen result set into at most two candidate surfaces. Start with Langfuse's bounded annotation queue and Promptfoo's local web viewer unless current evidence identifies a clearly better fit. 3. Review the same bounded case slice through each candidate. 4. Test whether criterion scores, comments, reviewer identity, completion state, and source provenance can return to the owning Agent Compose records without loss or manual reconstruction. 5. Record setup cost, recurring review time, data placement, export quality, dependency burden, and deletion or exit behavior. 6. Choose adopt, view-only, watch, or keep-current. ## Adoption gate Adopt a surface only when all of these are true: * It materially reduces recurring independent-review friction. * Agent Compose YAML, raw responses, pack digests, and generated scorecards remain canonical. * The integration is a thin adapter or derived view, not a second evaluation framework. * A reviewer can export and leave without losing evidence. * The tool does not require model calls merely to display or record human review. * The recurring cost and maintenance burden are proportionate to a small portfolio. ## Boundaries * Use committed public-safe evaluation evidence only. * Run no new model matrix for this comparison. * Upload no Voice Corpus records or other private material. * Do not replace the current pack, result, scorer, scorecard, or release-gate contracts. * Do not add production tracing, prompt management, or generalized observability. * Do not purchase a paid tier before the tracer demonstrates value. * Do not use career-signature value as the adoption criterion. Any later case-study value must come from a tool that first improves the real workflow. ## Complete when * The current path and candidate paths have comparable time and friction evidence. * Round-trip and exit behavior are proven or rejected. * One explicit adopt, view-only, watch, or keep-current decision is recorded with its revisit trigger. * Any adopted adapter remains bounded, documented, validated, and committed through the repository's declared Ward workflow.

coilyco-ops added the

autonomy

headless

priority

labels

2026-08-06 03:14:15 +00:00

coilyco-ops referenced this issue

2026-08-10 02:07:21 +00:00

Replace the LLM-reviewer evaluation methodology with a generator, subject, human-grader triple #262

coilyco-ops referenced this issue from a commit

2026-08-10 18:00:10 +00:00

docs: add reference pages for the six external eval reference points

coilyco-ops referenced this issue

2026-08-10 19:36:31 +00:00

Role adjacency in KDL, and evalkit as the Python half of the eval system #263

coilyco-ops commented

2026-08-16 04:48:45 +00:00

Author

Member

Kai has decided: adopt Phoenix for the annotation leg. Lucia (AI Engineer seat), 2026-08-15.

Recording it here because this issue owns the decision, and the deployment spec is coilyco-bridge/deploy #572.

Scope is one leg. docs/eval-references.md currently reads "Inspect is adopted for the run leg. Phoenix stays a reference, since annotation is local. A review UI remains the open decision in #213." Only that last sentence changes. evalkit.run and evalkit.filter are untouched and Phoenix runs no evaluations, which keeps it inside this issue's "thin adapter or derived view, not a second evaluation framework" criterion.

Phoenix was not on the candidate list here - this issue named Langfuse and Promptfoo. The choice came from Kai on legibility grounds rather than from the tracer comparison, so steps 1 through 5 of the comparison were not run. Worth stating rather than implying an evidence trail that does not exist.

Your adoption gate is now the acceptance criteria in #572, and two of its clauses settle the deployment's hardest question on their own. "YAML, raw responses, pack digests and generated scorecards remain canonical" plus "a reviewer can export and leave without losing evidence" mean Phoenix is a view and never the record, so its volume is disposable by design and no backup machinery is needed.

One clause becomes a deployment property rather than a policy. "The tool does not require model calls merely to display or record human review" is enforced by shipping Phoenix with no model credentials and no evaluator route. It cannot call what it has no key for. That also holds the line docs/eval-ref-phoenix.md draws against the evaluation/llm-evals page, which describes the LLM-as-judge approach this eval replaced.

Step 4 is the real gate and is unproven. Whether label, critique, evidence, reviewer identity, and provenance return to the owning records without manual reconstruction decides whether any of this is worth having. evalkit/schema.py looks built for it - its own comment says "Phoenix configures its annotation rubric as data rather than hardcoding it" - but looking tractable is not the same as proven. #572 sequences that round trip ahead of the deployment.

And the incumbent is real. evalkit/annotate.py appends after every decision, so an interrupted session keeps everything already annotated. For a 56-case pass that property matters, and Phoenix should be measured against it rather than against nothing.

This issue can close on the adopt outcome once the round trip is proven, or stay open if it is not.

**Kai has decided: adopt Phoenix for the annotation leg.** Lucia (AI Engineer seat), 2026-08-15. Recording it here because this issue owns the decision, and the deployment spec is `coilyco-bridge/deploy` #572. **Scope is one leg.** `docs/eval-references.md` currently reads *"Inspect is adopted for the run leg. Phoenix stays a reference, since annotation is local. A review UI remains the open decision in #213."* Only that last sentence changes. `evalkit.run` and `evalkit.filter` are untouched and Phoenix runs no evaluations, which keeps it inside this issue's "thin adapter or derived view, not a second evaluation framework" criterion. **Phoenix was not on the candidate list here** - this issue named Langfuse and Promptfoo. The choice came from Kai on legibility grounds rather than from the tracer comparison, so steps 1 through 5 of the comparison were not run. Worth stating rather than implying an evidence trail that does not exist. **Your adoption gate is now the acceptance criteria in #572**, and two of its clauses settle the deployment's hardest question on their own. "YAML, raw responses, pack digests and generated scorecards remain canonical" plus "a reviewer can export and leave without losing evidence" mean **Phoenix is a view and never the record**, so its volume is disposable by design and no backup machinery is needed. **One clause becomes a deployment property rather than a policy.** "The tool does not require model calls merely to display or record human review" is enforced by shipping Phoenix with no model credentials and no evaluator route. It cannot call what it has no key for. That also holds the line `docs/eval-ref-phoenix.md` draws against the `evaluation/llm-evals` page, which describes the LLM-as-judge approach this eval replaced. **Step 4 is the real gate and is unproven.** Whether label, critique, evidence, reviewer identity, and provenance return to the owning records without manual reconstruction decides whether any of this is worth having. `evalkit/schema.py` looks built for it - its own comment says "Phoenix configures its annotation rubric as data rather than hardcoding it" - but looking tractable is not the same as proven. #572 sequences that round trip ahead of the deployment. **And the incumbent is real.** `evalkit/annotate.py` appends after every decision, so an interrupted session keeps everything already annotated. For a 56-case pass that property matters, and Phoenix should be measured against it rather than against nothing. This issue can close on the adopt outcome once the round trip is proven, or stay open if it is not.

coilyco-ops commented

2026-08-16 05:05:05 +00:00

Author

Member

Correction to my previous comment. Phoenix is not being adopted as a review UI. Lucia (AI Engineer seat), 2026-08-15.

Kai clarified: the goal is screenshots and demos, not a workflow. Phoenix becomes a read-only display surface fed one way from committed records, and nobody annotates in it. evalkit.annotate stays the grading surface, unchanged.

So this issue's question is still open. I said last comment that your adoption gate had become the acceptance criteria for a deployment. That was wrong in an important way: a tool nobody reviews in cannot reduce review friction, which is the thing you set out to measure. The gate does not apply because the tool is not doing the job the gate is about.

Two clean outcomes, both of which your Outcome section already allows:

Close as "keep the current repository-native review path", which you explicitly name as a valid result, and record separately that Phoenix was adopted for presentation.
Stay open if the review-friction question is still live and Langfuse or Promptfoo deserve the tracer comparison you designed.

My read is the first. evalkit.annotate works, and the friction Kai actually named was other people's legibility rather than her own review time. But this is your decision to record and I am not recording it for you.

One thing worth keeping either way. Step 4, the round trip, was the sharpest requirement in this issue and it is the reason a display-only Phoenix is cheap: nothing is authored there, so nothing has to come back. If a review UI is ever revisited, step 4 remains the gate that separates a real adapter from a second system.

Deployment spec: coilyco-bridge/deploy #572.

**Correction to my previous comment. Phoenix is not being adopted as a review UI.** Lucia (AI Engineer seat), 2026-08-15. Kai clarified: the goal is screenshots and demos, not a workflow. Phoenix becomes a **read-only display surface** fed one way from committed records, and nobody annotates in it. `evalkit.annotate` stays the grading surface, unchanged. **So this issue's question is still open.** I said last comment that your adoption gate had become the acceptance criteria for a deployment. That was wrong in an important way: a tool nobody reviews in cannot reduce review friction, which is the thing you set out to measure. The gate does not apply because the tool is not doing the job the gate is about. Two clean outcomes, both of which your Outcome section already allows: * **Close as "keep the current repository-native review path"**, which you explicitly name as a valid result, and record separately that Phoenix was adopted for presentation. * **Stay open** if the review-friction question is still live and Langfuse or Promptfoo deserve the tracer comparison you designed. My read is the first. `evalkit.annotate` works, and the friction Kai actually named was other people's legibility rather than her own review time. But this is your decision to record and I am not recording it for you. **One thing worth keeping either way.** Step 4, the round trip, was the sharpest requirement in this issue and it is the reason a display-only Phoenix is cheap: nothing is authored there, so nothing has to come back. If a review UI is ever revisited, step 4 remains the gate that separates a real adapter from a second system. Deployment spec: `coilyco-bridge/deploy` #572.

coilyco-ops referenced this issue

2026-08-16 05:32:32 +00:00

feat(evalkit): project a committed run into a display payload, one way #290

coilyco-ops referenced this issue from a commit

2026-08-18 16:17:56 +00:00

docs(evals): bring the reference pages back, superseded parts only

coilyco-ops referenced this issue

2026-08-18 16:18:04 +00:00

docs(evals): bring the reference pages back, superseded parts only #306