coilyco-flight-deck/agent-compose

Watch

evalkit filter slot key omits `against`, so two role-fit cases can silently drop one #267

New issue

Open

opened 2026-08-12 05:21:53 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-12 05:21:53 +00:00

Member

Outcome

Add against to the competition slot key in evalkit/filter.py, so two role-fit cases for one role cannot collide and lose a case to a silent drop.

Found on 2026-08-11 during the ops pilot slice of the #262 board. Read at evalkit/filter.py on main.

What the slot key is for

filter.run groups candidate variants into slots, keeps the variant whose failure count sits closest to the midpoint, and records every loser as lost slot <slot>. The slot is what says "these two samples compete for the same board position".

filter.py:121:

def _slot(sample: Sample) -> str:
    parts = [sample.role, sample.test_type.value]
    for value in (sample.boundary, sample.half, sample.target, sample.trait):
        if value:
            parts.append(str(value))
    return ":".join(parts)

against is not in that tuple. For a role-fit sample, boundary, half, and trait are all None, so the key reduces to role, test type, and target text.

The failure

Every role carries three role-fit cases: one within-role and two against declared adjacents. Their slots are distinguished only by their target strings differing.

Two role-fit cases for one role that happen to share a target land in the same slot. The filter then treats four candidates as competitors for one position, keeps one, and reports the other three as lost slot. A whole board position disappears into a line that reads like normal item analysis.

The drop is reported, so this is not silent truncation in the sense FilterReport guards against. It is worse in one way: the report says the sample lost a competition it was never meant to enter.

Why the target is a weak key

_slot reads target because a target is a natural per-case string. It is prose written by a generator, and nothing enforces uniqueness across a role's three role-fit cases. Two adjacency edges with similar reasons invite similar targets. The ops pilot avoided it only because the three targets were written to differ.

against is the axis the case is actually about, and role_fit samples already require it (schema.py:105).

Fix

for value in (sample.boundary, sample.half, sample.against, sample.target, sample.trait):

One line, plus a regression test that two role-fit samples for one role with an identical target and different against stay in separate slots.

Note on the boundary tier

Boundary samples are safe today, because boundary and half already separate them. Adding against does not change their keys, since it is None there.

Complete when

_slot carries against.
A test covers two same-role, same-target, different-against role-fit samples staying separate.
ward exec evalkit-check passes.

## Outcome Add `against` to the competition slot key in `evalkit/filter.py`, so two role-fit cases for one role cannot collide and lose a case to a silent drop. Found on 2026-08-11 during the ops pilot slice of the #262 board. Read at `evalkit/filter.py` on `main`. ## What the slot key is for `filter.run` groups candidate variants into slots, keeps the variant whose failure count sits closest to the midpoint, and records every loser as `lost slot <slot>`. The slot is what says "these two samples compete for the same board position". `filter.py:121`: ```python def _slot(sample: Sample) -> str: parts = [sample.role, sample.test_type.value] for value in (sample.boundary, sample.half, sample.target, sample.trait): if value: parts.append(str(value)) return ":".join(parts) ``` `against` is not in that tuple. For a role-fit sample, `boundary`, `half`, and `trait` are all `None`, so the key reduces to role, test type, and target text. ## The failure Every role carries three role-fit cases: one within-role and two against declared adjacents. Their slots are distinguished only by their `target` strings differing. Two role-fit cases for one role that happen to share a target land in the same slot. The filter then treats four candidates as competitors for one position, keeps one, and reports the other three as `lost slot`. A whole board position disappears into a line that reads like normal item analysis. The drop is reported, so this is not silent truncation in the sense `FilterReport` guards against. It is worse in one way: the report says the sample lost a competition it was never meant to enter. ## Why the target is a weak key `_slot` reads `target` because a target is a natural per-case string. It is prose written by a generator, and nothing enforces uniqueness across a role's three role-fit cases. Two adjacency edges with similar reasons invite similar targets. The ops pilot avoided it only because the three targets were written to differ. `against` is the axis the case is actually about, and `role_fit` samples already require it (`schema.py:105`). ## Fix ```python for value in (sample.boundary, sample.half, sample.against, sample.target, sample.trait): ``` One line, plus a regression test that two role-fit samples for one role with an identical target and different `against` stay in separate slots. ## Note on the boundary tier Boundary samples are safe today, because `boundary` and `half` already separate them. Adding `against` does not change their keys, since it is `None` there. ## Complete when * `_slot` carries `against`. * A test covers two same-role, same-target, different-`against` role-fit samples staying separate. * `ward exec evalkit-check` passes.

coilyco-ops referenced this issue

2026-08-12 05:25:04 +00:00

Replace the LLM-reviewer evaluation methodology with a generator, subject, human-grader triple #262

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-compose#267

No description provided.

Rows
Columns

evalkit filter slot key omits against, so two role-fit cases can silently drop one #267