docs(eval): record the board's shape before building any of it #847

Merged

coilysiren merged 1 commit from aos/claude/xu59-eval-board into main

2026-08-16 01:25:54 +00:00

coilyco-ops commented

2026-08-16 00:34:11 +00:00

Member

What

One placeholder doc, docs/sirens-echo-eval-board.md. Nothing built, no verb, no case, no code.

Why now

The shape was agreed in conversation and would otherwise be re-derived from scratch. Four decisions are now written down:

the subject is pinned to deepseek-v4-flash
the three seats never overlap: frontier generator, commodity subject, human grader
the board is bot-neutral at this layer
the case list is derived from a boundaries declaration, not hand-maintained

Two differences from the reference board

Structure is taken from agent-compose#262, and the differences are stated rather than implied.

Scale. That board has three boundaries across its roles. This repository has tens, counted from source: 13 content classes in agent/content-classes.yaml (5 deny, 8 allow), 9 reply validators in internal/community/, and prose clauses across five policy skill roots. Paired, that is 60 to 90 cases, which is the same order as the reference's 78 and therefore the same rough 40 minutes of grading. It is only that order if the bot dimension collapses. Keeping Echo and Deep as separate axes multiplies the count past what a human can grade, which is the practical reason the layer is neutral, on top of the #836 reason.

A baseline arm. The reference has no equivalent. A clause that changes nothing between the baseline and policy arms is dead weight, and nothing in this repository currently shows that.

The baseline is not a bare model call

A bare call loses facts of reality that are not policy, such as the reply landing in a public channel in front of members, and then measures the wrong thing. So the baseline is a neutral situational preamble.

That raises a real seam, because some statements are both fact and policy. "You are an agent, not a person" shipped as a policy commit (d93f2c2) and is also a plain fact. The doc resolves it by subtraction, which is mechanically checkable rather than a judgement call:

anything with a boundary case is not in the preamble

So the identity line is a clause, and "this is a public channel" is preamble.

Relationship to the two open epics

#836 child 4 asked for a real decision about per-bot packs rather than a sweep. This is that decision. Existing run records under evaluations/ stay exactly as they are, as provenance, which matches the epic's own note that they are historical records.
#846 tracks it. The absorb path agreed with Kai is that the 11 breaching records become the first paired cases, and the old packs retire per boundary covered as they are.

Not in this PR

No just migration, no boundaries.yaml, no cases, no runner, no FEATURES entry. A placeholder for unbuilt work is not a shipped capability, so the FEATURES entry belongs with the thing itself.

Known gap, stated in the doc

The prose clauses across the five policy skill roots have no machine-readable form, so boundaries.yaml cannot be fully derived from source on day one. The 13 content classes and 9 validators can be. That gap is the first real build task, not a surprise waiting downstream.

🤖 Generated with Claude Code

## What One placeholder doc, `docs/sirens-echo-eval-board.md`. Nothing built, no verb, no case, no code. ## Why now The shape was agreed in conversation and would otherwise be re-derived from scratch. Four decisions are now written down: * the subject is pinned to **`deepseek-v4-flash`** * the three seats never overlap: **frontier generator, commodity subject, human grader** * the board is **bot-neutral at this layer** * the case list is **derived** from a boundaries declaration, not hand-maintained ## Two differences from the reference board Structure is taken from [agent-compose#262](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/issues/262), and the differences are stated rather than implied. **Scale.** That board has three boundaries across its roles. This repository has tens, counted from source: 13 content classes in `agent/content-classes.yaml` (5 deny, 8 allow), 9 reply validators in `internal/community/`, and prose clauses across five policy skill roots. Paired, that is 60 to 90 cases, which is the same order as the reference's 78 and therefore the same rough 40 minutes of grading. **It is only that order if the bot dimension collapses.** Keeping Echo and Deep as separate axes multiplies the count past what a human can grade, which is the practical reason the layer is neutral, on top of the #836 reason. **A baseline arm.** The reference has no equivalent. A clause that changes nothing between the baseline and policy arms is dead weight, and nothing in this repository currently shows that. ## The baseline is not a bare model call A bare call loses facts of reality that are not policy, such as the reply landing in a public channel in front of members, and then measures the wrong thing. So the baseline is a neutral situational preamble. That raises a real seam, because some statements are both fact and policy. "You are an agent, not a person" shipped as a policy commit (`d93f2c2`) and is also a plain fact. The doc resolves it by subtraction, which is mechanically checkable rather than a judgement call: > anything with a boundary case is not in the preamble So the identity line is a clause, and "this is a public channel" is preamble. ## Relationship to the two open epics * **#836 child 4** asked for a real decision about per-bot packs rather than a sweep. This is that decision. Existing run records under `evaluations/` stay exactly as they are, as provenance, which matches the epic's own note that they are historical records. * **#846** tracks it. The absorb path agreed with Kai is that the 11 breaching records become the first paired cases, and the old packs retire per boundary covered as they are. ## Not in this PR No `just` migration, no `boundaries.yaml`, no cases, no runner, no FEATURES entry. A placeholder for unbuilt work is not a shipped capability, so the FEATURES entry belongs with the thing itself. ## Known gap, stated in the doc The prose clauses across the five policy skill roots have no machine-readable form, so `boundaries.yaml` cannot be fully derived from source on day one. The 13 content classes and 9 validators can be. That gap is the first real build task, not a surprise waiting downstream. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-16 00:34:12 +00:00

docs(eval): record the board's shape before building any of it

ci / test (pull_request) Successful in 54s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 20s

Details

f3e12fb05d

Placeholder only. Nothing in it is built, and it ships no verb, no
case, and no code.

It exists because the shape was agreed in conversation and would
otherwise have to be re-derived. Four things are now written down that
were not: the subject is pinned to deepseek-v4-flash, the three seats
never overlap, the board is bot-neutral at this layer, and the case list
is derived from a boundaries declaration rather than hand-maintained.

The structure comes from Agent Compose's evaluation triple at
agent-compose#262, which is public and linked. Two differences from that
reference are called out rather than left implied. This repository has
tens of boundaries against the reference's three, so the bot dimension
has to collapse or the case count runs past what a human can grade. And
the board gains a baseline arm, because a clause that changes nothing
between the two arms is dead weight and nothing currently shows that.

The baseline is a neutral situational preamble rather than a bare model
call. A bare call loses facts of reality that are not policy, such as
the reply landing in a public channel, and then measures the wrong
thing. The fact-policy seam is decided by subtraction and is
mechanically checkable: anything carrying a boundary case is not in the
preamble.

This is also the answer to #836 child 4, which asked for a real decision
about per-bot packs rather than a sweep. Existing run records under
evaluations/ stay exactly as they are, as provenance.

No FEATURES entry. A placeholder for unbuilt work is not a shipped
capability, and the entry belongs with the thing itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-16 00:34:12 +00:00

coilyco-ops referenced this pull request

2026-08-16 00:34:38 +00:00

Epic: a third bot leaves no trace in this repository, so bot configuration moves to deploy #836

coilyco-ops referenced this pull request

2026-08-16 00:34:41 +00:00

Epic: measured breaches are back-filling prompt rules that shipped on assertion, and then not getting fixed #846

coilyco-ops referenced this pull request

2026-08-16 01:22:58 +00:00

feat(eval): declare every boundary, and move the command core to just #853

coilyco-ops referenced this pull request