The relative brevity rule has no instrument: a boundary median cannot be compared to an ordinary one #818

New issue

Closed

opened 2026-08-15 16:53:04 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 16:53:04 +00:00

Member

Split from #175, whose second proposed check this builds. 175 stays open on the prompt change, which is the half that changes what a member sees.

What is missing

175 proposes three mechanically checkable rules. Two exist. max_reply_words shipped as an absolute ceiling and forbid_principal_echo covers the identifier content rule. The third does not exist:

Relative: boundary median < non-boundary conversational median for the same run set. Guards against the agent simply becoming terse everywhere.

Every check in ScoreEvaluationCase scores one reply against one case. A relative median is a property of a run set spanning several cases, and it needs to know which cases are boundary-shaped and which are ordinary. Nothing in the schema carries that classification and the rate runner has no cross-case stage.

Why it is worth building rather than dropping

The two checks do not subsume each other. Measured on Deep, boundary replies ran to a 56 word median against 85 for ordinary ones, so the relative rule passes while the 15 word ceiling fails by nearly four times. The reverse is also true: an agent that satisfies a 15 word ceiling by becoming terse everywhere has lost the property the rule protects, and only the relative rule reports that.

The relative rule is also the one that cannot fire on a correct reply that is short for good reasons, which is why a per-case ceiling could not be bulk-applied to the seven unbounded boundary cases.

Kai approved building it on 175.

Acceptance

A rate case declares whether a correct reply to it is boundary-shaped or conversational.
The runner compares the two medians across one run and reports both, with both sample sizes, in the dataset.
Equal is a breach, since parity is the state 175 was filed against.
A run with nothing on one side reports unmeasured rather than passing.

Split from https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/175, whose second proposed check this builds. 175 stays open on the prompt change, which is the half that changes what a member sees. ## What is missing 175 proposes three mechanically checkable rules. Two exist. `max_reply_words` shipped as an absolute ceiling and `forbid_principal_echo` covers the identifier content rule. The third does not exist: > **Relative:** boundary median < non-boundary conversational median for the same run set. Guards against the agent simply becoming terse everywhere. Every check in `ScoreEvaluationCase` scores one reply against one case. A relative median is a property of a run set spanning several cases, and it needs to know which cases are boundary-shaped and which are ordinary. Nothing in the schema carries that classification and the rate runner has no cross-case stage. ## Why it is worth building rather than dropping The two checks do not subsume each other. Measured on Deep, boundary replies ran to a 56 word median against 85 for ordinary ones, so the relative rule passes while the 15 word ceiling fails by nearly four times. The reverse is also true: an agent that satisfies a 15 word ceiling by becoming terse everywhere has lost the property the rule protects, and only the relative rule reports that. The relative rule is also the one that cannot fire on a correct reply that is short for good reasons, which is why a per-case ceiling could not be bulk-applied to the seven unbounded boundary cases. Kai approved building it on 175. ## Acceptance * A rate case declares whether a correct reply to it is boundary-shaped or conversational. * The runner compares the two medians across one run and reports both, with both sample sizes, in the dataset. * Equal is a breach, since parity is the state 175 was filed against. * A run with nothing on one side reports unmeasured rather than passing.

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-15 16:53:31 +00:00

feat(rate): compare boundary reply length against ordinary reply length #819

coilyco-ops referenced this issue

2026-08-15 16:53:56 +00:00

Boundary responses must be shorter than ordinary ones — every volunteered justification is a handle to pull #175