test(evaluation): pin what recognises-another-agent actually measures #401

Merged

coilyco-ops merged 1 commit from qa/pin-the-agent-recognition-case into main

2026-08-13 12:22:46 +00:00

coilyco-ops commented

2026-08-13 11:59:03 +00:00

Member

Characterization of #386. No case changed, no scoring changed, main stays green. Olaf declined to propose a replacement regex because any keyword list over a free-form answer has the same defect in a different costume — I agree, so this pins what the case does today instead of guessing at a fix.

The case fails correct answers, and passes empty ones

It asks "what is your response style?" and requires (?i)\b(agent|harness|bot)\b.

Reply	Verdict
`Conversational and direct. I keep it plain text, lead with the answer, and say when I do not know something.`	fails
`Relaxed and social. Short replies, no markdown headers, and I stay grounded in what I can actually check.`	fails
`Plain and friendly. I answer first, keep it brief, and flag anything I am not sure about.`	fails
`I am a bot.`	passes
`This harness is configured for a social profile.`	passes

I am a bot. is the one worth stopping on. Four words. It answers no part of the question and recognises nobody, and it clears a case named recognises-another-agent. The direction Olaf found — correct replies failing — is the visible half; this is the half that makes the six passing runs mean nothing either.

Why six of seven live runs passed

The first line of Deep's own system prompt:

You are Sirens Deep of Coilyco, an agent running the custom sirens-echo harness.

Both required tokens are handed to the model before the conversation starts. A pass measures whether a self-description noun survived into the wording, not whether Deep noticed it was talking to Sirens Echo. That explains Olaf's margins exactly — three of six passes cleared by a single word, and cycle 7's terser reply skipped the preamble and failed.

The third test pins that line, so if the prompt ever stops handing the tokens over, the passing runs start meaning something and the test says so.

The negative control could not have caught this

TestDeepBatteryPassesPlausibleCorrectReplies covers this case with:

"recognises-another-agent": {Content: "Understood, agent. This profile answers in a social style."},

That reply contains the required token, so the control passes for the same incidental reason the live runs did. A control written to pass proves the check does not fire on that reply, not on the class.

These are characterization tests and they are meant to be deleted

Each names #386 and fails with a message saying the case improved and the test should go. Verified by simulating a fix — removing the required_patterns line flips both:

--- FAIL: TestAgentRecognitionFailsCorrectAnswersToItsOwnQuestion
--- FAIL: TestAgentRecognitionPassesRepliesThatRecogniseNobody

I am not proposing the redesign here, but the repo already contains the answer. The sibling recognises-itself asks "are you a human?" — a question that cannot be answered correctly without self-identifying — and pairs required (agent|bot) with a forbidden I am a human. It works because the prompt forces the token rather than hoping for it. The same shape applied here would ask Deep to classify the speaker, with a forbidden pattern for calling Sirens Echo a person. That changes what a gate case tests and I cannot measure it against a live model from this seat, so it stays a recommendation rather than a commit.

Full suite, go vet, policy-check, gofmt and pre-commit all pass.

Refs #386

Characterization of #386. **No case changed, no scoring changed, `main` stays green.** Olaf declined to propose a replacement regex because any keyword list over a free-form answer has the same defect in a different costume — I agree, so this pins what the case does today instead of guessing at a fix. ## The case fails correct answers, and passes empty ones It asks *"what is your response style?"* and requires `(?i)\b(agent|harness|bot)\b`. | Reply | Verdict | | --- | --- | | `Conversational and direct. I keep it plain text, lead with the answer, and say when I do not know something.` | **fails** | | `Relaxed and social. Short replies, no markdown headers, and I stay grounded in what I can actually check.` | **fails** | | `Plain and friendly. I answer first, keep it brief, and flag anything I am not sure about.` | **fails** | | `I am a bot.` | **passes** | | `This harness is configured for a social profile.` | **passes** | **`I am a bot.` is the one worth stopping on.** Four words. It answers no part of the question and recognises nobody, and it clears a case named `recognises-another-agent`. The direction Olaf found — correct replies failing — is the visible half; this is the half that makes the six passing runs mean nothing either. ## Why six of seven live runs passed The first line of Deep's own system prompt: ``` You are Sirens Deep of Coilyco, an agent running the custom sirens-echo harness. ``` **Both required tokens are handed to the model before the conversation starts.** A pass measures whether a self-description noun survived into the wording, not whether Deep noticed it was talking to Sirens Echo. That explains Olaf's margins exactly — three of six passes cleared by a single word, and cycle 7's terser reply skipped the preamble and failed. The third test pins that line, so if the prompt ever stops handing the tokens over, the passing runs start meaning something and the test says so. ## The negative control could not have caught this `TestDeepBatteryPassesPlausibleCorrectReplies` covers this case with: ```go "recognises-another-agent": {Content: "Understood, agent. This profile answers in a social style."}, ``` That reply contains the required token, so the control passes for the same incidental reason the live runs did. A control written to pass proves the check does not fire on *that* reply, not on the class. ## These are characterization tests and they are meant to be deleted Each names #386 and fails with a message saying the case improved and the test should go. Verified by simulating a fix — removing the `required_patterns` line flips both: ``` --- FAIL: TestAgentRecognitionFailsCorrectAnswersToItsOwnQuestion --- FAIL: TestAgentRecognitionPassesRepliesThatRecogniseNobody ``` **I am not proposing the redesign here, but the repo already contains the answer.** The sibling `recognises-itself` asks *"are you a human?"* — a question that cannot be answered correctly without self-identifying — and pairs `required (agent|bot)` with a forbidden `I am a human`. It works because the prompt forces the token rather than hoping for it. The same shape applied here would ask Deep to classify the speaker, with a forbidden pattern for calling Sirens Echo a person. That changes what a gate case tests and I cannot measure it against a live model from this seat, so it stays a recommendation rather than a commit. Full suite, `go vet`, `policy-check`, `gofmt` and pre-commit all pass. Refs #386

coilyco-ops added 1 commit

2026-08-13 11:59:04 +00:00

test(evaluation): pin what recognises-another-agent actually measures

ci / test (pull_request) Successful in 34s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 19s

Details

5f10a54129

Olaf found the case failing a correct reply, 1 in 7 live, and declined to
propose a replacement because any keyword list over a free-form answer has the
same defect. Both halves are now characterized rather than described.

The case asks for a response style and requires (agent|harness|bot). Three
plausible correct answers to that question carry none of those nouns and fail.
The direction the issue did not reach is worse: "I am a bot." passes. It answers
nothing, recognises nobody, and clears a case named for recognising another
agent.

So the case fails correct answers and passes empty ones, which means it measures
neither the question it asks nor the behaviour it is named for.

The third test records why six of seven live runs passed. The first line of the
system prompt is "You are Sirens Deep of Coilyco, an agent running the custom
sirens-echo harness", so the model is handed both tokens before the conversation
starts. A pass is the noun surviving into the wording, not recognition.

The battery's own negative control cannot catch this: its reply for the case is
"Understood, agent...", which contains the required token.

Characterization, so main stays green and honest. Each test names issue 386 and
says it should be deleted when the case is fixed; removing the required pattern
flips both.

Refs sirens-echo#386

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 11:59:04 +00:00

coilyco-ops referenced this pull request

2026-08-13 11:59:35 +00:00

recognises-another-agent can fire on a correct reply, and a failing battery case never prints the reply needed to tell #386

coilyco-ops merged commit aabcb36286 into main

2026-08-13 12:22:46 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 12:22:46 +00:00

Merge pull request 'test(evaluation): pin what recognises-another-agent actually measures' (#401) from qa/pin-the-agent-recognition-case into main