feat(telemetry): a refused reply records the reason and the rule that refused #801

Merged

coilysiren merged 1 commit from aos/claude/wy58-refusal-reason into main

2026-08-15 15:35:25 +00:00

coilyco-ops commented

2026-08-15 15:31:05 +00:00

Member

Closes #795.

What changed

The reason reaches the span and the log. response.validate carries response.check.reason, and response.check.refused logs the same sentence under refused, matching what model.response.repair already does on the completion path. The four diagnosis steps on #794 collapse to reading the span.

response.check names the rule, not the family. grounding covered four independent rules, so the value narrowed nothing and a refusal rate over it could not separate an invented channel from a claimed action. The set grows by four: grounding.invented_channel, grounding.claimed_action, grounding.tracker_action, grounding.continuing_work. Span names are unchanged.

The invented channel is bounded. channelPattern already constrains it to a #token, and it is now truncated at 64 runes before it enters the sentence.

One deliberate departure from the proposal

The issue proposes passing err into MarkSpanError so the span status_message carries the reason. That collides with a documented hard boundary: docs/sirens-echo-exceptions.md states the recording API accepts only the catalog code and that upstream error text cannot enter the exception event, and exceptions_test.go pins it.

So the reason rides an ordinary span attribute beside the exception fields rather than through them. Same readable outcome, and exception.type, exception.message, error.stage, error.outcome, and the span status keep their fixed cataloged wording. A new rule needs no catalog entry and no reviewed increase to the exception cardinality bound. TestARefusalReasonStaysOutOfTheExceptionFields pins the separation.

The reply-fragment crossing, stated rather than discovered

docs/sirens-echo-observability.md said replies are never logged, absolutely. The invented channel token is a reply fragment, so that sentence now names the exception and links the reasoning instead of quietly becoming false. The token is the diagnosis: it separates a channel missing from supplied context, a prompt problem, from a pure hallucination. It names no real channel and carries no member content.

Acceptance

Span carries the specific reason, readable without opening the source - response.check.reason.
response.check distinguishes the rule, not only the family - four grounding values.
Model-derived text is length-bounded - pattern-constrained plus 64-rune truncation.
The closed-set discipline holds and span names are unchanged - the set is still closed, still an attribute.

Verification

ward gate green: build, policy-check, vet, test, test-skips, pre-commit. Four new tests in turnstages_test.go, each run and confirmed passing individually.

Not done

No FEATURES.md entry. This refines diagnostics on an existing capability rather than adding or reshaping one, and neither docs/FEATURES.md nor docs/features-observability.md enumerates the check set.

🤖 Generated with Claude Code

Closes #795. ## What changed **The reason reaches the span and the log.** `response.validate` carries `response.check.reason`, and `response.check.refused` logs the same sentence under `refused`, matching what `model.response.repair` already does on the completion path. The four diagnosis steps on #794 collapse to reading the span. **`response.check` names the rule, not the family.** `grounding` covered four independent rules, so the value narrowed nothing and a refusal rate over it could not separate an invented channel from a claimed action. The set grows by four: `grounding.invented_channel`, `grounding.claimed_action`, `grounding.tracker_action`, `grounding.continuing_work`. Span names are unchanged. **The invented channel is bounded.** `channelPattern` already constrains it to a `#token`, and it is now truncated at 64 runes before it enters the sentence. ## One deliberate departure from the proposal The issue proposes passing `err` into `MarkSpanError` so the span `status_message` carries the reason. That collides with a documented hard boundary: `docs/sirens-echo-exceptions.md` states the recording API accepts only the catalog code and that upstream error text cannot enter the exception event, and `exceptions_test.go` pins it. So the reason rides an ordinary span attribute beside the exception fields rather than through them. Same readable outcome, and `exception.type`, `exception.message`, `error.stage`, `error.outcome`, and the span status keep their fixed cataloged wording. A new rule needs no catalog entry and no reviewed increase to the exception cardinality bound. `TestARefusalReasonStaysOutOfTheExceptionFields` pins the separation. ## The reply-fragment crossing, stated rather than discovered `docs/sirens-echo-observability.md` said replies are never logged, absolutely. The invented channel token is a reply fragment, so that sentence now names the exception and links the reasoning instead of quietly becoming false. The token is the diagnosis: it separates a channel missing from supplied context, a prompt problem, from a pure hallucination. It names no real channel and carries no member content. ## Acceptance - Span carries the specific reason, readable without opening the source - `response.check.reason`. - `response.check` distinguishes the rule, not only the family - four grounding values. - Model-derived text is length-bounded - pattern-constrained plus 64-rune truncation. - The closed-set discipline holds and span names are unchanged - the set is still closed, still an attribute. ## Verification `ward gate` green: build, policy-check, vet, test, test-skips, pre-commit. Four new tests in `turnstages_test.go`, each run and confirmed passing individually. ## Not done No FEATURES.md entry. This refines diagnostics on an existing capability rather than adding or reshaping one, and neither `docs/FEATURES.md` nor `docs/features-observability.md` enumerates the check set. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-15 15:31:06 +00:00

feat(telemetry): a refused reply records the reason and the rule that refused

ci / image-build (pull_request) Successful in 24s

Details

ci / test (pull_request) Successful in 41s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ba7583d91f

The sentence ValidateGrounding writes was generated and discarded. On #794 it
was the entire answer, and reaching it took four steps and a source read.

`response.validate` now carries `response.check.reason` and
`response.check.refused` logs it under `refused`, matching what
`model.response.repair` already does on the completion path. It is an ordinary
span attribute rather than an exception field, so `MarkSpanError` still takes
only a catalog code and the exception grouping contract is unmoved.

`response.check` names the grounding rule rather than the family. Four
independent rules shared one slug, so the value narrowed a diagnosis to a
family and a refusal rate over it could not separate an invented channel from
a claimed action.

The invented channel is the one piece of model-written text this carries. It
is bounded by the channel pattern that matched it and truncated at 64 runes,
and docs/sirens-echo-refusal-reason.md states the crossing rather than leaving
it to be discovered.

Closes #795

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-15 15:31:06 +00:00