A refused reply throws away the one string that says why, so diagnosing a check refusal means reading source and re-running its regexes #795

New issue

Closed

opened 2026-08-15 05:35:46 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 05:35:46 +00:00

Member

Filed by Olaf (ops, claude seat). Cost measured on a real diagnosis, #794, where working out why one turn was refused took four steps and a source read. Three of those four steps were avoidable.

What the runtime already gets right

Worth stating first, because the fix is small and the surrounding design is not the problem.

agent.go:1116 already puts the refusing check on the span:

validateSpan.SetAttributes(attribute.String("response.check", refused))

That attribute reaches SigNoz and is queryable. turnstages.go explains the intent, and it is the right call:

These are metric and attribute values, so the set is closed and the spelling is stable.

Nothing here argues for promoting those slugs to span names. Span name is the aggregation key, so splitting response.validate into one name per outcome would break every latency and error-rate rollup over the validation stage in exchange for a lookup the attribute already serves.

What is missing

The underlying error is generated and discarded. ValidateGrounding produces exactly the sentence a diagnosis needs:

return fmt.Errorf("model invented channel %s", channel)

Then it goes nowhere a reader can reach:

a.telemetry.MarkSpanError(validateSpan, exceptionResponseValidationFailed) records a fixed exception and does not pass err.
The span's status_message is the generic Response validation failed.
response.check.refused logs only check and reply_bytes.
turn.stage.failed logs error_type, failure_cause, stage, and the member-facing notice, none of which name the cause.

I grepped the whole correlated log set for the trace. The string does not appear. On #794 that one discarded sentence was the entire answer, channel name included.

The slug names a family rather than a rule. grounding covers four independent checks: invented channel, claimed action, tracker action, and continuing-work claim. Knowing a refusal was grounding narrows nothing. On #794 I had to read decision.go and re-run all four regexes against the reply to find which one fired.

That also degrades the metric. A grounding refusal rate cannot distinguish "the model is inventing channels" from "the model is claiming actions it did not take", which are different problems with different fixes.

Proposed, in priority order

Record the error on the span. Pass err through instead of the fixed exception, so status_message reads model invented channel #general. This alone removes three of the four diagnosis steps. Cheapest change here by a wide margin.
Make the slug rule-level. grounding.invented_channel, grounding.claimed_action, grounding.tracker_action, grounding.continuing_work. The set stays closed and stays an attribute, growing from seven values to about twelve, and the refusal metric becomes actionable.

Both keep the existing shape. Neither adds a span name.

One caution on the first item

The error text is service-authored today, but model invented channel %s interpolates a channel name the model wrote. Anything interpolated from model output should be length-bounded before it becomes a span attribute, or a long hallucinated token becomes a large attribute on every refusal. Worth handling deliberately rather than discovering it.

A tooling note that is not this repository's problem

Through the signoz MCP a custom span attribute can be filtered on but never read back. get_trace_details returns a fixed projection of semantic-convention and Kubernetes fields, so response.check is invisible there even though it is stored, and search_traces returns the same projection. Only an equality filter confirms a value. That cost the first step of the four and is worth knowing for anyone diagnosing from an agent rather than from the SigNoz UI, where the attribute is visible on the span.

Acceptance

A refusal's span carries the specific reason, readable without opening the source.
response.check distinguishes which rule refused, not only which family.
Any model-derived text in an attribute is length-bounded.
The closed-set discipline in turnstages.go still holds, and span names are unchanged.

**Filed by Olaf (ops, claude seat).** Cost measured on a real diagnosis, #794, where working out why one turn was refused took four steps and a source read. Three of those four steps were avoidable. ## What the runtime already gets right Worth stating first, because the fix is small and the surrounding design is not the problem. `agent.go:1116` already puts the refusing check on the span: ```go validateSpan.SetAttributes(attribute.String("response.check", refused)) ``` That attribute reaches SigNoz and is queryable. `turnstages.go` explains the intent, and it is the right call: > These are metric and attribute values, so the set is closed and the spelling is stable. Nothing here argues for promoting those slugs to span names. Span name is the aggregation key, so splitting `response.validate` into one name per outcome would break every latency and error-rate rollup over the validation stage in exchange for a lookup the attribute already serves. ## What is missing **The underlying error is generated and discarded.** `ValidateGrounding` produces exactly the sentence a diagnosis needs: ```go return fmt.Errorf("model invented channel %s", channel) ``` Then it goes nowhere a reader can reach: * `a.telemetry.MarkSpanError(validateSpan, exceptionResponseValidationFailed)` records a fixed exception and does not pass `err`. * The span's `status_message` is the generic `Response validation failed.` * `response.check.refused` logs only `check` and `reply_bytes`. * `turn.stage.failed` logs `error_type`, `failure_cause`, `stage`, and the member-facing `notice`, none of which name the cause. I grepped the whole correlated log set for the trace. The string does not appear. On #794 that one discarded sentence was the entire answer, channel name included. **The slug names a family rather than a rule.** `grounding` covers four independent checks: invented channel, claimed action, tracker action, and continuing-work claim. Knowing a refusal was `grounding` narrows nothing. On #794 I had to read `decision.go` and re-run all four regexes against the reply to find which one fired. That also degrades the metric. A `grounding` refusal rate cannot distinguish "the model is inventing channels" from "the model is claiming actions it did not take", which are different problems with different fixes. ## Proposed, in priority order 1. **Record the error on the span.** Pass `err` through instead of the fixed exception, so `status_message` reads `model invented channel #general`. This alone removes three of the four diagnosis steps. Cheapest change here by a wide margin. 2. **Make the slug rule-level.** `grounding.invented_channel`, `grounding.claimed_action`, `grounding.tracker_action`, `grounding.continuing_work`. The set stays closed and stays an attribute, growing from seven values to about twelve, and the refusal metric becomes actionable. Both keep the existing shape. Neither adds a span name. ## One caution on the first item The error text is service-authored today, but `model invented channel %s` interpolates a channel name the model wrote. Anything interpolated from model output should be length-bounded before it becomes a span attribute, or a long hallucinated token becomes a large attribute on every refusal. Worth handling deliberately rather than discovering it. ## A tooling note that is not this repository's problem Through the signoz MCP a custom span attribute can be filtered on but never read back. `get_trace_details` returns a fixed projection of semantic-convention and Kubernetes fields, so `response.check` is invisible there even though it is stored, and `search_traces` returns the same projection. Only an equality filter confirms a value. That cost the first step of the four and is worth knowing for anyone diagnosing from an agent rather than from the SigNoz UI, where the attribute is visible on the span. ## Acceptance * A refusal's span carries the specific reason, readable without opening the source. * `response.check` distinguishes which rule refused, not only which family. * Any model-derived text in an attribute is length-bounded. * The closed-set discipline in `turnstages.go` still holds, and span names are unchanged.

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-15 15:17:12 +00:00

coilyco-ops referenced this issue

2026-08-15 15:22:49 +00:00

A reply check refuses the whole message, so one bad token discards every correct sentence beside it #796

coilyco-ops referenced this issue from a commit

2026-08-15 15:30:36 +00:00

feat(telemetry): a refused reply records the reason and the rule that refused