Record body-safe OpenTelemetry exception events #55

New issue

Closed

opened 2026-08-01 19:06:49 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-01 19:06:49 +00:00

Member

Outcome

Emit body-safe OpenTelemetry exception events for Sirens Echo failures so SigNoz Error Management can group and link them to her existing traces.

Observed evidence

The ser8 SigNoz tenant saw 6,462 Sirens Echo spans and 19 error spans during the last 30 days.
Telemetry.MarkSpanError sets a closed-set error.type attribute and codes.Error status.
The source does not call Span.RecordError or otherwise add an OpenTelemetry exception event.
The deploy already exports OTLP/HTTP to the canonical collector. This is a producer instrumentation gap, not a deployment gap.

Safety contract

Sirens Echo telemetry must continue to exclude member, prompt, model, tool, and reply bodies. Do not pass an arbitrary wrapped error directly into telemetry when its message could contain those values. Export a closed-set exception type and message plus a useful stack trace.

Acceptance

Each terminal failed operation records an OpenTelemetry exception event and marks the same span as error.
The exception event uses a closed-set, body-safe exception.type and message suitable for low-cardinality grouping.
Stack diagnostics identify the failing code path without copying request, response, model, tool, or Discord content.
Parent propagation preserves the existing joined Sirens Echo, Agent Proxy, LiteLLM, and Eco trace.
Focused tests inspect exported span events, status, safe attributes, and the absence of forbidden content.
An Ops verification with one synthetic failure shows a Sirens Echo group in SigNoz Error Management and links it to the expected trace.
Update the shipped observability feature description when the producer behavior lands.

## Outcome Emit body-safe OpenTelemetry exception events for Sirens Echo failures so SigNoz Error Management can group and link them to her existing traces. ## Observed evidence * The ser8 SigNoz tenant saw 6,462 Sirens Echo spans and 19 error spans during the last 30 days. * `Telemetry.MarkSpanError` sets a closed-set `error.type` attribute and `codes.Error` status. * The source does not call `Span.RecordError` or otherwise add an OpenTelemetry `exception` event. * The deploy already exports OTLP/HTTP to the canonical collector. This is a producer instrumentation gap, not a deployment gap. ## Safety contract Sirens Echo telemetry must continue to exclude member, prompt, model, tool, and reply bodies. Do not pass an arbitrary wrapped error directly into telemetry when its message could contain those values. Export a closed-set exception type and message plus a useful stack trace. ## Acceptance * Each terminal failed operation records an OpenTelemetry exception event and marks the same span as error. * The exception event uses a closed-set, body-safe `exception.type` and message suitable for low-cardinality grouping. * Stack diagnostics identify the failing code path without copying request, response, model, tool, or Discord content. * Parent propagation preserves the existing joined Sirens Echo, Agent Proxy, LiteLLM, and Eco trace. * Focused tests inspect exported span events, status, safe attributes, and the absence of forbidden content. * An Ops verification with one synthetic failure shows a Sirens Echo group in SigNoz Error Management and links it to the expected trace. * Update the shipped observability feature description when the producer behavior lands.

coilyco-ops added the

priority

autonomy

live-collab

labels

2026-08-05 10:26:18 +00:00

coilyco-ops referenced this issue

2026-08-07 18:40:34 +00:00

Sirens Deep: what is already shipped, the August 19 gap, and stale repo identity #76

coilyco-ops referenced this issue

2026-08-11 09:41:25 +00:00

forgejo.issue.failed fires silently and carries no diagnosable detail #89

coilyco-ops commented

2026-08-11 09:41:42 +00:00

Author

Member

Checked against current main (e0b3dc1). This is delivered, with one acceptance bullet consciously resolved the other way. Reporting rather than reworking.

Met

internal/community/exceptions.go now carries a closed-set exception catalog (exceptionCode, exceptionSpec) and MarkSpanError at line 213 does all of:

span.AddEvent("exception", ...) with exception.type, exception.message, error.stage, error.outcome
span attributes error.type, error.stage, error.outcome
span.SetStatus(codes.Error, spec.message)

Every type is a compile-time constant, so no runtime data reaches the fields. Call sites cover the model transport, MCP list, tool call, session close, response validation, Forgejo issue, and the HTTP turn rejections. exceptions_test.go inspects exported span events, status, and attributes. docs/FEATURES.md:33 already describes "bounded SigNoZ exception groups with stage and outcome tags".

Deliberately not met

The acceptance asks for "a useful stack trace". The implementation ships none, and exceptions_test.go:38 and :115 actively fail if exception.stacktrace appears on the event.

That reads as the right call rather than an oversight. The safety contract in this same issue says not to pass a wrapped error into telemetry when its message could carry member, prompt, model, tool, or reply values, and a Go stack trace can carry argument data. The two bullets pull against each other and the implementation picked safety. Worth confirming that is what you want, since it is the one gap between the issue text and the code.

Genuinely outstanding

Only the last operational bullet: an Ops verification with one synthetic failure showing a Sirens Echo group in SigNoz Error Management, linked to the expected trace. That needs a live deployment and is not something I can close from here.

Suggest either closing this and tracking the SigNoz check separately, or narrowing it to that one bullet.

Checked against current `main` (`e0b3dc1`). This is delivered, with one acceptance bullet consciously resolved the other way. Reporting rather than reworking. ## Met `internal/community/exceptions.go` now carries a closed-set exception catalog (`exceptionCode`, `exceptionSpec`) and `MarkSpanError` at line 213 does all of: * `span.AddEvent("exception", ...)` with `exception.type`, `exception.message`, `error.stage`, `error.outcome` * span attributes `error.type`, `error.stage`, `error.outcome` * `span.SetStatus(codes.Error, spec.message)` Every type is a compile-time constant, so no runtime data reaches the fields. Call sites cover the model transport, MCP list, tool call, session close, response validation, Forgejo issue, and the HTTP turn rejections. `exceptions_test.go` inspects exported span events, status, and attributes. `docs/FEATURES.md:33` already describes "bounded SigNoZ exception groups with stage and outcome tags". ## Deliberately not met The acceptance asks for "a useful stack trace". The implementation ships none, and `exceptions_test.go:38` and `:115` actively fail if `exception.stacktrace` appears on the event. That reads as the right call rather than an oversight. The safety contract in this same issue says not to pass a wrapped error into telemetry when its message could carry member, prompt, model, tool, or reply values, and a Go stack trace can carry argument data. The two bullets pull against each other and the implementation picked safety. Worth confirming that is what you want, since it is the one gap between the issue text and the code. ## Genuinely outstanding Only the last operational bullet: an Ops verification with one synthetic failure showing a Sirens Echo group in SigNoz Error Management, linked to the expected trace. That needs a live deployment and is not something I can close from here. Suggest either closing this and tracking the SigNoz check separately, or narrowing it to that one bullet.