Mark recorded OpenTelemetry exceptions as error spans #71

New issue

Closed

opened 2026-08-01 19:06:51 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-01 19:06:51 +00:00

Member

Outcome

Make Agent Proxy's manually recorded OpenTelemetry exceptions satisfy the SigNoz Error Management producer contract.

Observed evidence

The ser8 SigNoz tenant saw 9,884 Agent Proxy spans and 24 error spans during the last 30 days.
app/resilience.py and app/upstream.py call record_exception on manually-created spans.
Those paths do not explicitly set the same span to StatusCode.ERROR.
HTTP auto-instrumentation still produces error spans, so APM error counts exist. That does not prove each manual exception event has the status required by the local Error Management contract.

Acceptance

Every manual record_exception call that represents a failed operation marks that same span as error.
Retried attempts remain individually visible without incorrectly marking a later successful request as failed.
Exception telemetry stays body-safe. It contains no prompt, response, tool payload, authorization value, or raw customer content.
Focused tests assert both the exception event and error status on non-streaming, streaming, and retry failure paths.
An Ops verification with one synthetic transport failure shows the exception under service.name=agent-proxy in SigNoz Error Management and links it to the expected trace.
No deploy endpoint change is required. The production chart already exports OTLP/HTTP to the canonical collector.

## Outcome Make Agent Proxy's manually recorded OpenTelemetry exceptions satisfy the SigNoz Error Management producer contract. ## Observed evidence * The ser8 SigNoz tenant saw 9,884 Agent Proxy spans and 24 error spans during the last 30 days. * `app/resilience.py` and `app/upstream.py` call `record_exception` on manually-created spans. * Those paths do not explicitly set the same span to `StatusCode.ERROR`. * HTTP auto-instrumentation still produces error spans, so APM error counts exist. That does not prove each manual exception event has the status required by the local Error Management contract. ## Acceptance * Every manual `record_exception` call that represents a failed operation marks that same span as error. * Retried attempts remain individually visible without incorrectly marking a later successful request as failed. * Exception telemetry stays body-safe. It contains no prompt, response, tool payload, authorization value, or raw customer content. * Focused tests assert both the exception event and error status on non-streaming, streaming, and retry failure paths. * An Ops verification with one synthetic transport failure shows the exception under `service.name=agent-proxy` in SigNoz Error Management and links it to the expected trace. * No deploy endpoint change is required. The production chart already exports OTLP/HTTP to the canonical collector.

coilyco-ops added the

priority

autonomy

live-collab

labels

2026-08-05 10:25:48 +00:00

coilyco-ops referenced this issue from a commit

2026-08-12 11:01:29 +00:00

Prove the error-span contract across the real failure paths

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-12 11:01:29 +00:00

Prove the error-span contract across the real failure paths #94