agent-proxy: instrument the request path with OTel spans + structlog events (tracer/logger are declared but never called) #28

New issue

Closed

opened 2026-07-02 09:31:15 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-02 09:31:15 +00:00

Owner

Want

Make agent-proxy actually emit the per-request detail its obs layer only declares today. app/obs.py wires get_tracer() (OTLP HTTP exporter, off unless PROXY_OTEL_EXPORTER_OTLP_ENDPOINT is set) and get_logger() (structlog JSON), but there are zero start_as_current_span and zero logger.* call sites on the request path - so a SigNoz sees no traces and docker logs shows nothing per-request. Prometheus /metrics is already fully wired (24 points); this ticket adds the trace + structured-log channels so localhost high-detail inspection works.

Ask

Instrument the request path end to end:

Spans. Open a root span per inbound request in main.py (/v1/chat/completions, /v1/completions), with child spans wrapping the real stages: queue enqueue/wait (queue.py), each upstream attempt (upstream.py), response validation (resilience.py), and each retry / fallback / circuit transition (resilience.py). The causal chain (which backend, which attempt, why it fell back) is the whole point.
Attributes. Tag spans with OTel GenAI semconv gen_ai.* (request model, gen_ai.usage.input_tokens/output_tokens, response.finish_reasons) plus agentproxy.* (logical model, resolved backend, dialect, trust tier if present) per the design notes in agent-proxy#18. Set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental.
Structured logs. Emit a logger event per backend attempt and per terminal outcome (ok / rejected / failed / truncation-avoided / validation-failure), carrying the same dims. These are the localhost docker logs/Dozzle high-detail stream.
Keep it best-effort: no endpoint set -> no-op tracer, exactly as today. Spans/logs must never break the request path.

Detail vs redaction

Full span/log detail (prompt/completion/tool payloads) is for a local OTLP backend only - the shared fleet SigNoz (ser8:4318) has no ingest redaction. So: put bodies behind an explicit high-detail flag (e.g. PROXY_TRACE_BODIES=1, default OFF) so the default export is metadata-only and safe for any sink, and full bodies are an opt-in you enable when pointing at localhost. Document the flag.

Coordination

upstream.py, analysis.py, and resilience.py are being actively rewritten by in-flight runs: agent-proxy#3 (dialect adapter in upstream.py), #4 (intent-detection in analysis.py), #19 (reliability harness). Expect to integrate main and rebase onto their changes before landing - instrument the post-rebase shape, do not fight them.

Acceptance

With PROXY_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318, a request produces a trace whose span tree shows queue -> upstream attempt(s) -> validation -> outcome, with gen_ai.* token/finish attributes.
docker logs agent-proxy shows a structured event per backend attempt and terminal outcome.
No endpoint set -> no spans exported, request path unaffected. Tests still green.

Refs

agent-proxy#18 (the gen_ai./agentproxy. attribute design).
Pairs with the infrastructure local-signoz role (filed alongside) as the backend to view this.

## Want Make agent-proxy actually emit the per-request detail its obs layer only declares today. `app/obs.py` wires `get_tracer()` (OTLP HTTP exporter, off unless `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` is set) and `get_logger()` (structlog JSON), but there are **zero** `start_as_current_span` and **zero** `logger.*` call sites on the request path - so a SigNoz sees no traces and `docker logs` shows nothing per-request. Prometheus `/metrics` is already fully wired (24 points); this ticket adds the trace + structured-log channels so localhost high-detail inspection works. ## Ask Instrument the request path end to end: 1. **Spans.** Open a root span per inbound request in `main.py` (`/v1/chat/completions`, `/v1/completions`), with child spans wrapping the real stages: queue enqueue/wait (`queue.py`), each upstream attempt (`upstream.py`), response validation (`resilience.py`), and each retry / fallback / circuit transition (`resilience.py`). The causal chain (which backend, which attempt, why it fell back) is the whole point. 2. **Attributes.** Tag spans with OTel GenAI semconv `gen_ai.*` (request model, `gen_ai.usage.input_tokens`/`output_tokens`, `response.finish_reasons`) plus `agentproxy.*` (logical model, resolved backend, dialect, trust tier if present) per the design notes in agent-proxy#18. Set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`. 3. **Structured logs.** Emit a `logger` event per backend attempt and per terminal outcome (ok / rejected / failed / truncation-avoided / validation-failure), carrying the same dims. These are the localhost `docker logs`/Dozzle high-detail stream. 4. Keep it **best-effort**: no endpoint set -> no-op tracer, exactly as today. Spans/logs must never break the request path. ## Detail vs redaction Full span/log detail (prompt/completion/tool payloads) is for a **local** OTLP backend only - the shared fleet SigNoz (`ser8:4318`) has no ingest redaction. So: put bodies behind an explicit high-detail flag (e.g. `PROXY_TRACE_BODIES=1`, default OFF) so the default export is metadata-only and safe for any sink, and full bodies are an opt-in you enable when pointing at localhost. Document the flag. ## Coordination `upstream.py`, `analysis.py`, and `resilience.py` are being actively rewritten by in-flight runs: agent-proxy#3 (dialect adapter in upstream.py), #4 (intent-detection in analysis.py), #19 (reliability harness). Expect to integrate `main` and rebase onto their changes before landing - instrument the post-rebase shape, do not fight them. ## Acceptance - With `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`, a request produces a trace whose span tree shows queue -> upstream attempt(s) -> validation -> outcome, with gen_ai.* token/finish attributes. - `docker logs agent-proxy` shows a structured event per backend attempt and terminal outcome. - No endpoint set -> no spans exported, request path unaffected. Tests still green. ## Refs - agent-proxy#18 (the gen_ai.*/agentproxy.* attribute design). - Pairs with the infrastructure local-signoz role (filed alongside) as the backend to view this.

coilyco-ops commented

2026-07-02 09:31:26 +00:00

Member

🔒 Reserved by ward agent --driver codex — container engineer-codex-agent-proxy-28 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T09:31:24Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Codex, via ward agent

🔒 Reserved by `ward agent --driver codex` — container `engineer-codex-agent-proxy-28` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T09:31:24Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Codex, via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 09:37:21 +00:00

Instrument request path observability

coilysiren closed this issue

2026-07-02 09:37:21 +00:00

coilysiren commented

2026-07-02 09:37:39 +00:00

Author

Owner

WARD-OUTCOME: done - added request-path spans and structured logs
The work stayed mostly mechanical once the request flow was mapped. The only friction was keeping the async span boundaries and the no-op path aligned without breaking the existing tests. Confidence is good, but the stream path still deserves a little real-world smoke testing with OTLP enabled.

WARD-OUTCOME: done - added request-path spans and structured logs The work stayed mostly mechanical once the request flow was mapped. The only friction was keeping the async span boundaries and the no-op path aligned without breaking the existing tests. Confidence is good, but the stream path still deserves a little real-world smoke testing with OTLP enabled.