coilyco-flight-deck/agent-proxy

Fork 0

Build out agent-proxy observability (thin OTLP today -> full traces/metrics/logs) #29

New issue

Closed

opened 2026-07-02 10:09:55 +00:00 by coilysiren · 4 comments

coilysiren commented

2026-07-02 10:09:55 +00:00

Owner

Problem

agent-proxy speaks OTLP but emits very little. We want rich observability so the host-local SigNoz backend (coilyco-flight-deck/infrastructure#435, wiring in coilyco-flight-deck/infrastructure#436) is actually useful for high-detail debugging of the proxy's request path.

First step: advisor design pass

Before implementing, an advisor run should read the current code and produce a concrete plan ON THIS THREAD:

Inventory what agent-proxy emits today (spans, metrics, logs, which OTLP signals, what SDK/exporter, where it reads the endpoint env).
Confirm the endpoint env contract with infrastructure#436 (PROXY_OTEL_EXPORTER_OTLP_ENDPOINT, an HTTP OTLP collector URL).
Propose the full observability surface across the areas below, ranked by value, with rough effort per item, so it can be split into engineer issues.

Desired surface (advisor to confirm/refine)

Request-path traces: a span per proxied request with child spans for upstream/tower call, retries, routing decision, timing, status and error attributes. This is the payoff for the unsampled local SigNoz.
Metrics: request latency histograms, error and retry counters, in-flight/concurrency gauge, per-model and per-route breakdowns, token/usage counters if the proxy sees them.
Structured logs: OTLP log records carrying the request/trace id so logs correlate to traces in SigNoz, not just stdout.
GenAI/LLM OTel semantic conventions: use the standard gen_ai.* attributes (model, prompt/completion tokens, latency) so it renders correctly in SigNoz's LLM views.

Constraints

The local SigNoz is unsampled and may hold full request/response bodies (loopback-only). Body capture must be opt-in/guarded so it does not leak to the shared fleet SigNoz on ser8 (redacted/aggregate).
Keep the exporter endpoint fully env-driven (no hardcoded host), per the infrastructure#436 contract.

Context

coilyco-flight-deck/infrastructure#435 (local SigNoz role), #436 (proxy->SigNoz OTLP wiring), #430/#434 (proxy role). See infrastructure docs/signoz-local.md and docs/agent-proxy.md.

## Problem agent-proxy speaks OTLP but emits very little. We want rich observability so the host-local SigNoz backend (coilyco-flight-deck/infrastructure#435, wiring in coilyco-flight-deck/infrastructure#436) is actually useful for high-detail debugging of the proxy's request path. ## First step: advisor design pass Before implementing, an advisor run should read the current code and produce a concrete plan ON THIS THREAD: 1. Inventory what agent-proxy emits today (spans, metrics, logs, which OTLP signals, what SDK/exporter, where it reads the endpoint env). 2. Confirm the endpoint env contract with infrastructure#436 (`PROXY_OTEL_EXPORTER_OTLP_ENDPOINT`, an HTTP OTLP collector URL). 3. Propose the full observability surface across the areas below, ranked by value, with rough effort per item, so it can be split into engineer issues. ## Desired surface (advisor to confirm/refine) - **Request-path traces**: a span per proxied request with child spans for upstream/tower call, retries, routing decision, timing, status and error attributes. This is the payoff for the unsampled local SigNoz. - **Metrics**: request latency histograms, error and retry counters, in-flight/concurrency gauge, per-model and per-route breakdowns, token/usage counters if the proxy sees them. - **Structured logs**: OTLP log records carrying the request/trace id so logs correlate to traces in SigNoz, not just stdout. - **GenAI/LLM OTel semantic conventions**: use the standard `gen_ai.*` attributes (model, prompt/completion tokens, latency) so it renders correctly in SigNoz's LLM views. ## Constraints - The local SigNoz is unsampled and may hold full request/response bodies (loopback-only). Body capture must be opt-in/guarded so it does not leak to the shared fleet SigNoz on ser8 (redacted/aggregate). - Keep the exporter endpoint fully env-driven (no hardcoded host), per the infrastructure#436 contract. ## Context coilyco-flight-deck/infrastructure#435 (local SigNoz role), #436 (proxy->SigNoz OTLP wiring), #430/#434 (proxy role). See infrastructure `docs/signoz-local.md` and `docs/agent-proxy.md`.

coilysiren commented

2026-07-02 10:12:10 +00:00

Author

Owner

Infra side of infrastructure#436 has landed on main. Recording the env var contract and a concrete finding for the step-2 endpoint check on this thread.

Contract established by infrastructure#436

Env var: PROXY_OTEL_EXPORTER_OTLP_ENDPOINT (matches your Settings.otel_exporter_otlp_endpoint, env_prefix="PROXY_").
Value shape: an OTLP/HTTP base collector URL, no signal path. Default the role writes: http://host.docker.internal:4318.
host.docker.internal, not localhost: agent-proxy runs in a container while the host-local SigNoz binds 127.0.0.1 on the host, so inside the container localhost is the container itself. The role's compose also adds extra_hosts: host.docker.internal:host-gateway so the name resolves on Linux too.
Empty/unset disables export (fine, since _configure_otel degrades to a no-op tracer).

Finding worth a look before/during the build-out

app/obs.py::_configure_otel passes the endpoint straight through: OTLPSpanExporter(endpoint=endpoint).

The opentelemetry HTTP exporter only appends /v1/traces when it resolves the endpoint from the OTEL_EXPORTER_OTLP_ENDPOINT env var. When endpoint= is passed explicitly to the constructor it is used verbatim (no signal path appended). So with the base value the role now supplies (http://host.docker.internal:4318), spans would POST to the root path and SigNoz's OTLP/HTTP receiver would reject them, so nothing lands.

Two clean fixes, either works:

Append the signal path when building the exporter: endpoint.rstrip("/") + "/v1/traces".
Or feed the base URL to the SDK via the standard OTEL_EXPORTER_OTLP_ENDPOINT env and let the SDK append per-signal paths (also gets metrics/logs exporters for free later).

Option 2 fits the "full traces/metrics/logs" goal better. I could not run the mac converge end-to-end from the infra container to confirm the 404, so flagging it from a code read rather than an observed failure.

Infra side of infrastructure#436 has landed on `main`. Recording the env var contract and a concrete finding for the step-2 endpoint check on this thread. **Contract established by infrastructure#436** - Env var: `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` (matches your `Settings.otel_exporter_otlp_endpoint`, `env_prefix="PROXY_"`). - Value shape: an OTLP/HTTP **base** collector URL, no signal path. Default the role writes: `http://host.docker.internal:4318`. - `host.docker.internal`, not `localhost`: agent-proxy runs in a container while the host-local SigNoz binds `127.0.0.1` on the host, so inside the container `localhost` is the container itself. The role's compose also adds `extra_hosts: host.docker.internal:host-gateway` so the name resolves on Linux too. - Empty/unset disables export (fine, since `_configure_otel` degrades to a no-op tracer). **Finding worth a look before/during the build-out** `app/obs.py::_configure_otel` passes the endpoint straight through: `OTLPSpanExporter(endpoint=endpoint)`. The opentelemetry HTTP exporter only appends `/v1/traces` when it resolves the endpoint from the `OTEL_EXPORTER_OTLP_ENDPOINT` env var. When `endpoint=` is passed **explicitly** to the constructor it is used **verbatim** (no signal path appended). So with the base value the role now supplies (`http://host.docker.internal:4318`), spans would POST to the root path and SigNoz's OTLP/HTTP receiver would reject them, so nothing lands. Two clean fixes, either works: 1. Append the signal path when building the exporter: `endpoint.rstrip("/") + "/v1/traces"`. 2. Or feed the base URL to the SDK via the standard `OTEL_EXPORTER_OTLP_ENDPOINT` env and let the SDK append per-signal paths (also gets metrics/logs exporters for free later). Option 2 fits the "full traces/metrics/logs" goal better. I could not run the mac converge end-to-end from the infra container to confirm the 404, so flagging it from a code read rather than an observed failure.

coilyco-ops commented

2026-07-02 10:13:56 +00:00

Member

🔎 ward agent advisor

ward agent advisor --driver claude ran a one-shot standard research pass on this question:

Read the current agent-proxy code and produce a concrete observability buildout plan on issue #29. Inventory what OTLP the proxy emits today (signals, SDK/exporter, how it reads PROXY_OTEL_EXPORTER_OTLP_ENDPOINT), then propose the full surface (request-path traces, metrics, structured logs correlated by trace id, GenAI gen_ai.* semantic conventions), ranked by value with rough effort each, split into implementable engineer issues. Respect the loopback-only unsampled local SigNoz vs redacted fleet SigNoz distinction: body capture must be opt-in and must not leak to the fleet. Confirm the endpoint env contract with coilyco-flight-deck/infrastructure#436.

Advisor design pass - agent-proxy observability (#29)

I read the current app/ tree at 4b71877. Headline: the request-path span tree is already written and quite complete, but the proxy exports zero OTLP today because the OpenTelemetry SDK is not an installed dependency. Every OTel import is a best-effort try/except that currently fails, so _tracer is None and every hot path takes the if tracer is None branch. The single highest-value item is not writing new spans - it is installing the deps so the spans that already exist light up. Below is the inventory, the endpoint-contract confirmation, and the ranked buildout split into implementable engineer issues (all in agent-proxy, so filed as one plan rather than fanned out across repos).

1. What agent-proxy emits today

Traces (OTLP) - scaffolded, but a no-op at runtime. app/obs.py:_configure_otel builds a TracerProvider with Resource{service.name} and, when an endpoint is set, a BatchSpanProcessor(OTLPSpanExporter(endpoint=...)) from opentelemetry.exporter.otlp.proto.http.trace_exporter (HTTP OTLP). The span tree already implemented across the request path:

request.chat / request.completions (root, app/main.py) - sets agentproxy.logical_model, gen_ai.request.model, agentproxy.request_kind, agentproxy.request_id, and on completion gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, response.finish_reasons.
queue.wait (app/queue.py) - queue-admission span with the same gen_ai usage attrs.
resilience.attempt (app/resilience.py, one per retry) - agentproxy.backend, agentproxy.backend_dialect, agentproxy.resolved_backend, agentproxy.attempt, agentproxy.outcome (ok / failed / validation-failure), agentproxy.validation_reason, plus record_exception on failure.
upstream.chat / upstream.chat_stream (app/upstream.py) - the actual backend call, records exception and agentproxy.upstream.error.

Body capture is already gated: PROXY_TRACE_BODIES (default off) is the only thing that puts agentproxy.messages / agentproxy.prompt / agentproxy.tools / agentproxy.options on the root span. FastAPIInstrumentor and HTTPXClientInstrumentor are wired in the lifespan, also behind try/except.

The blocker: grep opentelemetry pyproject.toml uv.lock returns 0 in both files. sentry_sdk is also absent. So none of the above executes - _configure_otel throws ImportError inside its try, _tracer stays None, the instrumentors no-op, and Sentry no-ops. The proxy currently produces no OTLP traces, no OTLP metrics, and no OTLP logs at all. The issue's framing ("speaks OTLP but emits very little") is optimistic - the wiring is there but dark.

Metrics - real, but Prometheus-pull, not OTLP. prometheus-client is a declared dep and is fully wired. All ten instruments are actually recorded across the path: llm_requests_total{logical_model,outcome}, llm_prompt_tokens histogram, llm_upstream_latency_seconds{logical_model,backend} histogram, llm_queue_depth gauge, llm_queue_rejected_total, llm_retries_total, llm_fallbacks_total, llm_circuit_state{backend} gauge, llm_truncation_avoided_total, llm_validation_failures_total{reason}. Exposed at GET /metrics (app/main.py:73). This is a scrape surface, not OTLP push - SigNoz can ingest it only via a Prometheus receiver, which is a separate ingestion path from the OTLP metrics endpoint.

Logs - structlog JSON to stdout only. app/obs.py:_configure_structlog emits JSON to stdout via PrintLoggerFactory. There is no OTLP log exporter and no trace-id/span-id injected into log records, so logs cannot correlate to traces in SigNoz - they are stdout only (container logs), decoupled from the trace view.

2. Endpoint env contract vs infrastructure#436

Confirmed and mostly aligned. app/config.py uses env_prefix=\"PROXY_\" with field otel_exporter_otlp_endpoint, so the env var is PROXY_OTEL_EXPORTER_OTLP_ENDPOINT exactly as #436 states, it is fully env-driven with a \"\" default (no hardcoded host), and it degrades to no-op when unset. The exporter is HTTP OTLP, matching the "HTTP OTLP collector URL" contract.

One caveat to confirm with #436 before it bites. The code passes the value straight to OTLPSpanExporter(endpoint=endpoint). When you pass endpoint= to the constructor explicitly, the SDK uses it verbatim - it does not append the /v1/traces signal path the way the standard OTEL_EXPORTER_OTLP_ENDPOINT base-URL convention does. So if #436 provides a base collector URL (e.g. http://127.0.0.1:4318), the proxy will POST to http://127.0.0.1:4318 with no /v1/traces suffix and the collector will 404. Resolve one of two ways: (a) have the proxy honor the base-URL convention itself (read via the SDK's env handling, or append /v1/traces, /v1/metrics, /v1/logs per signal), or (b) have #436 hand over the full per-signal URL. Option (a) is more robust and is the OTel-idiomatic choice. This gets its own small issue below (P1b).

3. Ranked buildout, split into engineer issues

All work lands in coilyco-flight-deck/agent-proxy. Effort is rough t-shirt (S ~= half day, M ~= 1-2 days, L ~= 3+ days). Ordered by value, with dependencies noted.

P0 - Install and pin the OTel SDK so existing spans export (S). Unblocks everything else.
Add runtime deps: opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http, opentelemetry-instrumentation-fastapi, opentelemetry-instrumentation-httpx (and sentry-sdk if Sentry is still wanted - otherwise delete the dead _configure_sentry). Re-lock uv.lock, confirm uv sync --frozen --no-dev in the Dockerfile still boots (there is already ward boot-probe / ward test-container for exactly this). Add a boot log line stating whether the tracer is active vs no-op no-SDK vs no-op no-endpoint, so the silent try/except degrade can never again hide a misconfigured deploy. This one item turns the entire span tree in section 1 on with zero new span code. Highest value by a wide margin.

P1b - Confirm/fix the endpoint suffix contract (S). Depends on P0.
Decide base-URL-vs-full-URL with #436 (section 2) and make the exporter construction handle it for all three signals. Land alongside P0 or immediately after.

P2 - GenAI gen_ai.* semantic-convention completeness (M). Depends on P0.
This is what makes SigNoz's LLM view render. Today only gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, response.finish_reasons are set. Add: gen_ai.system (ollama / openai per backend dialect), gen_ai.operation.name (chat / text_completion), gen_ai.response.model (the resolved backend model), gen_ai.request.temperature / top_p / max_tokens from the mapped options, gen_ai.response.id, and rename response.finish_reasons to gen_ai.response.finish_reasons. Consider aligning the root span name to the convention (chat {model}). Prompt/completion content belongs in gen_ai events/log-records, not span attributes - fold that into P5 (body capture) since it is body data.

P3 - Request-path trace enrichment (S-M). Depends on P0.
The span skeleton exists - add the debugging payload. Per-attempt: backoff delay actually waited, attempt wall-clock duration, upstream HTTP status code, a normalized error-taxonomy attribute (timeout / connection / http-4xx / http-5xx / validation). Routing decision: emit the circuit state of each candidate backend at selection time and why the chosen backend won (first-healthy vs fallback), as attributes on resilience.attempt or a short resilience.route span. This is the concrete "why did this request go where it went" payoff for the unsampled local SigNoz.

P4 - OTLP metrics (M). Depends on P0, and on the P1b endpoint decision for the metrics signal.
Decide push-vs-scrape first: either point a SigNoz Prometheus receiver at the existing /metrics (near-zero proxy work, but a second ingestion path to operate) or add a MeterProvider + PeriodicExportingMetricReader(OTLPMetricExporter) and mirror the instruments as OTel metrics (or bridge Prometheus). Recommend OTLP push for one coherent pipeline. While here, add the gaps the current set lacks: an end-to-end request latency histogram (today only upstream latency exists, not proxy-observed total), an in-flight/concurrency gauge, and a per-route (endpoint) label. Token/usage counters can be derived from the gen_ai usage already on spans, or added as counters here.

P5 - Structured logs over OTLP, correlated by trace id (M). Depends on P0.
Add a LoggerProvider + OTLP log exporter and a structlog processor that pulls trace_id / span_id from the active span context into every record, so SigNoz can jump log<->trace. Bridge the existing structlog chain to OTLP logs rather than replacing stdout (keep stdout for kubectl logs). This is where gen_ai prompt/completion content should land when body capture is on (as log records / events), keeping heavy bodies off span attributes.

P6 - Body-capture leak safety, the hard constraint (M-L). Must land before PROXY_TRACE_BODIES is ever enabled in any shared context. Depends on P2/P5 for where bodies live.
Today there is a single exporter endpoint and a single global trace_bodies bool. The "loopback-only unsampled local vs redacted fleet" split is enforced today purely by which URL the deploy happens to set - if PROXY_TRACE_BODIES=true and the endpoint is the fleet SigNoz on ser8, full request/response bodies leak. Close this structurally, do not rely on deploy discipline:

Make body emission impossible unless the exporter endpoint is loopback - refuse to enable trace_bodies (log a hard warning and force it off) when PROXY_OTEL_EXPORTER_OTLP_ENDPOINT does not resolve to 127.0.0.1 / localhost / a unix socket.
Or split into two exporters - a local unsampled exporter that carries bodies and a fleet exporter that is always metadata-only/redacted/aggregate - so the fleet path structurally cannot receive bodies regardless of the flag.
Add a redaction hook and a size cap on captured bodies even on the local path (cap attribute/log-record size, redact obvious secret-shaped fields).

Confirm the intended topology with #436/#435: is the proxy pointed only at the local loopback SigNoz, with a separate redacted path to the fleet, or is there one endpoint? That answer decides between the two options above.

Suggested landing order

P0 -> P1b (unlock: existing traces flow to local SigNoz correctly) -> P2 + P3 (make the traces genuinely useful for LLM debugging) -> P4 (metrics) and P5 (log correlation) in parallel -> P6 before any body capture is turned on outside a pure-loopback box. P0 + P1b alone deliver most of the immediate debugging value and are less than a day of work combined, so they should ship first as a fast win, then the rest can be split among engineers.

Notes for whoever implements

Keep the try/except-to-no-op degrade pattern (obs must never block startup) but pair it with the P0 boot log line so "no telemetry" is never silent again.
docs/FEATURES.md:11 currently claims "OpenTelemetry tracer" as landed - it is wired but dark. Update it in the P0 commit to reflect reality, per the FEATURES trifecta rule.
The ward boot-probe / ward test-container checks already assert /healthz + /v1/models + /metrics. Extend the container test to assert the tracer reports active when an endpoint is set, so a future dep drop can't silently re-dark the pipeline.

Researched and posted automatically by ward agent advisor --driver claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --driver claude` ran a one-shot **standard** research pass on this question: > Read the current agent-proxy code and produce a concrete observability buildout plan on issue #29. Inventory what OTLP the proxy emits today (signals, SDK/exporter, how it reads PROXY_OTEL_EXPORTER_OTLP_ENDPOINT), then propose the full surface (request-path traces, metrics, structured logs correlated by trace id, GenAI gen_ai.* semantic conventions), ranked by value with rough effort each, split into implementable engineer issues. Respect the loopback-only unsampled local SigNoz vs redacted fleet SigNoz distinction: body capture must be opt-in and must not leak to the fleet. Confirm the endpoint env contract with coilyco-flight-deck/infrastructure#436. --- ## Advisor design pass - agent-proxy observability (#29) I read the current `app/` tree at `4b71877`. Headline: the request-path **span tree is already written and quite complete**, but the proxy **exports zero OTLP today** because the OpenTelemetry SDK is not an installed dependency. Every OTel import is a best-effort `try/except` that currently fails, so `_tracer` is `None` and every hot path takes the `if tracer is None` branch. The single highest-value item is not writing new spans - it is installing the deps so the spans that already exist light up. Below is the inventory, the endpoint-contract confirmation, and the ranked buildout split into implementable engineer issues (all in `agent-proxy`, so filed as one plan rather than fanned out across repos). ## 1. What agent-proxy emits today **Traces (OTLP) - scaffolded, but a no-op at runtime.** `app/obs.py:_configure_otel` builds a `TracerProvider` with `Resource{service.name}` and, when an endpoint is set, a `BatchSpanProcessor(OTLPSpanExporter(endpoint=...))` from `opentelemetry.exporter.otlp.proto.http.trace_exporter` (HTTP OTLP). The span tree already implemented across the request path: * `request.chat` / `request.completions` (root, `app/main.py`) - sets `agentproxy.logical_model`, `gen_ai.request.model`, `agentproxy.request_kind`, `agentproxy.request_id`, and on completion `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `response.finish_reasons`. * `queue.wait` (`app/queue.py`) - queue-admission span with the same gen_ai usage attrs. * `resilience.attempt` (`app/resilience.py`, one per retry) - `agentproxy.backend`, `agentproxy.backend_dialect`, `agentproxy.resolved_backend`, `agentproxy.attempt`, `agentproxy.outcome` (ok / failed / validation-failure), `agentproxy.validation_reason`, plus `record_exception` on failure. * `upstream.chat` / `upstream.chat_stream` (`app/upstream.py`) - the actual backend call, records exception and `agentproxy.upstream.error`. Body capture is already gated: `PROXY_TRACE_BODIES` (default off) is the only thing that puts `agentproxy.messages` / `agentproxy.prompt` / `agentproxy.tools` / `agentproxy.options` on the root span. `FastAPIInstrumentor` and `HTTPXClientInstrumentor` are wired in the lifespan, also behind `try/except`. **The blocker:** `grep opentelemetry pyproject.toml uv.lock` returns **0** in both files. `sentry_sdk` is also absent. So none of the above executes - `_configure_otel` throws `ImportError` inside its `try`, `_tracer` stays `None`, the instrumentors no-op, and Sentry no-ops. **The proxy currently produces no OTLP traces, no OTLP metrics, and no OTLP logs at all.** The issue's framing (\"speaks OTLP but emits very little\") is optimistic - the wiring is there but dark. **Metrics - real, but Prometheus-pull, not OTLP.** `prometheus-client` is a declared dep and is fully wired. All ten instruments are actually recorded across the path: `llm_requests_total{logical_model,outcome}`, `llm_prompt_tokens` histogram, `llm_upstream_latency_seconds{logical_model,backend}` histogram, `llm_queue_depth` gauge, `llm_queue_rejected_total`, `llm_retries_total`, `llm_fallbacks_total`, `llm_circuit_state{backend}` gauge, `llm_truncation_avoided_total`, `llm_validation_failures_total{reason}`. Exposed at `GET /metrics` (`app/main.py:73`). This is a **scrape** surface, not OTLP push - SigNoz can ingest it only via a Prometheus receiver, which is a separate ingestion path from the OTLP metrics endpoint. **Logs - structlog JSON to stdout only.** `app/obs.py:_configure_structlog` emits JSON to stdout via `PrintLoggerFactory`. There is **no** OTLP log exporter and **no** trace-id/span-id injected into log records, so logs cannot correlate to traces in SigNoz - they are stdout only (container logs), decoupled from the trace view. ## 2. Endpoint env contract vs infrastructure#436 Confirmed and mostly aligned. `app/config.py` uses `env_prefix=\"PROXY_\"` with field `otel_exporter_otlp_endpoint`, so the env var is **`PROXY_OTEL_EXPORTER_OTLP_ENDPOINT`** exactly as #436 states, it is fully env-driven with a `\"\"` default (no hardcoded host), and it degrades to no-op when unset. The exporter is HTTP OTLP, matching the \"HTTP OTLP collector URL\" contract. **One caveat to confirm with #436 before it bites.** The code passes the value straight to `OTLPSpanExporter(endpoint=endpoint)`. When you pass `endpoint=` to the constructor explicitly, the SDK uses it **verbatim** - it does **not** append the `/v1/traces` signal path the way the standard `OTEL_EXPORTER_OTLP_ENDPOINT` base-URL convention does. So if #436 provides a **base** collector URL (e.g. `http://127.0.0.1:4318`), the proxy will POST to `http://127.0.0.1:4318` with no `/v1/traces` suffix and the collector will 404. Resolve one of two ways: (a) have the proxy honor the base-URL convention itself (read via the SDK's env handling, or append `/v1/traces`, `/v1/metrics`, `/v1/logs` per signal), or (b) have #436 hand over the full per-signal URL. Option (a) is more robust and is the OTel-idiomatic choice. This gets its own small issue below (P1b). ## 3. Ranked buildout, split into engineer issues All work lands in `coilyco-flight-deck/agent-proxy`. Effort is rough t-shirt (S ~= half day, M ~= 1-2 days, L ~= 3+ days). Ordered by value, with dependencies noted. **P0 - Install and pin the OTel SDK so existing spans export (S). Unblocks everything else.** Add runtime deps: `opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-http`, `opentelemetry-instrumentation-fastapi`, `opentelemetry-instrumentation-httpx` (and `sentry-sdk` if Sentry is still wanted - otherwise delete the dead `_configure_sentry`). Re-lock `uv.lock`, confirm `uv sync --frozen --no-dev` in the Dockerfile still boots (there is already `ward boot-probe` / `ward test-container` for exactly this). Add a boot log line stating whether the tracer is **active** vs **no-op no-SDK** vs **no-op no-endpoint**, so the silent `try/except` degrade can never again hide a misconfigured deploy. This one item turns the entire span tree in section 1 on with zero new span code. Highest value by a wide margin. **P1b - Confirm/fix the endpoint suffix contract (S). Depends on P0.** Decide base-URL-vs-full-URL with #436 (section 2) and make the exporter construction handle it for all three signals. Land alongside P0 or immediately after. **P2 - GenAI `gen_ai.*` semantic-convention completeness (M). Depends on P0.** This is what makes SigNoz's LLM view render. Today only `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `response.finish_reasons` are set. Add: `gen_ai.system` (ollama / openai per backend dialect), `gen_ai.operation.name` (`chat` / `text_completion`), `gen_ai.response.model` (the resolved backend model), `gen_ai.request.temperature` / `top_p` / `max_tokens` from the mapped options, `gen_ai.response.id`, and rename `response.finish_reasons` to `gen_ai.response.finish_reasons`. Consider aligning the root span name to the convention (`chat {model}`). Prompt/completion content belongs in gen_ai **events/log-records**, not span attributes - fold that into P5 (body capture) since it is body data. **P3 - Request-path trace enrichment (S-M). Depends on P0.** The span skeleton exists - add the debugging payload. Per-attempt: backoff delay actually waited, attempt wall-clock duration, upstream HTTP status code, a normalized error-taxonomy attribute (timeout / connection / http-4xx / http-5xx / validation). Routing decision: emit the circuit state of each candidate backend at selection time and why the chosen backend won (first-healthy vs fallback), as attributes on `resilience.attempt` or a short `resilience.route` span. This is the concrete \"why did this request go where it went\" payoff for the unsampled local SigNoz. **P4 - OTLP metrics (M). Depends on P0, and on the P1b endpoint decision for the metrics signal.** Decide push-vs-scrape first: either point a SigNoz Prometheus receiver at the existing `/metrics` (near-zero proxy work, but a second ingestion path to operate) or add a `MeterProvider` + `PeriodicExportingMetricReader(OTLPMetricExporter)` and mirror the instruments as OTel metrics (or bridge Prometheus). Recommend OTLP push for one coherent pipeline. While here, add the gaps the current set lacks: an **end-to-end request latency** histogram (today only `upstream` latency exists, not proxy-observed total), an **in-flight/concurrency** gauge, and a per-route (endpoint) label. Token/usage counters can be derived from the gen_ai usage already on spans, or added as counters here. **P5 - Structured logs over OTLP, correlated by trace id (M). Depends on P0.** Add a `LoggerProvider` + OTLP log exporter and a structlog processor that pulls `trace_id` / `span_id` from the active span context into every record, so SigNoz can jump log<->trace. Bridge the existing structlog chain to OTLP logs rather than replacing stdout (keep stdout for `kubectl logs`). This is where gen_ai prompt/completion content should land when body capture is on (as log records / events), keeping heavy bodies off span attributes. **P6 - Body-capture leak safety, the hard constraint (M-L). Must land before `PROXY_TRACE_BODIES` is ever enabled in any shared context. Depends on P2/P5 for where bodies live.** Today there is a **single** exporter endpoint and a **single** global `trace_bodies` bool. The \"loopback-only unsampled local vs redacted fleet\" split is enforced today purely by which URL the deploy happens to set - if `PROXY_TRACE_BODIES=true` and the endpoint is the fleet SigNoz on ser8, full request/response bodies leak. Close this structurally, do not rely on deploy discipline: * Make body emission impossible unless the exporter endpoint is loopback - refuse to enable `trace_bodies` (log a hard warning and force it off) when `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` does not resolve to `127.0.0.1` / `localhost` / a unix socket. * Or split into two exporters - a local unsampled exporter that carries bodies and a fleet exporter that is always metadata-only/redacted/aggregate - so the fleet path structurally cannot receive bodies regardless of the flag. * Add a redaction hook and a size cap on captured bodies even on the local path (cap attribute/log-record size, redact obvious secret-shaped fields). Confirm the intended topology with #436/#435: is the proxy pointed only at the local loopback SigNoz, with a separate redacted path to the fleet, or is there one endpoint? That answer decides between the two options above. ## Suggested landing order P0 -> P1b (unlock: existing traces flow to local SigNoz correctly) -> P2 + P3 (make the traces genuinely useful for LLM debugging) -> P4 (metrics) and P5 (log correlation) in parallel -> P6 before any body capture is turned on outside a pure-loopback box. P0 + P1b alone deliver most of the immediate debugging value and are less than a day of work combined, so they should ship first as a fast win, then the rest can be split among engineers. ## Notes for whoever implements * Keep the `try/except`-to-no-op degrade pattern (obs must never block startup) but pair it with the P0 boot log line so \"no telemetry\" is never silent again. * `docs/FEATURES.md:11` currently claims \"OpenTelemetry tracer\" as landed - it is wired but dark. Update it in the P0 commit to reflect reality, per the FEATURES trifecta rule. * The `ward boot-probe` / `ward test-container` checks already assert `/healthz` + `/v1/models` + `/metrics`. Extend the container test to assert the tracer reports **active** when an endpoint is set, so a future dep drop can't silently re-dark the pipeline. --- Researched and posted automatically by `ward agent advisor --driver claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-02 10:15:17 +00:00

Member

🔎 ward agent advisor

ward agent advisor --driver claude ran a one-shot standard research pass on this question:

Read the current agent-proxy code and produce a concrete observability buildout plan on issue #29. Inventory what OTLP the proxy emits today, then propose the full surface (request-path traces, metrics, structured logs correlated by trace id, GenAI gen_ai.* semantic conventions), ranked by value with rough effort each, split into implementable engineer issues. Respect the loopback-only unsampled local SigNoz vs redacted fleet SigNoz distinction: body capture must be opt-in and must not leak to the fleet. Confirm the endpoint env contract with coilyco-flight-deck/infrastructure#436.

agent-proxy observability buildout — confirmed plan (#29)

This consolidates the thread (the advisor code-read at 4b71877 + the infra#436 contract comment) into a decision-resolved plan. All work is single-repo (agent-proxy), so this stays one plan, not fanned-out child issues. The three open decisions from the advisor pass are resolved below with rationale.

Headline (unchanged, and it's the whole story)

The request-path span tree is already written across app/main.py, app/queue.py, app/resilience.py, app/upstream.py, but the proxy exports zero OTLP today because the OpenTelemetry SDK is not an installed dependency (grep opentelemetry pyproject.toml uv.lock → 0). Every OTel import sits in a best-effort try/except that currently throws ImportError, so _tracer stays None and every hot path takes the no-op branch. docs/FEATURES.md:11 claiming a landed "OpenTelemetry tracer" is wired but dark. The single highest-value action is not new instrumentation — it is installing the deps so the existing spans light up.

Inventory (what's actually emitted today)

Traces (OTLP/HTTP): scaffolded, runtime no-op. Span tree already exists: request.chat/request.completions (root, with gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens, response.finish_reasons, agentproxy.* ids), queue.wait, resilience.attempt (per retry, with backend/dialect/outcome/validation attrs + record_exception), upstream.chat/upstream.chat_stream. Body capture already gated behind PROXY_TRACE_BODIES (default off). FastAPIInstrumentor + HTTPXClientInstrumentor wired but behind the same dead try/except.
Metrics: real but Prometheus-pull, not OTLP. prometheus-client is installed and all ~10 instruments record across the path (llm_requests_total, llm_prompt_tokens, llm_upstream_latency_seconds, llm_queue_depth, llm_retries_total, llm_fallbacks_total, llm_circuit_state, llm_validation_failures_total, etc.), exposed at GET /metrics. Reaches SigNoz only via a Prometheus receiver — a separate ingestion path from OTLP metrics.
Logs: structlog JSON to stdout only (_configure_structlog, PrintLoggerFactory). No OTLP log exporter, no trace-id/span-id on records → logs cannot correlate to traces in SigNoz.
Sentry: sentry_sdk also absent; _configure_sentry is dead. Decide keep-or-delete in P0.

Endpoint contract vs infrastructure#436 — CONFIRMED, with one required fix

The env var is exactly PROXY_OTEL_EXPORTER_OTLP_ENDPOINT (Settings.otel_exporter_otlp_endpoint, env_prefix="PROXY_"), fully env-driven with a "" default (no hardcoded host), degrades to no-op when unset. infra#436 has landed on main and supplies an OTLP/HTTP base URL with no signal path — default http://host.docker.internal:4318 (host.docker.internal because the proxy runs in a container while host-local SigNoz binds 127.0.0.1; compose adds extra_hosts: host.docker.internal:host-gateway for Linux).

The bug both the code-read and the infra comment flagged is real and I verified it against the OTel Python docs: OTLPSpanExporter(endpoint=endpoint) with an explicit endpoint= kwarg uses the value verbatim and does not append /v1/traces. The /v1/<signal> suffix is only auto-appended when the endpoint is resolved from the standard OTEL_EXPORTER_OTLP_ENDPOINT env var (base-URL convention). So with #436's base value, spans POST to the root path and SigNoz's OTLP/HTTP receiver rejects them — nothing lands.

DECISION 1 — resolve via the SDK env convention, not manual suffixing (Option 2). Rather than string-appending /v1/traces in the exporter constructor, have _configure_otel set OTEL_EXPORTER_OTLP_ENDPOINT (and/or per-signal OTEL_EXPORTER_OTLP_TRACES/METRICS/LOGS_ENDPOINT) from PROXY_OTEL_EXPORTER_OTLP_ENDPOINT and let the SDK append per-signal paths. This is the OTel-idiomatic path, keeps the base-URL contract with #436 intact (infra keeps handing a base URL), and gives the future metrics + logs exporters correct suffixes for free — so it composes with P4/P5 rather than needing three separate manual fixes. Manual rstrip("/") + "/v1/traces" is the fallback if env-injection proves awkward, but it must then be applied to all three signals.

Ranked buildout (engineer-issue-sized; effort S≈½day, M≈1–2d, L≈3d+)

P0 — Install & pin the OTel SDK so existing spans export (S). Unblocks everything.
Add runtime deps: opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http, opentelemetry-instrumentation-fastapi, opentelemetry-instrumentation-httpx (+ sentry-sdk only if Sentry is still wanted — otherwise delete dead _configure_sentry). Re-lock uv.lock; confirm uv sync --frozen --no-dev in the Dockerfile still boots (ward boot-probe/ward test-container exist for exactly this). Add a boot log line reporting tracer state — active / no-op (no SDK) / no-op (no endpoint) so the silent try/except degrade can never again hide a dark pipeline. Update docs/FEATURES.md:11 in the same commit (FEATURES trifecta rule). Turns on the entire span tree with zero new span code — highest value by a wide margin.

P1 — Endpoint suffix fix per DECISION 1 (S). Depends on P0. Land with or immediately after P0. Extend ward test-container to assert the tracer reports active when an endpoint is set, so a future dep drop can't silently re-dark it.

P2 — GenAI gen_ai.* semantic-convention completeness (M). Depends on P0. What makes SigNoz's LLM view render. Today only gen_ai.request.model + usage tokens + response.finish_reasons. Add gen_ai.system (ollama/openai per dialect), gen_ai.operation.name (chat/text_completion), gen_ai.response.model (resolved backend model), gen_ai.request.temperature/top_p/max_tokens from mapped options, gen_ai.response.id; rename response.finish_reasons → gen_ai.response.finish_reasons; align root span name to chat {model}. Prompt/completion content belongs in gen_ai events/log-records, not span attributes → defer content to P6 (it's body data).

P3 — Request-path trace enrichment (S–M). Depends on P0. The skeleton exists; add the debugging payload. Per resilience.attempt: actual backoff delay waited, attempt wall-clock, upstream HTTP status, a normalized error-taxonomy attr (timeout/connection/http-4xx/http-5xx/validation). Routing: circuit state of each candidate backend at selection time + why the chosen one won (first-healthy vs fallback) as attrs on resilience.attempt or a short resilience.route span. This is the concrete "why did this request go where it went" payoff for the unsampled local SigNoz.

P4 — OTLP metrics (M). Depends on P0 + the P1 endpoint mechanism. DECISION 2 — OTLP push, one coherent pipeline. Add a MeterProvider + PeriodicExportingMetricReader(OTLPMetricExporter) and mirror the Prometheus instruments as OTel metrics (or bridge), rather than standing up a second Prometheus-receiver ingestion path to operate. Keep /metrics for anything already scraping it. While here, close the gaps the current set lacks: an end-to-end proxy-observed request-latency histogram (today only upstream latency exists), an in-flight/concurrency gauge, and a per-route/endpoint label.

P5 — Structured logs over OTLP, correlated by trace id (M). Depends on P0. Add a LoggerProvider + OTLP log exporter and a structlog processor that injects trace_id/span_id from the active span context into every record so SigNoz can jump log↔trace. Bridge the existing structlog chain to OTLP (keep stdout for kubectl logs), don't replace it.

P6 — Body-capture leak safety — the hard constraint (M–L). MUST land before PROXY_TRACE_BODIES is enabled in any shared context. Depends on P2/P5 for where bodies live. Today there is one exporter endpoint and one global trace_bodies bool, so the loopback-local-vs-fleet split is enforced only by which URL a deploy sets — if PROXY_TRACE_BODIES=true and the endpoint is the ser8 fleet SigNoz, full request/response bodies leak. DECISION 3 — close it structurally, not by deploy discipline. Preferred: refuse to enable trace_bodies unless the exporter endpoint resolves to loopback (127.0.0.1/localhost/host.docker.internal→host-gateway/unix socket) — force it off and log a hard warning otherwise. Stronger alternative if a fleet path must coexist: two exporters — a loopback exporter that carries bodies and a fleet exporter that is always metadata-only/redacted/aggregate — so the fleet path structurally cannot receive bodies regardless of the flag. Either way add a redaction hook + size cap on captured bodies even on the local path. Content from P2 (prompts/completions) rides P5's log-records here, keeping heavy bodies off span attributes. Confirm topology with #435/#436: is the proxy pointed only at loopback SigNoz (separate redacted fleet path), or one shared endpoint? That answer picks between the two P6 options.

Landing order

P0 → P1 (existing traces flow correctly to local SigNoz — <1 day combined, ship as the fast win) → P2 + P3 (make traces genuinely useful for LLM debugging) → P4 and P5 in parallel → P6 gated before any body capture outside a pure-loopback box.

Notes for implementers

Keep the try/except→no-op degrade (obs must never block startup) but always pair it with the P0 boot log line so "no telemetry" is never silent again.
Sources for the verified endpoint behavior: OTel Python exporters docs, OTLP exporter config.

Researched and posted automatically by ward agent advisor --driver claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --driver claude` ran a one-shot **standard** research pass on this question: > Read the current agent-proxy code and produce a concrete observability buildout plan on issue #29. Inventory what OTLP the proxy emits today, then propose the full surface (request-path traces, metrics, structured logs correlated by trace id, GenAI gen_ai.* semantic conventions), ranked by value with rough effort each, split into implementable engineer issues. Respect the loopback-only unsampled local SigNoz vs redacted fleet SigNoz distinction: body capture must be opt-in and must not leak to the fleet. Confirm the endpoint env contract with coilyco-flight-deck/infrastructure#436. --- ## agent-proxy observability buildout — confirmed plan (#29) This consolidates the thread (the advisor code-read at `4b71877` + the infra#436 contract comment) into a decision-resolved plan. **All work is single-repo (`agent-proxy`), so this stays one plan, not fanned-out child issues.** The three open decisions from the advisor pass are resolved below with rationale. ### Headline (unchanged, and it's the whole story) The request-path **span tree is already written** across `app/main.py`, `app/queue.py`, `app/resilience.py`, `app/upstream.py`, but the proxy **exports zero OTLP today** because the OpenTelemetry SDK is not an installed dependency (`grep opentelemetry pyproject.toml uv.lock` → 0). Every OTel import sits in a best-effort `try/except` that currently throws `ImportError`, so `_tracer` stays `None` and every hot path takes the no-op branch. `docs/FEATURES.md:11` claiming a landed "OpenTelemetry tracer" is **wired but dark**. The single highest-value action is not new instrumentation — it is **installing the deps so the existing spans light up.** ### Inventory (what's actually emitted today) - **Traces (OTLP/HTTP):** scaffolded, runtime no-op. Span tree already exists: `request.chat`/`request.completions` (root, with `gen_ai.request.model`, `gen_ai.usage.input_tokens/output_tokens`, `response.finish_reasons`, `agentproxy.*` ids), `queue.wait`, `resilience.attempt` (per retry, with backend/dialect/outcome/validation attrs + `record_exception`), `upstream.chat`/`upstream.chat_stream`. Body capture already gated behind `PROXY_TRACE_BODIES` (default off). `FastAPIInstrumentor` + `HTTPXClientInstrumentor` wired but behind the same dead `try/except`. - **Metrics:** real but **Prometheus-pull, not OTLP**. `prometheus-client` is installed and all ~10 instruments record across the path (`llm_requests_total`, `llm_prompt_tokens`, `llm_upstream_latency_seconds`, `llm_queue_depth`, `llm_retries_total`, `llm_fallbacks_total`, `llm_circuit_state`, `llm_validation_failures_total`, etc.), exposed at `GET /metrics`. Reaches SigNoz only via a Prometheus receiver — a separate ingestion path from OTLP metrics. - **Logs:** structlog JSON to **stdout only** (`_configure_structlog`, `PrintLoggerFactory`). No OTLP log exporter, no trace-id/span-id on records → logs cannot correlate to traces in SigNoz. - **Sentry:** `sentry_sdk` also absent; `_configure_sentry` is dead. Decide keep-or-delete in P0. ### Endpoint contract vs infrastructure#436 — CONFIRMED, with one required fix The env var is exactly `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` (`Settings.otel_exporter_otlp_endpoint`, `env_prefix="PROXY_"`), fully env-driven with a `""` default (no hardcoded host), degrades to no-op when unset. infra#436 has **landed on `main`** and supplies an OTLP/HTTP **base** URL with no signal path — default `http://host.docker.internal:4318` (`host.docker.internal` because the proxy runs in a container while host-local SigNoz binds `127.0.0.1`; compose adds `extra_hosts: host.docker.internal:host-gateway` for Linux). **The bug both the code-read and the infra comment flagged is real and I verified it against the OTel Python docs:** `OTLPSpanExporter(endpoint=endpoint)` with an **explicit** `endpoint=` kwarg uses the value **verbatim** and does **not** append `/v1/traces`. The `/v1/<signal>` suffix is only auto-appended when the endpoint is resolved from the standard `OTEL_EXPORTER_OTLP_ENDPOINT` **env var** (base-URL convention). So with #436's base value, spans POST to the root path and SigNoz's OTLP/HTTP receiver rejects them — **nothing lands.** **DECISION 1 — resolve via the SDK env convention, not manual suffixing (Option 2).** Rather than string-appending `/v1/traces` in the exporter constructor, have `_configure_otel` set `OTEL_EXPORTER_OTLP_ENDPOINT` (and/or per-signal `OTEL_EXPORTER_OTLP_TRACES/METRICS/LOGS_ENDPOINT`) from `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` and let the SDK append per-signal paths. This is the OTel-idiomatic path, keeps the base-URL contract with #436 intact (infra keeps handing a base URL), and gives the future metrics + logs exporters correct suffixes **for free** — so it composes with P4/P5 rather than needing three separate manual fixes. Manual `rstrip("/") + "/v1/traces"` is the fallback if env-injection proves awkward, but it must then be applied to all three signals. --- ### Ranked buildout (engineer-issue-sized; effort S≈½day, M≈1–2d, L≈3d+) **P0 — Install & pin the OTel SDK so existing spans export (S). Unblocks everything.** Add runtime deps: `opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-http`, `opentelemetry-instrumentation-fastapi`, `opentelemetry-instrumentation-httpx` (+ `sentry-sdk` only if Sentry is still wanted — otherwise delete dead `_configure_sentry`). Re-lock `uv.lock`; confirm `uv sync --frozen --no-dev` in the Dockerfile still boots (`ward boot-probe`/`ward test-container` exist for exactly this). **Add a boot log line reporting tracer state — `active` / `no-op (no SDK)` / `no-op (no endpoint)`** so the silent `try/except` degrade can never again hide a dark pipeline. Update `docs/FEATURES.md:11` in the same commit (FEATURES trifecta rule). Turns on the entire span tree with zero new span code — highest value by a wide margin. **P1 — Endpoint suffix fix per DECISION 1 (S). Depends on P0.** Land with or immediately after P0. Extend `ward test-container` to assert the tracer reports `active` when an endpoint is set, so a future dep drop can't silently re-dark it. **P2 — GenAI `gen_ai.*` semantic-convention completeness (M). Depends on P0.** What makes SigNoz's LLM view render. Today only `gen_ai.request.model` + usage tokens + `response.finish_reasons`. Add `gen_ai.system` (ollama/openai per dialect), `gen_ai.operation.name` (`chat`/`text_completion`), `gen_ai.response.model` (resolved backend model), `gen_ai.request.temperature`/`top_p`/`max_tokens` from mapped options, `gen_ai.response.id`; **rename `response.finish_reasons` → `gen_ai.response.finish_reasons`**; align root span name to `chat {model}`. Prompt/completion **content** belongs in gen_ai **events/log-records, not span attributes** → defer content to P6 (it's body data). **P3 — Request-path trace enrichment (S–M). Depends on P0.** The skeleton exists; add the debugging payload. Per `resilience.attempt`: actual backoff delay waited, attempt wall-clock, upstream HTTP status, a normalized error-taxonomy attr (`timeout`/`connection`/`http-4xx`/`http-5xx`/`validation`). Routing: circuit state of each candidate backend at selection time + why the chosen one won (first-healthy vs fallback) as attrs on `resilience.attempt` or a short `resilience.route` span. This is the concrete "why did this request go where it went" payoff for the unsampled local SigNoz. **P4 — OTLP metrics (M). Depends on P0 + the P1 endpoint mechanism.** **DECISION 2 — OTLP push, one coherent pipeline.** Add a `MeterProvider` + `PeriodicExportingMetricReader(OTLPMetricExporter)` and mirror the Prometheus instruments as OTel metrics (or bridge), rather than standing up a second Prometheus-receiver ingestion path to operate. Keep `/metrics` for anything already scraping it. While here, close the gaps the current set lacks: an **end-to-end proxy-observed request-latency** histogram (today only `upstream` latency exists), an **in-flight/concurrency** gauge, and a per-route/endpoint label. **P5 — Structured logs over OTLP, correlated by trace id (M). Depends on P0.** Add a `LoggerProvider` + OTLP log exporter and a structlog processor that injects `trace_id`/`span_id` from the active span context into every record so SigNoz can jump log↔trace. **Bridge** the existing structlog chain to OTLP (keep stdout for `kubectl logs`), don't replace it. **P6 — Body-capture leak safety — the hard constraint (M–L). MUST land before `PROXY_TRACE_BODIES` is enabled in any shared context. Depends on P2/P5 for where bodies live.** Today there is **one** exporter endpoint and **one** global `trace_bodies` bool, so the loopback-local-vs-fleet split is enforced *only* by which URL a deploy sets — if `PROXY_TRACE_BODIES=true` and the endpoint is the ser8 fleet SigNoz, full request/response bodies leak. **DECISION 3 — close it structurally, not by deploy discipline.** Preferred: **refuse to enable `trace_bodies` unless the exporter endpoint resolves to loopback** (`127.0.0.1`/`localhost`/`host.docker.internal`→host-gateway/unix socket) — force it off and log a hard warning otherwise. Stronger alternative if a fleet path must coexist: **two exporters** — a loopback exporter that carries bodies and a fleet exporter that is always metadata-only/redacted/aggregate — so the fleet path structurally cannot receive bodies regardless of the flag. Either way add a **redaction hook + size cap** on captured bodies even on the local path. Content from P2 (prompts/completions) rides P5's log-records here, keeping heavy bodies off span attributes. **Confirm topology with #435/#436**: is the proxy pointed only at loopback SigNoz (separate redacted fleet path), or one shared endpoint? That answer picks between the two P6 options. ### Landing order `P0 → P1` (existing traces flow correctly to local SigNoz — <1 day combined, ship as the fast win) → `P2 + P3` (make traces genuinely useful for LLM debugging) → `P4` and `P5` in parallel → `P6` gated before any body capture outside a pure-loopback box. ### Notes for implementers - Keep the `try/except`→no-op degrade (obs must never block startup) but always pair it with the P0 boot log line so "no telemetry" is never silent again. - Sources for the verified endpoint behavior: [OTel Python exporters docs](https://opentelemetry.io/docs/languages/python/exporters/), [OTLP exporter config](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/). --- Researched and posted automatically by `ward agent advisor --driver claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Claude (she/her), via `ward agent`

coilyco-ops added the

consult

labels

2026-07-02 16:42:47 +00:00

coilysiren commented

2026-07-03 19:34:35 +00:00

Author

Owner

Director pass from the Windows tower - measured the current posture live and pinned what "thin OTLP today -> full" concretely needs.

Current posture (verified live against real ollama)

Prometheus /metrics: working. Full llm_* family emits and is now labeled
by the pass-through model name (logical_model="qwen3:4b") after #32:
llm_requests_total, llm_queue_depth, llm_retries_total,
llm_fallbacks_total, llm_circuit_state, llm_truncation_avoided_total,
llm_validation_failures_total, llm_prompt_tokens, llm_upstream_latency_seconds.
OTel traces: wired but dormant. otel_exporter_otlp_endpoint defaults empty,
so export is a graceful no-op. The upstream.chat span already sets
gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens,
and response.finish_reasons - a solid GenAI-semconv base already in place.
structlog JSON + Sentry init: present (Sentry DSN from SSM/env, off when unset).

What "full" needs, in order

A collector to export to. Set PROXY_OTEL_EXPORTER_OTLP_ENDPOINT at the
SigNoz OTLP collector. Blocked until the stack converges on this host -
tracked in coilyco-flight-deck/infrastructure#459 (SigNoz is down on the
Windows tower today: 3301/4317/4318 all refuse). End-to-end trace validation
cannot happen until that lands.
Complete the OTel GenAI gen_ai.* semconv per #18: the span base above plus
the metrics gen_ai.client.token.usage and gen_ai.client.operation.duration,
and opt into latest with OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental.
Composes with VictoriaTraces / OpenLLMetry, no private schema.
Tail-sampling per #18: ~5% sampled, 100% on errors, ML-analysis sheddable
and off the hot path.

Sequencing: steps 2-3 are proxy work dispatchable now (offline-testable);
step 1's validation waits on infra#459. Recommend the proxy work proceed in
parallel and validate against SigNoz once the collector is up.

Director pass from the Windows tower - measured the current posture live and pinned what "thin OTLP today -> full" concretely needs. **Current posture (verified live against real ollama)** - **Prometheus `/metrics`: working.** Full `llm_*` family emits and is now labeled by the pass-through model name (`logical_model="qwen3:4b"`) after #32: `llm_requests_total`, `llm_queue_depth`, `llm_retries_total`, `llm_fallbacks_total`, `llm_circuit_state`, `llm_truncation_avoided_total`, `llm_validation_failures_total`, `llm_prompt_tokens`, `llm_upstream_latency_seconds`. - **OTel traces: wired but dormant.** `otel_exporter_otlp_endpoint` defaults empty, so export is a graceful no-op. The `upstream.chat` span already sets `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and `response.finish_reasons` - a solid GenAI-semconv base already in place. - **structlog JSON + Sentry init: present** (Sentry DSN from SSM/env, off when unset). **What "full" needs, in order** 1. **A collector to export to.** Set `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` at the SigNoz OTLP collector. **Blocked** until the stack converges on this host - tracked in `coilyco-flight-deck/infrastructure#459` (SigNoz is down on the Windows tower today: 3301/4317/4318 all refuse). End-to-end trace validation cannot happen until that lands. 2. **Complete the OTel GenAI `gen_ai.*` semconv** per #18: the span base above plus the metrics `gen_ai.client.token.usage` and `gen_ai.client.operation.duration`, and opt into latest with `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`. Composes with VictoriaTraces / OpenLLMetry, no private schema. 3. **Tail-sampling** per #18: ~5% sampled, 100% on errors, ML-analysis sheddable and off the hot path. **Sequencing:** steps 2-3 are proxy work dispatchable now (offline-testable); step 1's *validation* waits on infra#459. Recommend the proxy work proceed in parallel and validate against SigNoz once the collector is up.

coilysiren closed this issue

2026-07-04 10:36:32 +00:00