Add opt-in OpenTelemetry traces and metrics #43

New issue

Closed

opened 2026-08-01 22:31:38 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-01 22:31:38 +00:00

Member

Problem

ward-mcp currently emits no application-level traces or metrics. The deployed consumers therefore expose only workload and log-level evidence, even though every guarded call converges on a small set of shared MCP and HTTP handler paths.

OpenTelemetry now defines MCP-native server and client spans, context propagation through params._meta, and mcp.server.operation.duration / mcp.client.operation.duration metrics. ward-mcp should implement that contract once in the generic runtime so every spec-backed, SSM, and upstream-proxy consumer inherits it.

Reference: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md

Runtime contract

Configuration uses standard OTEL_* environment variables.
With no explicit exporter or endpoint configuration, startup remains a no-network no-op. Do not let autoexport's default OTLP selector dial localhost.
Honor OTEL_SDK_DISABLED=true, OTEL_TRACES_EXPORTER=none, and OTEL_METRICS_EXPORTER=none.
Explicit invalid telemetry configuration fails startup with a useful error. Export-time failure never changes a tool result.
Default resource identity uses the resolved MCP server name. OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES override or extend it.
Shutdown flushes providers within a bounded timeout.

Signals

MCP server

Emit one SERVER span for each MCP request or notification.
Use the current MCP semantic conventions, including mcp.method.name.
For tools/call, set gen_ai.operation.name=execute_tool and the low-cardinality gen_ai.tool.name.
A CallToolResult with isError=true sets span status ERROR and error.type=tool_error.
Emit mcp.server.operation.duration with the recommended explicit bucket boundaries.
Extract W3C trace context and baggage from params._meta as the remote parent. Link the ambient HTTP transport context when present, as the MCP convention specifies.

Upstream proxy client

serve-upstream emits MCP CLIENT spans and mcp.client.operation.duration.
Inject W3C trace context and baggage into upstream params._meta.
Preserve the inbound server span to outbound client span chain across allowlist checks, schema refresh, and the actual tool call.

Direct HTTP tool API

Instrument POST /api/{tool-name} with standard HTTP server telemetry.
Emit one logical execute_tool <tool-name> span through the shared tool wrapper.
Do not duplicate the MCP tools/call execution span.
Exclude /healthz from application telemetry noise. Admin endpoints may retain ordinary HTTP telemetry.

Safety and cardinality

Never capture tool arguments, tool results, request or response bodies, authorization headers, tokens, Guardfile contents, spec paths, or upstream URLs.
Keep metric attributes to bounded method, tool, transport, mode, and error classifications.
Do not place request IDs, session IDs, client addresses, URLs, or free-form error messages on metrics.
Record exceptions on the active span with a closed-set error classification. Dynamic text remains in structured logs only.

Implementation anchors

cmd/ward-mcp/main.go owns provider initialization, explicit configuration failure, graceful server shutdown, and bounded provider flush.
internal/mcpserver.Server.installCallRewrite is the existing receiving-middleware convergence point for MCP methods.
The shared entries in Server.handlers cover both MCP and /api/{tool-name} execution.
internal/mcpserver/upstream.go owns upstream client spans and propagation.
Keep cli-guard responsible for guarded HTTP execution. Do not reimplement or widen policy.

Acceptance

In-memory exporter tests prove successful and failed tool calls for the MCP and direct HTTP surfaces.
Tests prove the three runtime constructors inherit the shared instrumentation path.
Proxy tests prove parent-child linkage and params._meta propagation without schema or allowlist drift.
Tests prove no arguments, results, credentials, spec paths, or upstream URLs enter telemetry.
Tests prove an unset environment and explicit none selectors stay no-op.
Tests prove provider shutdown is bounded and flushes completed telemetry.
ward exec test and ward exec vet pass.
README and docs/FEATURES.md document the shipped feature and configuration.
The auth-neutral chart documents the standard environment variables but carries no fleet endpoint.

Ownership boundary

This issue owns generic runtime instrumentation and the auth-neutral chart contract. Fleet endpoints, service identities, immutable image pins, rollout order, live verification, and rollback remain in coilyco-bridge/deploy.

## Problem ward-mcp currently emits no application-level traces or metrics. The deployed consumers therefore expose only workload and log-level evidence, even though every guarded call converges on a small set of shared MCP and HTTP handler paths. OpenTelemetry now defines MCP-native server and client spans, context propagation through `params._meta`, and `mcp.server.operation.duration` / `mcp.client.operation.duration` metrics. ward-mcp should implement that contract once in the generic runtime so every spec-backed, SSM, and upstream-proxy consumer inherits it. Reference: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md ## Runtime contract * Configuration uses standard `OTEL_*` environment variables. * With no explicit exporter or endpoint configuration, startup remains a no-network no-op. Do not let autoexport's default OTLP selector dial localhost. * Honor `OTEL_SDK_DISABLED=true`, `OTEL_TRACES_EXPORTER=none`, and `OTEL_METRICS_EXPORTER=none`. * Explicit invalid telemetry configuration fails startup with a useful error. Export-time failure never changes a tool result. * Default resource identity uses the resolved MCP server name. `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` override or extend it. * Shutdown flushes providers within a bounded timeout. ## Signals ### MCP server * Emit one SERVER span for each MCP request or notification. * Use the current MCP semantic conventions, including `mcp.method.name`. * For `tools/call`, set `gen_ai.operation.name=execute_tool` and the low-cardinality `gen_ai.tool.name`. * A `CallToolResult` with `isError=true` sets span status ERROR and `error.type=tool_error`. * Emit `mcp.server.operation.duration` with the recommended explicit bucket boundaries. * Extract W3C trace context and baggage from `params._meta` as the remote parent. Link the ambient HTTP transport context when present, as the MCP convention specifies. ### Upstream proxy client * `serve-upstream` emits MCP CLIENT spans and `mcp.client.operation.duration`. * Inject W3C trace context and baggage into upstream `params._meta`. * Preserve the inbound server span to outbound client span chain across allowlist checks, schema refresh, and the actual tool call. ### Direct HTTP tool API * Instrument `POST /api/{tool-name}` with standard HTTP server telemetry. * Emit one logical `execute_tool <tool-name>` span through the shared tool wrapper. * Do not duplicate the MCP `tools/call` execution span. * Exclude `/healthz` from application telemetry noise. Admin endpoints may retain ordinary HTTP telemetry. ## Safety and cardinality * Never capture tool arguments, tool results, request or response bodies, authorization headers, tokens, Guardfile contents, spec paths, or upstream URLs. * Keep metric attributes to bounded method, tool, transport, mode, and error classifications. * Do not place request IDs, session IDs, client addresses, URLs, or free-form error messages on metrics. * Record exceptions on the active span with a closed-set error classification. Dynamic text remains in structured logs only. ## Implementation anchors * `cmd/ward-mcp/main.go` owns provider initialization, explicit configuration failure, graceful server shutdown, and bounded provider flush. * `internal/mcpserver.Server.installCallRewrite` is the existing receiving-middleware convergence point for MCP methods. * The shared entries in `Server.handlers` cover both MCP and `/api/{tool-name}` execution. * `internal/mcpserver/upstream.go` owns upstream client spans and propagation. * Keep cli-guard responsible for guarded HTTP execution. Do not reimplement or widen policy. ## Acceptance * In-memory exporter tests prove successful and failed tool calls for the MCP and direct HTTP surfaces. * Tests prove the three runtime constructors inherit the shared instrumentation path. * Proxy tests prove parent-child linkage and `params._meta` propagation without schema or allowlist drift. * Tests prove no arguments, results, credentials, spec paths, or upstream URLs enter telemetry. * Tests prove an unset environment and explicit `none` selectors stay no-op. * Tests prove provider shutdown is bounded and flushes completed telemetry. * `ward exec test` and `ward exec vet` pass. * README and `docs/FEATURES.md` document the shipped feature and configuration. * The auth-neutral chart documents the standard environment variables but carries no fleet endpoint. ## Ownership boundary This issue owns generic runtime instrumentation and the auth-neutral chart contract. Fleet endpoints, service identities, immutable image pins, rollout order, live verification, and rollback remain in `coilyco-bridge/deploy`.

coilyco-ops commented

2026-08-01 22:32:33 +00:00

Author

Member

Fleet rollout, canary order, live verification, and rollback are tracked in coilyco-bridge/deploy#297.

coilyco-ops added the

priority

label

2026-08-01 22:33:01 +00:00

coilyco-ops referenced this issue from a commit

2026-08-01 23:08:15 +00:00

feat(telemetry): add opt-in OpenTelemetry signals