OTel-instrument the dispatch lifecycle (trace as verification artifact) #135

New issue

Closed

opened 2026-06-18 02:34:54 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-18 02:34:54 +00:00

Owner

Summary

Instrument the ward dispatch / ward agent lifecycle with OpenTelemetry so a whole dispatch is one trace tree: container spin-up → fresh-clone → worker tool-calls → review panel → CI → merge. Emit against the OTel GenAI / OpenInference semantic conventions so the spans are portable across the OTel-native backends already in use (Phoenix, Honeycomb). Reviewer verdicts and CI results land as span attributes.

The goal is to make a diff interrogable rather than something the operator reads logs to verify. Verification becomes a trace query, not a log read - which is the actual bottleneck (see companion issue #134).

Motivation

Raw docker logs (dozzle/lazydocker handle the observation layer fine) answers "what is it printing." It does not answer the verification questions: which tool calls touched files outside the issue's scope fence, what the panel scored and why, where the worker looped twice on the same failing approach. Those are span queries, not log scrolls. The trace doubles as the audit artifact - one object serves observation, verification, and audit.

This is also the o2r (otel-a2a-relay) convergence point: o2r already speaks OTel, so agent-to-agent relay messages become span links/events in the same trace tree as execution. Agent comms and agent execution unify under one trace instead of living in separate channels.

Design

Trace shape. Root span per dispatch (ref, surface, harness). Child spans: container-up, clone, each worker tool-call, each panel reviewer, CI, merge. The trace id is the dispatch's durable handle.
Semantic conventions. Emit gen_ai.* (OTel GenAI) + OpenInference attributes for LLM-call spans. Pin a semconv version - the LLM-call conventions are stable but the agent/tool-level conventions were still moving; expect churn and isolate the mapping so a spec bump is a one-file change.
Verdicts as attributes. Panel reviewer outputs (verdict, reason, confidence, model family) attach as span attributes on the dispatch trace, so the merge gate in the companion issue can be expressed as a query and the audit row reconstructs from the trace.
Exporter is config, not hardcode. OTLP exporter targeting a configurable endpoint; default to the local Phoenix collector. No vendor lock - any OTel backend consumes it.
o2r seam. Where the relay is involved, emit span links between the dispatch trace and the relay envelope so the two render in one tree. (Full o2r integration can be a follow-up; this issue lands the span-link seam.)

Acceptance

A dispatch emits a single OTel trace covering spin-up → clone → worker tool-calls → panel → CI → merge.
LLM-call spans carry gen_ai.* / OpenInference attributes against a pinned semconv version.
Panel verdicts attach as span attributes (interlocks with #134).
OTLP exporter endpoint is configurable; default targets local Phoenix.
Span-link seam exists between dispatch trace and o2r relay envelopes (full o2r wiring may be a follow-up).
Docs: a docs/dispatch-otel.md walkthrough showing the trace in Phoenix/Honeycomb and the three example verification queries (scope-fence escape, panel verdict, repeated-failure loop), linked from docs/FEATURES.md.

Non-goals

Building a bespoke dashboard - Phoenix/Honeycomb render the traces; dozzle owns raw stdout.
Full o2r relay-message integration beyond the span-link seam (follow-up).
Backend selection / hosting decisions - exporter is config; this issue does not pick a vendor.

## Summary Instrument the `ward dispatch` / `ward agent` lifecycle with OpenTelemetry so a whole dispatch is **one trace tree**: container spin-up → fresh-clone → worker tool-calls → review panel → CI → merge. Emit against the OTel GenAI / OpenInference semantic conventions so the spans are portable across the OTel-native backends already in use (Phoenix, Honeycomb). Reviewer verdicts and CI results land as span attributes. The goal is to make a diff **interrogable** rather than something the operator reads logs to verify. Verification becomes a trace query, not a log read - which is the actual bottleneck (see companion issue #134). ## Motivation Raw `docker logs` (dozzle/lazydocker handle the observation layer fine) answers "what is it printing." It does not answer the verification questions: which tool calls touched files outside the issue's scope fence, what the panel scored and why, where the worker looped twice on the same failing approach. Those are span queries, not log scrolls. The trace doubles as the audit artifact - one object serves observation, verification, and audit. This is also the o2r (otel-a2a-relay) convergence point: o2r already speaks OTel, so agent-to-agent relay messages become span links/events in the **same** trace tree as execution. Agent comms and agent execution unify under one trace instead of living in separate channels. ## Design - **Trace shape.** Root span per dispatch (`ref`, surface, harness). Child spans: container-up, clone, each worker tool-call, each panel reviewer, CI, merge. The trace id is the dispatch's durable handle. - **Semantic conventions.** Emit `gen_ai.*` (OTel GenAI) + OpenInference attributes for LLM-call spans. **Pin a semconv version** - the LLM-call conventions are stable but the agent/tool-level conventions were still moving; expect churn and isolate the mapping so a spec bump is a one-file change. - **Verdicts as attributes.** Panel reviewer outputs (`verdict`, `reason`, `confidence`, model family) attach as span attributes on the dispatch trace, so the merge gate in the companion issue can be expressed as a query and the audit row reconstructs from the trace. - **Exporter is config, not hardcode.** OTLP exporter targeting a configurable endpoint; default to the local Phoenix collector. No vendor lock - any OTel backend consumes it. - **o2r seam.** Where the relay is involved, emit span links between the dispatch trace and the relay envelope so the two render in one tree. (Full o2r integration can be a follow-up; this issue lands the span-link seam.) ## Acceptance - A dispatch emits a single OTel trace covering spin-up → clone → worker tool-calls → panel → CI → merge. - LLM-call spans carry `gen_ai.*` / OpenInference attributes against a pinned semconv version. - Panel verdicts attach as span attributes (interlocks with #134). - OTLP exporter endpoint is configurable; default targets local Phoenix. - Span-link seam exists between dispatch trace and o2r relay envelopes (full o2r wiring may be a follow-up). - Docs: a `docs/dispatch-otel.md` walkthrough showing the trace in Phoenix/Honeycomb and the three example verification queries (scope-fence escape, panel verdict, repeated-failure loop), linked from `docs/FEATURES.md`. ## Non-goals - Building a bespoke dashboard - Phoenix/Honeycomb render the traces; dozzle owns raw stdout. - Full o2r relay-message integration beyond the span-link seam (follow-up). - Backend selection / hosting decisions - exporter is config; this issue does not pick a vendor.

coilysiren referenced this issue

2026-06-18 02:35:15 +00:00

In-container adversarial-review quorum gate before PR opens #134

coilysiren added the

consult

labels

2026-06-18 06:17:22 +00:00

coilyco-ops added

interactive

and removed

consult

labels

2026-06-24 04:11:55 +00:00

coilyco-ops commented

2026-06-24 17:26:17 +00:00

Member

Re-triage (goose-triage method, claude-macos-kais-macbook-pro-2 as the judgment engine, 2026-06-24)

P3 - OTel-instrument the dispatch lifecycle, real observability feature, not launch-blocking, kept.
interactive - Large instrumentation build with a semantic-convention design, agent works it and pauses.

**Re-triage** (goose-triage method, claude-macos-kais-macbook-pro-2 as the judgment engine, 2026-06-24) - **P3** - OTel-instrument the dispatch lifecycle, real observability feature, not launch-blocking, kept. - **interactive** - Large instrumentation build with a semantic-convention design, agent works it and pauses.

coilyco-ops commented

2026-06-24 17:28:34 +00:00

Member

Re-triage (goose-triage method, claude-macos-kais-macbook-pro-2 as the judgment engine, 2026-06-24)

P3 - OTel-instrument the dispatch lifecycle, real observability feature, not launch-blocking, kept.
interactive - Large instrumentation build with a semantic-convention design, agent works it and pauses.