Add required log+metric+span instrumentation wrapper #37

New issue

Closed

opened 2026-07-09 18:02:55 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 18:02:55 +00:00

Member

Add a small instrumentation wrapper so code paths that matter emit all three observability signals together: structured log event, Prometheus metric, and OpenTelemetry span/attributes.

Context from director surface:

agent-proxy already has structlog JSON logs, Prometheus metrics, and OTel spans in app/obs.py, with call sites spread across app/main.py, app/queue.py, app/resilience.py, and app/analysis.py.
Backend-delivered context truncation currently logs dispatch.context_truncated, increments llm_context_truncated_total, and sets span attributes.
Pre-dispatch prompt overflow trimming currently only increments llm_truncation_avoided_total and updates prompt-token metrics. It does not emit a structured log or span event.

Goal:

Introduce a helper/API where every instrumented action is described once and requires inputs for: log event, metric update, and span annotation/event.
Use it for the context-window paths first, especially prompt-budget trimming and backend delivered-context truncation.
Keep the API small and idiomatic for this repo. Do not build a generic observability framework larger than needed.

Acceptance criteria:

A call site cannot silently emit only one or two of log/metric/span when using the wrapper. The wrapper should require the event name, metric action, and span fields/event details explicitly.
Prompt-budget trimming emits a structured log event, increments llm_truncation_avoided_total, and annotates/adds an OTel span event when a current span exists. Include useful fields such as logical model, original token count, final token count, budget/target window, and dropped message count.
Backend delivered-context truncation continues to emit the existing metric and structured warning, and also goes through the new wrapper where practical without losing existing fields (prompt_tokens_sent, prompt_eval_count, target_num_ctx, num_parallel, backend, outcome).
Existing metrics names should remain stable unless there is a strong reason to add a new one.
Add focused tests proving the wrapper emits/logs the expected behavior for at least prompt trimming and delivered-context truncation.
Update docs/FEATURES.md or docs/proxy.md if the user-facing observability behavior materially changes.
Run the repo gate: ward exec test, uv run ruff check ., uv run black --check ., uv run mypy app, ward exec smoke, and ward exec boot-probe.

Notes:

This should make it easy to answer: every action agent-proxy cares to instrument has a log + metric + span, not a partial signal.
Keep public text and commit messages within repo style: she/her for Kai where relevant, no em dashes, no semicolons in prose.

Add a small instrumentation wrapper so code paths that matter emit all three observability signals together: structured log event, Prometheus metric, and OpenTelemetry span/attributes. Context from director surface: - agent-proxy already has structlog JSON logs, Prometheus metrics, and OTel spans in `app/obs.py`, with call sites spread across `app/main.py`, `app/queue.py`, `app/resilience.py`, and `app/analysis.py`. - Backend-delivered context truncation currently logs `dispatch.context_truncated`, increments `llm_context_truncated_total`, and sets span attributes. - Pre-dispatch prompt overflow trimming currently only increments `llm_truncation_avoided_total` and updates prompt-token metrics. It does not emit a structured log or span event. Goal: - Introduce a helper/API where every instrumented action is described once and requires inputs for: log event, metric update, and span annotation/event. - Use it for the context-window paths first, especially prompt-budget trimming and backend delivered-context truncation. - Keep the API small and idiomatic for this repo. Do not build a generic observability framework larger than needed. Acceptance criteria: - A call site cannot silently emit only one or two of log/metric/span when using the wrapper. The wrapper should require the event name, metric action, and span fields/event details explicitly. - Prompt-budget trimming emits a structured log event, increments `llm_truncation_avoided_total`, and annotates/adds an OTel span event when a current span exists. Include useful fields such as logical model, original token count, final token count, budget/target window, and dropped message count. - Backend delivered-context truncation continues to emit the existing metric and structured warning, and also goes through the new wrapper where practical without losing existing fields (`prompt_tokens_sent`, `prompt_eval_count`, `target_num_ctx`, `num_parallel`, backend, outcome). - Existing metrics names should remain stable unless there is a strong reason to add a new one. - Add focused tests proving the wrapper emits/logs the expected behavior for at least prompt trimming and delivered-context truncation. - Update `docs/FEATURES.md` or `docs/proxy.md` if the user-facing observability behavior materially changes. - Run the repo gate: `ward exec test`, `uv run ruff check .`, `uv run black --check .`, `uv run mypy app`, `ward exec smoke`, and `ward exec boot-probe`. Notes: - This should make it easy to answer: every action agent-proxy cares to instrument has a log + metric + span, not a partial signal. - Keep public text and commit messages within repo style: she/her for Kai where relevant, no em dashes, no semicolons in prose.

coilyco-ops commented

2026-07-09 18:03:03 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-agent-proxy-37 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T18:03:01Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agent-proxy#37 · branch issue-37 · harness codex · workflow direct-to-main
Run: engineer-codex-agent-proxy-37 · ward v0.504.0 · dispatched 2026-07-09T18:03:01Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.504.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-agent-proxy-37` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T18:03:01Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agent-proxy#37` · branch `issue-37` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-agent-proxy-37` · ward `v0.504.0` · dispatched `2026-07-09T18:03:01Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.504.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-09 18:11:42 +00:00

feat: add shared observability wrapper

coilyco-ops closed this issue

2026-07-09 18:11:42 +00:00

coilyco-ops commented

2026-07-09 18:12:21 +00:00

Author

Member

WARD-OUTCOME: done ✅

details

review gate skipped by ~/.ward/config.yaml default.
workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default
felt: steady after the wrapper shape settled.
confidence: high.
surprises: ward exec boot-probe pruned dev deps, so black and mypy had to be rerun after resyncing.
follow-ups: none.

WARD-OUTCOME: done ✅ <details><summary>details</summary> review gate skipped by `~/.ward/config.yaml` default. workflow: direct-to-main; review summary: skipped by `~/.ward/config.yaml` default felt: steady after the wrapper shape settled. confidence: high. surprises: `ward exec boot-probe` pruned dev deps, so black and mypy had to be rerun after resyncing. follow-ups: none. </details>

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-07-09 18:23:04 +00:00

Add git-tracked local SigNoz dashboard for agent-proxy effectiveness #494