Add git-tracked local SigNoz dashboard for agent-proxy effectiveness #494

New issue

Closed

opened 2026-07-09 18:23:04 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 18:23:04 +00:00

Member

Create a git-tracked SigNoz dashboard for evaluating agent-proxy effectiveness in the host-local SigNoz stack.

Context from director surface:

Local SigNoz is reachable on the host at 127.0.0.1:3301 and from containers as host.docker.internal:3301; /api/v1/version returns v0.90.1 and setupCompleted=true.
Dashboard API endpoints are authenticated, but the role already mounts ansible/roles/signoz/files/stack/common/dashboards into the SigNoz container at /root/config/dashboards with DASHBOARDS_PATH=/root/config/dashboards. That is the right tracked home if v0.90.1 imports dashboard JSON from disk.
agent-proxy exports Prometheus metrics and OTel spans. PROXY_TRACE_BODIES controls prompt/response body capture and should stay local-only.
A sibling issue in agent-proxy, coilyco-flight-deck/agent-proxy#37, is adding a required log+metric+span wrapper. This dashboard should work with current metrics and become richer after that lands.

Goal:

Add a tracked dashboard JSON for local SigNoz that answers whether agent-proxy is improving reliability and whether it is healthy.

Dashboard panels should cover, using existing stable metric names where possible:

Request volume by outcome: llm_requests_total.
Upstream latency p50/p95/p99 by backend/model: llm_upstream_latency_seconds.
Queue health: llm_queue_depth, llm_queue_rejected_total.
Retry and fallback activity: llm_retries_total, llm_fallbacks_total.
Circuit breaker state: llm_circuit_state.
Context behavior: llm_prompt_tokens, llm_truncation_avoided_total, llm_context_truncated_total.
Validation failures by reason: llm_validation_failures_total.
Add trace/log explorer links or saved-query style panels if SigNoz dashboard format supports them for service.name=agent-proxy, especially dispatch.context_truncated, request.context_truncated, request.ok, and validation events.

Acceptance criteria:

Dashboard is stored in the repo under the host-local SigNoz dashboard path or another documented tracked path that the role loads.
If SigNoz v0.90.1 requires a specific import schema, use that schema and document how it was verified.
Add docs in docs/signoz-local.md or docs/agent-proxy.md explaining how Kai opens the dashboard and how to enable/disable prompt body tracing safely.
Keep shared/k3s observability redacted by default. Do not turn on PROXY_TRACE_BODIES outside the host-local debugging path.
Validate with the repo's ward command(s), and if live SigNoz import cannot be tested from the engineer container, document the exact host-side verification command.

Notes:

A SigNoz MCP may be useful later for ad hoc readback, but a git-tracked dashboard is the durable first step.
Public-safe text only. No secrets, opaque host identifiers, or prompt bodies in tracked files.

Create a git-tracked SigNoz dashboard for evaluating agent-proxy effectiveness in the host-local SigNoz stack. Context from director surface: - Local SigNoz is reachable on the host at `127.0.0.1:3301` and from containers as `host.docker.internal:3301`; `/api/v1/version` returns v0.90.1 and setupCompleted=true. - Dashboard API endpoints are authenticated, but the role already mounts `ansible/roles/signoz/files/stack/common/dashboards` into the SigNoz container at `/root/config/dashboards` with `DASHBOARDS_PATH=/root/config/dashboards`. That is the right tracked home if v0.90.1 imports dashboard JSON from disk. - `agent-proxy` exports Prometheus metrics and OTel spans. `PROXY_TRACE_BODIES` controls prompt/response body capture and should stay local-only. - A sibling issue in agent-proxy, coilyco-flight-deck/agent-proxy#37, is adding a required log+metric+span wrapper. This dashboard should work with current metrics and become richer after that lands. Goal: - Add a tracked dashboard JSON for local SigNoz that answers whether agent-proxy is improving reliability and whether it is healthy. Dashboard panels should cover, using existing stable metric names where possible: - Request volume by outcome: `llm_requests_total`. - Upstream latency p50/p95/p99 by backend/model: `llm_upstream_latency_seconds`. - Queue health: `llm_queue_depth`, `llm_queue_rejected_total`. - Retry and fallback activity: `llm_retries_total`, `llm_fallbacks_total`. - Circuit breaker state: `llm_circuit_state`. - Context behavior: `llm_prompt_tokens`, `llm_truncation_avoided_total`, `llm_context_truncated_total`. - Validation failures by reason: `llm_validation_failures_total`. - Add trace/log explorer links or saved-query style panels if SigNoz dashboard format supports them for `service.name=agent-proxy`, especially `dispatch.context_truncated`, `request.context_truncated`, `request.ok`, and validation events. Acceptance criteria: - Dashboard is stored in the repo under the host-local SigNoz dashboard path or another documented tracked path that the role loads. - If SigNoz v0.90.1 requires a specific import schema, use that schema and document how it was verified. - Add docs in `docs/signoz-local.md` or `docs/agent-proxy.md` explaining how Kai opens the dashboard and how to enable/disable prompt body tracing safely. - Keep shared/k3s observability redacted by default. Do not turn on `PROXY_TRACE_BODIES` outside the host-local debugging path. - Validate with the repo's ward command(s), and if live SigNoz import cannot be tested from the engineer container, document the exact host-side verification command. Notes: - A SigNoz MCP may be useful later for ad hoc readback, but a git-tracked dashboard is the durable first step. - Public-safe text only. No secrets, opaque host identifiers, or prompt bodies in tracked files.

coilyco-ops commented

2026-07-09 18:23:11 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-494 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T18:23:09Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#494 · branch issue-494 · harness codex · workflow direct-to-main
Run: engineer-codex-infrastructure-494 · ward v0.504.0 · dispatched 2026-07-09T18:23:09Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.504.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-494` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T18:23:09Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#494` · branch `issue-494` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-infrastructure-494` · ward `v0.504.0` · dispatched `2026-07-09T18:23:09Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.504.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-09 18:38:13 +00:00

feat: add agent-proxy SigNoz dashboard

coilyco-ops closed this issue

2026-07-09 18:38:13 +00:00

coilyco-ops commented

2026-07-09 18:39:39 +00:00

Author

Member

WARD-OUTCOME: done ✅

details

workflow: direct-to-main; review summary: skip; review gate skipped by ~/.ward/config.yaml default.
felt: mostly schema and doc shaping.
confidence: high
surprise: UV_PYTHON_INSTALL_DIR was needed because /opt/uv/python was unwritable.
follow-ups: none

WARD-OUTCOME: done ✅ <details><summary>details</summary> workflow: direct-to-main; review summary: skip; review gate skipped by ~/.ward/config.yaml default. felt: mostly schema and doc shaping. confidence: high surprise: UV_PYTHON_INSTALL_DIR was needed because /opt/uv/python was unwritable. follow-ups: none </details>