feat(observability): agent-health heartbeat emitting OTLP to SigNoz

coilyco-ops commented

2026-06-23 08:30:19 +00:00

Member

There is no o11y on the agent serving layer. The local-model harnesses (goose, opencode, qwen) bind to Ollama on the tower over tailnet, and "why are the agents down/slow?" currently requires hand-curling the tower. Nothing feeds that signal into the o11y stack.

Build an always-on heartbeat as a third sibling to scripts/process-memory-heartbeat.py and thermal-heartbeat-install.sh:

Sample on an interval: Ollama /api/version (up + round-trip latency), /api/ps (which model is resident, size_vram, expires_at), and GPU/VRAM via nvidia.
Emit metrics over OTLP/HTTP. Per the SigNoz consolidation decision, target SigNoz's OTel collector (OTLP 4317/4318), not VictoriaMetrics/Grafana, so this survives the Grafana teardown (#388).
Roll out via the same ansible pattern the other heartbeats use (authoring here, fleet rollout via ansible).
Add a SigNoz view so "are the agents down?" is a glance, not an investigation.

Reuse existing probe logic: the reachability + residency probe is already prototyped in agentic-os scripts/goose-health.sh (see coilyco-flight-deck/agentic-os#268 for that script's disposition). Lift the /api/version + /api/ps parsing from there.

Why it matters: the incident that surfaced this was VRAM contention - qwen3-coder:30b evicted by deepseek-v2:16b on the tower, so every goose call cold-loads. A residency + latency metric would have answered it instantly instead of four hops of manual investigation.

There is no o11y on the agent serving layer. The local-model harnesses (goose, opencode, qwen) bind to Ollama on the tower over tailnet, and "why are the agents down/slow?" currently requires hand-curling the tower. Nothing feeds that signal into the o11y stack. **Build an always-on heartbeat** as a third sibling to `scripts/process-memory-heartbeat.py` and `thermal-heartbeat-install.sh`: - Sample on an interval: Ollama `/api/version` (up + round-trip latency), `/api/ps` (which model is resident, `size_vram`, `expires_at`), and GPU/VRAM via nvidia. - Emit metrics over OTLP/HTTP. Per the SigNoz consolidation decision, target **SigNoz's OTel collector (OTLP `4317`/`4318`)**, not VictoriaMetrics/Grafana, so this survives the Grafana teardown (#388). - Roll out via the same ansible pattern the other heartbeats use (authoring here, fleet rollout via ansible). - Add a SigNoz view so "are the agents down?" is a glance, not an investigation. **Reuse existing probe logic:** the reachability + residency probe is already prototyped in `agentic-os` `scripts/goose-health.sh` (see coilyco-flight-deck/agentic-os#268 for that script's disposition). Lift the `/api/version` + `/api/ps` parsing from there. **Why it matters:** the incident that surfaced this was VRAM contention - `qwen3-coder:30b` evicted by `deepseek-v2:16b` on the tower, so every goose call cold-loads. A residency + latency metric would have answered it instantly instead of four hops of manual investigation.

coilyco-ops referenced this issue

2026-06-23 08:38:13 +00:00

chore(observability): delete Grafana live resources (manual followup to #388) #391

coilyco-ops added the

headless

P2

labels

2026-06-24 05:38:18 +00:00

coilyco-ops commented

2026-06-24 05:38:19 +00:00

Author

Member

Triage: P2 / headless - Agent-health heartbeat to SigNoz; well-specified, lifts existing goose-health probe logic.

(goose-style pass; claude as the judgment engine, applied 2026-06-23)

**Triage:** `P2` / `headless` - Agent-health heartbeat to SigNoz; well-specified, lifts existing goose-health probe logic. (goose-style pass; claude as the judgment engine, applied 2026-06-23)

coilyco-ops commented

2026-07-01 08:08:31 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-389 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-01T08:08:29Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-389` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-01T08:08:29Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-01 08:32:37 +00:00

feat(observability): agent-health heartbeat emitting OTLP to SigNoz

coilysiren closed this issue

2026-07-01 08:32:37 +00:00

coilysiren referenced this issue from a commit

2026-07-01 08:32:37 +00:00

Merge issue-389: agent-health heartbeat to SigNoz (#389)

coilysiren commented

2026-07-01 08:33:44 +00:00

Owner

WARD-OUTCOME: done - agent-health heartbeat landed on main (Ollama + GPU probe, OTLP to SigNoz, ansible user-timer, NodePort, glance dashboard), CI green.

Felt clean once the topology clicked. The two named siblings are root system-timers installed by shell scripts, but "fleet rollout via ansible" really pointed at the git-state-watch pattern, so I went with a no-sudo user systemd timer on kai-server. The one thing that actually fought back was pylint's duplicate-code check. My minimal OTLP encoder was a near-verbatim copy of process-memory-heartbeat's, so I pulled it into scripts/_otlp.py and moved both scripts onto it, adding as_double for the latency and utilization gauges. One encoder now instead of two.

Confidence is high on the script itself. I decoded its payload against the real opentelemetry-proto library and round-tripped both the spilled-model detection and the RFC3339 keep-alive math. Lower on two deploy-time unknowns I could not exercise from inside the container: whether the NodePort selector matches the live SigNoz chart's otel-collector labels (I left the one-liner to confirm in the README), and whether kai-server resolves the tower's Ollama at kai-tower-3026:11434. Both are label or env knobs, not code.

Follow-ups worth filing: GPU rows only appear when the probe host has a card, so on kai-server they stay empty and the tower GPU keeps living on the Windows gpu-heartbeat path. If we want GPU-in-SigNoz we install the same role on a GPU Linux host. And the dashboard JSON is best-effort against the SigNoz v4 builder schema, so it may want a nudge on import.

WARD-OUTCOME: done - agent-health heartbeat landed on main (Ollama + GPU probe, OTLP to SigNoz, ansible user-timer, NodePort, glance dashboard), CI green. Felt clean once the topology clicked. The two named siblings are root system-timers installed by shell scripts, but "fleet rollout via ansible" really pointed at the git-state-watch pattern, so I went with a no-sudo user systemd timer on kai-server. The one thing that actually fought back was pylint's duplicate-code check. My minimal OTLP encoder was a near-verbatim copy of process-memory-heartbeat's, so I pulled it into `scripts/_otlp.py` and moved both scripts onto it, adding `as_double` for the latency and utilization gauges. One encoder now instead of two. Confidence is high on the script itself. I decoded its payload against the real opentelemetry-proto library and round-tripped both the spilled-model detection and the RFC3339 keep-alive math. Lower on two deploy-time unknowns I could not exercise from inside the container: whether the NodePort selector matches the live SigNoz chart's otel-collector labels (I left the one-liner to confirm in the README), and whether kai-server resolves the tower's Ollama at `kai-tower-3026:11434`. Both are label or env knobs, not code. Follow-ups worth filing: GPU rows only appear when the probe host has a card, so on kai-server they stay empty and the tower GPU keeps living on the Windows gpu-heartbeat path. If we want GPU-in-SigNoz we install the same role on a GPU Linux host. And the dashboard JSON is best-effort against the SigNoz v4 builder schema, so it may want a nudge on import.

Rows
Columns

feat(observability): agent-health heartbeat emitting OTLP to SigNoz #389