Retire VictoriaMetrics + vmagent from the observability stack #9

New issue

Closed

opened 2026-05-23 20:54:26 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:26 +00:00

Owner

Originally filed by @coilysiren on 2026-05-22T14:08:34Z - https://github.com/coilysiren/infrastructure/issues/282

Problem - The observability namespace runs two metrics-capable backends. VictoriaMetrics + Grafana for host metrics, and SigNoz (ClickHouse) which also stores metrics and landed recently (commits 13b771c, 4db972d). Two stores, two query languages, two ingestion paths for the same data.

Decision: retire VictoriaMetrics, vmagent, and the vmsingle tailnet/nodeport plumbing. SigNoz becomes the single self-hosted telemetry backend - traces now, metrics and logs fed in after. SigNoz cannot use VM as a backend (ClickHouse-locked), so consolidation means dropping VM, not migrating it.

Scope - remove from `deploy/observability/`

victoria-metrics-values.yml
vmagent-values.yml
vmsingle-nodeport-service.yml
vmsingle-tailscale-service.yml
VM datasource block in grafana-values.yml
README.md rewrite - currently titled "VictoriaMetrics + Grafana", and the eco-telemetry plug-in section points at vmsingle

Cluster teardown

helm uninstall victoria-metrics vmagent -n observability
delete the vmsingle PVCs (helm uninstall leaves them)
drop the vmsingle entry from terraform/tailscale-devices/services.yaml, re-apply to deregister the tailnet device and its SSM authkey

Downstream consumers to repoint before teardown

eco-telemetry - OtlpEndpoint targets vmsingle at :8428/opentelemetry/api/v1/push. Repoint at the SigNoz OTel collector (4317 gRPC / 4318 HTTP).
thermal-heartbeat - dual-pushes to VM/Grafana via the node-exporter textfile collector. The Sentry leg stays. The VM leg needs a new home or gets dropped.

Open question - host metrics path

node-exporter + vmagent currently scrape host metrics into vmsingle with Grafana as the UI. Retiring VM removes that path. Two options:

Feed node-exporter into the SigNoz OTel collector via a prometheus receiver. SigNoz becomes the host-metrics UI too.
Hand host monitoring to Beszel per #241 and drop the node-exporter + Grafana pane entirely.

Resolve this alongside #241 (Replace Grafana for host monitoring with Beszel). Grafana's only datasource is VM, so if Grafana goes, VM has no consumer left and option 2 falls out for free.

Out of scope

Sentry stays as the app-error pane. This issue is only about the self-hosted metrics backend.

Cross-ref: #241.

_Originally filed by @coilysiren on 2026-05-22T14:08:34Z - [https://github.com/coilysiren/infrastructure/issues/282](https://github.com/coilysiren/infrastructure/issues/282)_ **Problem** - The `observability` namespace runs two metrics-capable backends. VictoriaMetrics + Grafana for host metrics, and SigNoz (ClickHouse) which also stores metrics and landed recently (commits `13b771c`, `4db972d`). Two stores, two query languages, two ingestion paths for the same data. Decision: retire VictoriaMetrics, vmagent, and the vmsingle tailnet/nodeport plumbing. SigNoz becomes the single self-hosted telemetry backend - traces now, metrics and logs fed in after. SigNoz cannot use VM as a backend (ClickHouse-locked), so consolidation means dropping VM, not migrating it. ## Scope - remove from `deploy/observability/` - `victoria-metrics-values.yml` - `vmagent-values.yml` - `vmsingle-nodeport-service.yml` - `vmsingle-tailscale-service.yml` - VM datasource block in `grafana-values.yml` - `README.md` rewrite - currently titled "VictoriaMetrics + Grafana", and the eco-telemetry plug-in section points at vmsingle ## Cluster teardown - `helm uninstall victoria-metrics vmagent -n observability` - delete the vmsingle PVCs (helm uninstall leaves them) - drop the vmsingle entry from `terraform/tailscale-devices/services.yaml`, re-apply to deregister the tailnet device and its SSM authkey ## Downstream consumers to repoint before teardown - **eco-telemetry** - `OtlpEndpoint` targets vmsingle at `:8428/opentelemetry/api/v1/push`. Repoint at the SigNoz OTel collector (`4317` gRPC / `4318` HTTP). - **thermal-heartbeat** - dual-pushes to VM/Grafana via the node-exporter textfile collector. The Sentry leg stays. The VM leg needs a new home or gets dropped. ## Open question - host metrics path node-exporter + vmagent currently scrape host metrics into vmsingle with Grafana as the UI. Retiring VM removes that path. Two options: 1. Feed node-exporter into the SigNoz OTel collector via a prometheus receiver. SigNoz becomes the host-metrics UI too. 2. Hand host monitoring to Beszel per #241 and drop the node-exporter + Grafana pane entirely. Resolve this alongside #241 (Replace Grafana for host monitoring with Beszel). Grafana's only datasource is VM, so if Grafana goes, VM has no consumer left and option 2 falls out for free. ## Out of scope Sentry stays as the app-error pane. This issue is only about the self-hosted metrics backend. Cross-ref: #241.

coilysiren added the

label

2026-06-04 08:16:57 +00:00

coilysiren commented

2026-06-17 08:23:01 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:23:01 +00:00