chore(observability): tear down Grafana from tracked infra code (keep VictoriaMetrics) #388

New issue

Closed

opened 2026-06-23 08:26:48 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-06-23 08:26:48 +00:00

Owner

Consolidating observability on SigNoz. Grafana is being retired. This ticket covers removing Grafana from the tracked infra code only. Deleting the live k8s/DNS resources is a separate manual followup by Kai (checklist at the bottom), so this is safe to carry headless.

Scope guards (important)

Tracked code only. Do not run helm, kubectl, or terraform apply/destroy against the live cluster. Edit the repo, open the branch to merge. The ephemeral container has no cluster access anyway.
Keep VictoriaMetrics, vmagent, vmsingle, node-exporter. Only Grafana goes. VM stays as the headless metrics store - it has a downstream consumer (the eco-telemetry agent-loop reads it via a VM MCP), so do not remove it.
Do not touch GlitchTip or SigNoz.

Delete outright (Grafana-only)

terraform/grafana/ - the whole directory (main.tf, dashboards.tf, README.md, .terraform.lock.hcl, dashboards/eco-telemetry.yaml).
deploy/observability/grafana-values.yml
deploy/observability/admin-password-externalsecret.yml (Grafana admin-password ExternalSecret).
scripts/k8s/terraform_grafana.py
scripts/k8s/observability_admin_password.py (prints the Grafana admin password).

Edit (remove the Grafana parts, keep the rest)

Makefile - drop the terraform-grafana and observability-admin-password targets, and remove the helm install grafana step from the observability target. Reword its help from "VictoriaMetrics + Grafana" to VictoriaMetrics only.
.ward/ward.yaml - remove the terraform-grafana and observability-admin-password verbs, reword the observability verb description.
deploy/observability/README.md - remove the Grafana install step, the admin-password retrieval step, the Grafana pane bullet, the DNS prereq for grafana.coilysiren.me, and retitle off "VictoriaMetrics + Grafana". State that VM remains the metrics store and SigNoz is now where dashboards/glancing live.
docs/FEATURES.md - remove or rewrite the Grafana entry.
README.md (repo root), docs/tailscale.md, docs/external-dns.md, docs/glitchtip-deploy-plan.md, docs/forgejo-deploy-plan.md, ansible/inventory/group_vars/mac.yml, ansible/inventory/group_vars/linux.yml, scripts/k8s/observability.py, terraform/aws-inventory/main.tf - remove Grafana references (URLs, the grafana.coilysiren.me DNS/Route53 record definition, datasource mentions). Removing the DNS record from tracked terraform is in scope; applying that removal is Kai's followup, not the agent's.

Callout

The eco-telemetry Grafana dashboard (terraform/grafana/dashboards/eco-telemetry.yaml) is orphaned by this teardown. Recreating an equivalent view in SigNoz is part of the separate metrics-to-SigNoz migration, not this ticket. Remove it here.

Validation

pre-commit run --all-files clean.
grep -rinE 'grafana' . returns nothing but intentional historical mentions (e.g. a changelog), no live config/verbs/docs pointing at a Grafana that no longer ships.
The VictoriaMetrics stack (vmsingle, vmagent, node-exporter) is still internally coherent after the edits.

Kai's manual followup (live resources - NOT for the agent)

helm uninstall grafana -n observability
Delete the Grafana PVC in the observability namespace.
terraform apply in aws-inventory to drop the grafana.coilysiren.me A record, then remove the cert.
Tear down the terraform/grafana remote state.

Consolidating observability on SigNoz. Grafana is being retired. This ticket covers removing Grafana from the **tracked infra code** only. Deleting the live k8s/DNS resources is a separate manual followup by Kai (checklist at the bottom), so this is safe to carry headless. ## Scope guards (important) - **Tracked code only.** Do not run `helm`, `kubectl`, or `terraform apply/destroy` against the live cluster. Edit the repo, open the branch to merge. The ephemeral container has no cluster access anyway. - **Keep VictoriaMetrics, vmagent, vmsingle, node-exporter.** Only Grafana goes. VM stays as the headless metrics store - it has a downstream consumer (the eco-telemetry agent-loop reads it via a VM MCP), so do not remove it. - **Do not touch GlitchTip or SigNoz.** ## Delete outright (Grafana-only) - `terraform/grafana/` - the whole directory (`main.tf`, `dashboards.tf`, `README.md`, `.terraform.lock.hcl`, `dashboards/eco-telemetry.yaml`). - `deploy/observability/grafana-values.yml` - `deploy/observability/admin-password-externalsecret.yml` (Grafana admin-password ExternalSecret). - `scripts/k8s/terraform_grafana.py` - `scripts/k8s/observability_admin_password.py` (prints the Grafana admin password). ## Edit (remove the Grafana parts, keep the rest) - `Makefile` - drop the `terraform-grafana` and `observability-admin-password` targets, and remove the `helm install grafana` step from the `observability` target. Reword its help from "VictoriaMetrics + Grafana" to VictoriaMetrics only. - `.ward/ward.yaml` - remove the `terraform-grafana` and `observability-admin-password` verbs, reword the `observability` verb description. - `deploy/observability/README.md` - remove the Grafana install step, the admin-password retrieval step, the Grafana pane bullet, the DNS prereq for `grafana.coilysiren.me`, and retitle off "VictoriaMetrics + Grafana". State that VM remains the metrics store and SigNoz is now where dashboards/glancing live. - `docs/FEATURES.md` - remove or rewrite the Grafana entry. - `README.md` (repo root), `docs/tailscale.md`, `docs/external-dns.md`, `docs/glitchtip-deploy-plan.md`, `docs/forgejo-deploy-plan.md`, `ansible/inventory/group_vars/mac.yml`, `ansible/inventory/group_vars/linux.yml`, `scripts/k8s/observability.py`, `terraform/aws-inventory/main.tf` - remove Grafana references (URLs, the `grafana.coilysiren.me` DNS/Route53 record definition, datasource mentions). Removing the DNS record from tracked terraform is in scope; **applying** that removal is Kai's followup, not the agent's. ## Callout The `eco-telemetry` Grafana dashboard (`terraform/grafana/dashboards/eco-telemetry.yaml`) is orphaned by this teardown. Recreating an equivalent view in SigNoz is part of the separate metrics-to-SigNoz migration, **not** this ticket. Remove it here. ## Validation - `pre-commit run --all-files` clean. - `grep -rinE 'grafana' .` returns nothing but intentional historical mentions (e.g. a changelog), no live config/verbs/docs pointing at a Grafana that no longer ships. - The VictoriaMetrics stack (vmsingle, vmagent, node-exporter) is still internally coherent after the edits. ## Kai's manual followup (live resources - NOT for the agent) - `helm uninstall grafana -n observability` - Delete the Grafana PVC in the `observability` namespace. - `terraform` apply in `aws-inventory` to drop the `grafana.coilysiren.me` A record, then remove the cert. - Tear down the `terraform/grafana` remote state.

coilyco-ops commented

2026-06-23 08:27:21 +00:00

Author

Owner

🔒 Reserved by ward agent claude — container ward-infrastructure-issue-388-claude-739cc73a on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-23T08:27:21Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent claude` — container `ward-infrastructure-issue-388-claude-739cc73a` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-23T08:27:21Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue

2026-06-23 08:30:19 +00:00

feat(observability): agent-health heartbeat emitting OTLP to SigNoz #389