Retire duplicate kai-server observability backend after ser8 cutover #680

New issue

Closed

opened 2026-07-29 00:01:32 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-29 00:01:32 +00:00

Owner

The duplicate kai-server observability backend was reversibly scaled to zero after confirming active OTLP producers send to ser8.

Completed live state:

Grafana, SigNoz, ClickHouse, ZooKeeper, the SigNoz collector and operator, VictoriaMetrics, vmagent, and both local Tailscale proxies are at zero replicas.
PVCs and Helm release metadata remain for rollback.
The two kai-server-logs OTel collectors remain healthy because they forward logs and Kubernetes events to ser8.
node-exporter remains temporarily because the guarded Kubernetes surface cannot scale a DaemonSet and no guarded Helm uninstall action exists.

Acceptance:

Add a guarded retirement path for the grafana, signoz, victoria-metrics, vmagent, and node-exporter Helm releases on kai-server.
Remove node-exporter after confirming no remaining local scrape consumer.
Remove the retired local SigNoz and VictoriaMetrics exposure resources and prevent the old Ward install verbs from recreating them on kai-server.
Preserve the ser8-bound kai-server-logs forwarders.
Inventory retained PVC sizes and obtain an explicit retention or deletion decision before deleting telemetry data.
Update observability documentation and FEATURES to describe ser8 as the canonical backend.
Verify kai-server pod occupancy and ser8 ingestion after retirement.

The duplicate kai-server observability backend was reversibly scaled to zero after confirming active OTLP producers send to ser8. Completed live state: * Grafana, SigNoz, ClickHouse, ZooKeeper, the SigNoz collector and operator, VictoriaMetrics, vmagent, and both local Tailscale proxies are at zero replicas. * PVCs and Helm release metadata remain for rollback. * The two `kai-server-logs` OTel collectors remain healthy because they forward logs and Kubernetes events to ser8. * node-exporter remains temporarily because the guarded Kubernetes surface cannot scale a DaemonSet and no guarded Helm uninstall action exists. Acceptance: * Add a guarded retirement path for the `grafana`, `signoz`, `victoria-metrics`, `vmagent`, and `node-exporter` Helm releases on kai-server. * Remove node-exporter after confirming no remaining local scrape consumer. * Remove the retired local SigNoz and VictoriaMetrics exposure resources and prevent the old Ward install verbs from recreating them on kai-server. * Preserve the ser8-bound `kai-server-logs` forwarders. * Inventory retained PVC sizes and obtain an explicit retention or deletion decision before deleting telemetry data. * Update observability documentation and FEATURES to describe ser8 as the canonical backend. * Verify kai-server pod occupancy and ser8 ingestion after retirement.

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-29 03:46:36 +00:00

coilyco-ops added the

role/sysadmin

label

2026-08-25 08:02:20 +00:00

coilyco-ops referenced this issue

2026-08-27 03:21:45 +00:00

chore(observability): delete Grafana live resources (manual followup to #388) #391

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops commented

2026-08-28 21:32:50 +00:00

Author

Owner

Duplicate of #505, which tracks the same kai-server observability retirement after the ser8 cutover and carries the discussion. Closing this one, #505 survives.

Part of the 2026-08-28 burn-down, rationale in #981. Tagged burndown-2026-08, so state:closed label:burndown-2026-08 recovers this and everything else closed today. Reopen freely if the two turn out to be different problems.

Duplicate of **#505**, which tracks the same kai-server observability retirement after the ser8 cutover and carries the discussion. Closing this one, #505 survives. Part of the 2026-08-28 burn-down, rationale in #981. Tagged `burndown-2026-08`, so `state:closed label:burndown-2026-08` recovers this and everything else closed today. Reopen freely if the two turn out to be different problems.

coilyco-ops

2026-08-28 21:32:56 +00:00