coilyco-flight-deck/infrastructure

Watch

chore(observability): delete Grafana live resources (manual followup to #388) #391

New issue

Closed

opened 2026-06-23 08:38:13 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-06-23 08:38:13 +00:00

Owner

The tracked-code Grafana teardown landed in #388 (merged as 15b0f98). This issue tracks the live-resource deletion, which needs kubectl/terraform against kai-server's k3s and is Kai's manual followup - not a headless-agent task.

Checklist

helm uninstall grafana -n observability
Delete the Grafana PVC in the observability namespace (the 2 GiB data volume).
Confirm the Grafana admin-password ExternalSecret/Secret is gone (its manifest was removed in #388; delete any lingering live object).
terraform apply in aws-inventory to drop the grafana.coilysiren.me Route53 A record (the record was removed from tracked code in #388, the apply deletes the live record).
Remove the issued TLS cert for grafana.coilysiren.me.
Tear down the terraform/grafana remote state (the module dir was deleted in #388).

Notes

Keep VictoriaMetrics, vmagent, vmsingle, node-exporter running - only Grafana goes. VM stays as the headless metrics store (the eco-telemetry agent-loop reads it via a VM MCP).
Order does not matter much, but drop the DNS record + cert last so nothing is briefly pointing at a torn-down service longer than needed.
After this, metrics live in VM (headless) and dashboards/glancing move to SigNoz - see the metrics-to-SigNoz migration (#389 covers the agent-health slice).

Refs

Tracked-code teardown: #388 (closed).
Heartbeat to SigNoz: #389.

The tracked-code Grafana teardown landed in #388 (merged as `15b0f98`). This issue tracks the **live-resource deletion**, which needs `kubectl`/`terraform` against kai-server's k3s and is Kai's manual followup - not a headless-agent task. ## Checklist - [ ] `helm uninstall grafana -n observability` - [ ] Delete the Grafana PVC in the `observability` namespace (the 2 GiB data volume). - [ ] Confirm the Grafana admin-password ExternalSecret/Secret is gone (its manifest was removed in #388; delete any lingering live object). - [ ] `terraform` apply in `aws-inventory` to drop the `grafana.coilysiren.me` Route53 A record (the record was removed from tracked code in #388, the apply deletes the live record). - [ ] Remove the issued TLS cert for `grafana.coilysiren.me`. - [ ] Tear down the `terraform/grafana` remote state (the module dir was deleted in #388). ## Notes - **Keep VictoriaMetrics, vmagent, vmsingle, node-exporter running** - only Grafana goes. VM stays as the headless metrics store (the eco-telemetry agent-loop reads it via a VM MCP). - Order does not matter much, but drop the DNS record + cert last so nothing is briefly pointing at a torn-down service longer than needed. - After this, metrics live in VM (headless) and dashboards/glancing move to SigNoz - see the metrics-to-SigNoz migration (#389 covers the agent-health slice). ## Refs - Tracked-code teardown: #388 (closed). - Heartbeat to SigNoz: #389.

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-06-24 05:38:16 +00:00

coilyco-ops commented

2026-06-24 05:38:17 +00:00

Author

Owner

Triage: P3 / interactive - Delete live Grafana resources; body states it is a manual destructive kubectl/terraform followup, not a headless task.

(goose-style pass; claude as the judgment engine, applied 2026-06-23)

**Triage:** `P3` / `interactive` - Delete live Grafana resources; body states it is a manual destructive kubectl/terraform followup, not a headless task. (goose-style pass; claude as the judgment engine, applied 2026-06-23)

coilyco-ops added

and removed

labels

2026-07-10 09:00:04 +00:00

coilyco-ops removed the

autonomy

async-consult

label

2026-07-16 01:08:04 +00:00

coilyco-ops added the

autonomy

live-collab

label

2026-07-29 03:45:50 +00:00

coilyco-ops referenced this issue

2026-08-22 20:43:03 +00:00

[interactive] Rotate Telegram bot credential exposed in Alertmanager logs #708

coilyco-ops added the

role/sysadmin

label

2026-08-25 08:02:33 +00:00

coilyco-ops commented

2026-08-27 03:21:45 +00:00

Author

Owner

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901).

Closing as largely overtaken. #388 removed every tracked Grafana reference and #680 records the release already scaled to zero on kai-server. The residual PVC, DNS record, and remote-state cleanup belongs with #680's guarded retirement path rather than as its own issue.

This was priority/P3, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless.

If this is live work, reopen it. The whole set is recoverable with state:closed label:burndown-2026-08.

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901). Closing as largely overtaken. #388 removed every tracked Grafana reference and #680 records the release already scaled to zero on kai-server. The residual PVC, DNS record, and remote-state cleanup belongs with #680's guarded retirement path rather than as its own issue. This was `priority/P3`, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless. If this is live work, reopen it. The whole set is recoverable with `state:closed label:burndown-2026-08`.

coilyco-ops

2026-08-27 03:21:46 +00:00

closed this issue
added the
burndown-2026-08
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#391

No description provided.

Rows
Columns