attended: finish deprecated service and tailnet device deletion #683

New issue

Closed

opened 2026-07-29 01:00:09 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-29 01:00:09 +00:00

Owner

Source and reversible runtime work are complete.

Infrastructure main retires the backend, Repo Recall, Open WebUI, ntfy, predecessor Eco, and obsolete session-watcher surfaces. The desired Tailscale service inventory now retains only forgejo, comfyui, lunch-money-mcp, and playwright-mcp.
Deploy main removes Open WebUI and removes the legacy Tailscale and Sentry wiring from the retained Eco App manifest.
The backend app and database, Open WebUI, Repo Recall, ntfy, and both predecessor Eco deployments are scaled to zero.
Eight non-Terraform application parameters were deleted from SSM and the generated inventory was refreshed. Terraform-owned Tailscale auth-key parameters remain for the attended apply.

Done when:

Kai completes infrastructure issue 684, re-dispatches deploy-eco-app.yml, and verifies the coilysiren-eco-app app Deployment has only its application container and no Sentry environment reference.
After that rollout, Kai deletes the retired /sentry-dsn/eco-mcp-app parameter and confirms its old ExternalSecret and generated Secret are absent.
Kai pulls canonical infrastructure main and runs ward exec terraform-tailscale action=plan followed by action=apply with her human-held Tailscale admin credential. The plan must retire only the declared service identities removed from services.yaml, including repo-recall, eco-mcp, backend, open-webui, galaxy-gen, signoz, and vmsingle.
Kai deletes the retired namespaces coilysiren-backend, open-webui, repo-recall, coilysiren-eco-mcp-app, coilysiren-eco-spec-tracker, and ntfy. Namespace deletion intentionally removes their remaining workloads, ExternalSecrets, Services, CronJobs, and PVCs.
Kai removes the misplaced ComfyUI release resources from the forgejo namespace by the exact label app.kubernetes.io/instance=comfyui. Kai preserves the real dormant comfyui namespace.
Kai uninstalls the scaled-zero grafana, signoz, victoria-metrics, and vmagent Helm releases from observability, then removes only their remaining storage and release resources. Kai preserves kai-server-logs, node-exporter, and the observability namespace.
Kai deletes the retired tailnet machines after their workloads are gone. This includes ntfy, galaxy-gen, signoz, vmsingle, observability-vmsingle-tailscale, api, backend-db, open-webui, repo-recall, eco-mcp, both predecessor Eco Service peers, and only the offline duplicate Forgejo peer. Kai preserves the online forgejo, tailscale-proxy, retained MCP services, and dormant real comfyui peer.
Kai leaves the two ambiguous localhost peers untouched until their owner is identified, then verifies the surviving kai-server collectors and the ser8 SigNoz plane remain healthy.

AOSGuard intentionally denies Kubernetes delete and Tailscale administration uses a human-only credential, so these destructive steps remain attended.

Source and reversible runtime work are complete. * Infrastructure main retires the backend, Repo Recall, Open WebUI, ntfy, predecessor Eco, and obsolete session-watcher surfaces. The desired Tailscale service inventory now retains only forgejo, comfyui, lunch-money-mcp, and playwright-mcp. * Deploy main removes Open WebUI and removes the legacy Tailscale and Sentry wiring from the retained Eco App manifest. * The backend app and database, Open WebUI, Repo Recall, ntfy, and both predecessor Eco deployments are scaled to zero. * Eight non-Terraform application parameters were deleted from SSM and the generated inventory was refreshed. Terraform-owned Tailscale auth-key parameters remain for the attended apply. Done when: 1. Kai completes infrastructure issue 684, re-dispatches deploy-eco-app.yml, and verifies the coilysiren-eco-app app Deployment has only its application container and no Sentry environment reference. 2. After that rollout, Kai deletes the retired /sentry-dsn/eco-mcp-app parameter and confirms its old ExternalSecret and generated Secret are absent. 3. Kai pulls canonical infrastructure main and runs ward exec terraform-tailscale action=plan followed by action=apply with her human-held Tailscale admin credential. The plan must retire only the declared service identities removed from services.yaml, including repo-recall, eco-mcp, backend, open-webui, galaxy-gen, signoz, and vmsingle. 4. Kai deletes the retired namespaces coilysiren-backend, open-webui, repo-recall, coilysiren-eco-mcp-app, coilysiren-eco-spec-tracker, and ntfy. Namespace deletion intentionally removes their remaining workloads, ExternalSecrets, Services, CronJobs, and PVCs. 5. Kai removes the misplaced ComfyUI release resources from the forgejo namespace by the exact label app.kubernetes.io/instance=comfyui. Kai preserves the real dormant comfyui namespace. 6. Kai uninstalls the scaled-zero grafana, signoz, victoria-metrics, and vmagent Helm releases from observability, then removes only their remaining storage and release resources. Kai preserves kai-server-logs, node-exporter, and the observability namespace. 7. Kai deletes the retired tailnet machines after their workloads are gone. This includes ntfy, galaxy-gen, signoz, vmsingle, observability-vmsingle-tailscale, api, backend-db, open-webui, repo-recall, eco-mcp, both predecessor Eco Service peers, and only the offline duplicate Forgejo peer. Kai preserves the online forgejo, tailscale-proxy, retained MCP services, and dormant real comfyui peer. 8. Kai leaves the two ambiguous localhost peers untouched until their owner is identified, then verifies the surviving kai-server collectors and the ser8 SigNoz plane remain healthy. AOSGuard intentionally denies Kubernetes delete and Tailscale administration uses a human-only credential, so these destructive steps remain attended.

coilyco-ops changed title from ~~attended: finish retired service and tailnet device deletion~~ to attended: finish deprecated service and tailnet device deletion

2026-07-29 02:07:39 +00:00

coilyco-ops added the

autonomy

live-collab

label

2026-07-29 02:07:43 +00:00

coilyco-ops commented

2026-07-29 02:11:50 +00:00

Author

Owner

SigNoz error tracking follow-through is already tracked in coilyco-bridge/deploy#249. That issue has the Exceptions grouping and Eco App instrumentation landed, with attended rollout and one safe exception verification remaining.

coilyco-ops

2026-07-29 03:46:37 +00:00

closed this issue
added the
priority
P4
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#683

No description provided.

Rows
Columns