coilyco-flight-deck/infrastructure

Fork 0

Deprovision legacy kai-server namespaces: repo-recall, ntfy, lunch-money, eco-spec-tracker #500

New issue

Open

opened 2026-07-09 21:10:10 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 21:10:10 +00:00

Member

Kai asked to deprovision these legacy kai-server workloads after a memory rollup still showed them live:

21Mi repo-recall
5Mi ntfy
3Mi lunch-money-mcp
2Mi lunch-money
2Mi coilysiren-eco-spec-tracker

Decision on the ambiguous Lunch Money pair: retire the older standalone lunch-money namespace, not lunch-money-mcp. lunch-money-mcp is the newer deploy-repo bundle (coilyco-bridge/deploy/services/lunch-money-mcp) and coilyco-flight-deck/lunch-money-k8s#22 already tracks source-side retirement of the older standalone deploy.

Please deprovision the live legacy workloads safely:

repo-recall - live namespace still present even though coilyco-bridge/deploy#48 closed and the deploy repo no longer has services/repo-recall.
ntfy - remove the legacy ntfy namespace/service if Telegram cutover is complete. Also remove remaining backup/timer/docs references that still treat ntfy as a live PVC/service.
lunch-money - retire the older standalone Helm deploy and namespace after verifying lunch-money-mcp is the serving replacement.
coilysiren-eco-spec-tracker - retire the predecessor namespace now that services/eco-app is the fused Eco app. The deploy docs already say eco-app replaces coilysiren-eco-mcp-app and coilysiren-eco-spec-tracker; this issue only names the spec-tracker namespace Kai requested.

Safety requirements:

Inventory each namespace first: deployments/statefulsets/jobs, ingress/service/tailscale sidecars, PVCs, ExternalSecrets/Secrets, and any restic backup inclusion.
Take or verify any needed snapshot before deleting PVC-backed state. If the service is intentionally disposable, record that in the issue before deletion.
Remove tracked infrastructure references after live deletion where appropriate: examples found in substrate include deploy/lunch-money/, scripts/k8s/lunch_money.py, Makefile target lunch-money, scripts/restic-backup.sh RESTIC_BACKUP_NAMESPACES including ntfy, and Tailscale service entries for repo-recall/eco-spec.
Coordinate with coilyco-flight-deck/lunch-money-k8s#22 for source-side chart/docs retirement rather than duplicating that work here.
Post the before/after kubectl get ns and namespace memory rollup in the issue.

Acceptance: the four legacy namespaces above no longer appear in kubectl top pod -A --containers namespace rollups, and tracked infra no longer recreates them.

Kai asked to deprovision these legacy kai-server workloads after a memory rollup still showed them live: ```text 21Mi repo-recall 5Mi ntfy 3Mi lunch-money-mcp 2Mi lunch-money 2Mi coilysiren-eco-spec-tracker ``` Decision on the ambiguous Lunch Money pair: retire the older standalone `lunch-money` namespace, not `lunch-money-mcp`. `lunch-money-mcp` is the newer deploy-repo bundle (`coilyco-bridge/deploy/services/lunch-money-mcp`) and `coilyco-flight-deck/lunch-money-k8s#22` already tracks source-side retirement of the older standalone deploy. Please deprovision the live legacy workloads safely: * `repo-recall` - live namespace still present even though `coilyco-bridge/deploy#48` closed and the deploy repo no longer has `services/repo-recall`. * `ntfy` - remove the legacy ntfy namespace/service if Telegram cutover is complete. Also remove remaining backup/timer/docs references that still treat ntfy as a live PVC/service. * `lunch-money` - retire the older standalone Helm deploy and namespace after verifying `lunch-money-mcp` is the serving replacement. * `coilysiren-eco-spec-tracker` - retire the predecessor namespace now that `services/eco-app` is the fused Eco app. The deploy docs already say eco-app replaces `coilysiren-eco-mcp-app` and `coilysiren-eco-spec-tracker`; this issue only names the spec-tracker namespace Kai requested. Safety requirements: * Inventory each namespace first: deployments/statefulsets/jobs, ingress/service/tailscale sidecars, PVCs, ExternalSecrets/Secrets, and any restic backup inclusion. * Take or verify any needed snapshot before deleting PVC-backed state. If the service is intentionally disposable, record that in the issue before deletion. * Remove tracked infrastructure references after live deletion where appropriate: examples found in substrate include `deploy/lunch-money/`, `scripts/k8s/lunch_money.py`, `Makefile` target `lunch-money`, `scripts/restic-backup.sh` `RESTIC_BACKUP_NAMESPACES` including `ntfy`, and Tailscale service entries for `repo-recall`/`eco-spec`. * Coordinate with `coilyco-flight-deck/lunch-money-k8s#22` for source-side chart/docs retirement rather than duplicating that work here. * Post the before/after `kubectl get ns` and namespace memory rollup in the issue. Acceptance: the four legacy namespaces above no longer appear in `kubectl top pod -A --containers` namespace rollups, and tracked infra no longer recreates them.

coilyco-ops commented

2026-07-09 21:10:17 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-500 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T21:10:15Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#500 · branch issue-500 · harness codex · workflow direct-to-main
Run: engineer-codex-infrastructure-500 · ward v0.501.0 · dispatched 2026-07-09T21:10:15Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.501.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-500` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T21:10:15Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#500` · branch `issue-500` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-infrastructure-500` · ward `v0.501.0` · dispatched `2026-07-09T21:10:15Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.501.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-09 21:19:08 +00:00

chore: remove legacy namespace recreate paths

coilyco-ops added the

interactive

labels

2026-07-09 21:46:09 +00:00

coilyco-ops added

consult

and removed

interactive

labels

2026-07-10 08:59:52 +00:00

coilyco-ops commented

2026-07-10 13:05:03 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Advisory recommendation

This should stay centered on infrastructure#500 for the live kai-server teardown, but it has one real cross-repo dependency: coilyco-bridge/deploy still treats ntfy as an alert delivery target for ser8 observability. The infrastructure work should not delete the kai-server ntfy service until that deploy-side ntfy bridge is removed or explicitly accepted as dead.

Decisions already settled

Retire repo-recall, ntfy, lunch-money, and coilysiren-eco-spec-tracker from kai-server.
Do not delete lunch-money-mcp. /substrate/deploy/services/lunch-money-mcp is the newer deploy-repo bundle, serves https://lunch-money.coilysiren.me/mcp, and exposes the local harness at http://kai-server:30080/mcp.
Do not broaden the Eco cleanup to coilysiren-eco-mcp-app in this issue unless Kai explicitly expands scope. The deploy repo says eco-app replaces both predecessors, but this issue names only coilysiren-eco-spec-tracker.
Treat coilyco-flight-deck/lunch-money-k8s#22 as the source-side retirement track for the old standalone chart/docs. Do not duplicate that cleanup here.

Repo evidence

deploy/lunch-money/, scripts/k8s/lunch_money.py, the Makefile lunch-money target, and .ward/ward.yaml still define the old standalone lunch-money deployment path in this repo.
scripts/restic-backup.sh, docs/restic-backups.md, and docs/FEATURES.md still include ntfy in the PVC backup set.
terraform/tailscale/services.yaml still mints service identities for repo-recall and eco-spec. scripts/k8s/terraform_tailscale_merge.py also carries those names in its historical migration lists, so an engineer should inspect whether those references are still live code or only migration history before editing.
The deploy repo no longer has services/repo-recall, but this repo still has stale pointer docs at docs/repo-recall-deploy.md and docs/repo-recall-deploy-ops.md.
/substrate/deploy/services/eco-app/README.md and deploy/main.yml confirm the fused eco-app replaces coilysiren-eco-spec-tracker and retires the eco-spec tailnet device.
/substrate/deploy/services/lunch-money-mcp/README.md confirms the replacement Lunch Money surface is the deploy-repo bundle, not the older Helm release.
/substrate/deploy/services/ser8-observability still includes an ntfy-alertmanager bridge and vmalert-values.yml still sends to it. That is the main blocker to deleting ntfy cleanly.

Open questions for the implementer

Is Telegram-only alerting now confirmed on ser8 after removing the ntfy bridge? If not, do that first in coilyco-bridge/deploy.
Do any of the four target namespaces have PVCs? ntfy is expected to, because restic still includes it. The other three may be disposable, but the issue should record the live PVC inventory before deletion.
Are any ExternalSecrets still syncing secrets only for these retired workloads? Likely candidates are /coilysiren/repo-recall/*, /coilysiren/eco-spec/ts-authkey, /sentry-dsn/eco-spec-tracker, and the old lunch-money-token ExternalSecret. Do not delete SSM values in this issue unless Kai explicitly wants secret retirement too.
Does lunch-money-mcp pass its readiness checks before deleting the old lunch-money namespace? Use the deploy repo's documented checks for the metadata endpoint, protected MCP endpoint, and namespace pods.

Recommended execution order

Land the deploy-repo ntfy bridge removal first, then verify ser8 alerting is Telegram-only.
On kai-server, capture before evidence: namespace list, namespace memory rollup, and per-namespace inventory for workloads, ingress/service/tailscale sidecars, PVCs, ExternalSecrets, Secrets, and Helm releases.
For each namespace with PVCs, either take or verify a restic snapshot, or explicitly record that the service is disposable.
Delete the live resources namespace by namespace. Prefer Helm uninstall where a release exists, then remove remaining namespace resources.
Remove infrastructure paths that could recreate them: old lunch-money deploy path and ward verb, ntfy backup inclusion/docs, stale Tailscale service entries for repo-recall and eco-spec, and stale repo-recall pointer docs if they no longer point to a real service.
Post after evidence: kubectl get ns, namespace memory rollup, and proof that none of repo-recall, ntfy, lunch-money, or coilysiren-eco-spec-tracker appears in container memory rollups.

Main risks

Deleting ntfy before removing the ser8 bridge can make Alertmanager carry a dead receiver and hide notification regressions behind retry noise.
Deleting PVC-backed namespaces without snapshot evidence can lose the only local state for ntfy or any unexpectedly stateful legacy app.
Removing repo-recall may break old agent-context paths if anything still resolves its MCP URL from SSM. The runtime namespace can go, but leave SSM cleanup to a separate explicit secret-retirement decision.
Editing Tailscale Terraform state-backed service lists needs care. Remove desired resources through the current terraform/tailscale stack, do not hand-edit state or revive the retired merge flow.

Remove legacy ntfy alertmanager bridge before kai-server ntfy teardown

Upstream dependency for coilyco-flight-deck/infrastructure#500.

Infrastructure#500 plans to delete the legacy kai-server ntfy namespace and remove ntfy from restic backup coverage. Before that deletion, this repo should stop treating ntfy as a live alert delivery path.

Evidence from the deploy repo:

services/ser8-observability/deploy/vmalert-values.yml still sends Alertmanager notifications to http://ntfy-alertmanager.observability.svc:8000.
services/ser8-observability/deploy/ntfy-alertmanager.yml still defines the bridge workload.
services/ser8-observability/docs/ser8-k3s.md, docs/o11y.md, docs/o11y-sources.md, docs/playbook_kai_server_crashloop_pods.md, and services/ser8-observability/README.md still describe ntfy as part of alert delivery or dual-shipping.

Recommended scope:

Remove the ntfy webhook receiver and route from ser8 Alertmanager values, leaving Telegram as the alert path.
Remove the ntfy-alertmanager manifest if no longer applied.
Update the docs and runbooks so ntfy is not described as live or dual-shipping.
Roll out ser8 observability and verify at least one Telegram alert path or test notification succeeds.

Downstream dependency:

After this lands and is verified, infrastructure#500 can safely delete the kai-server ntfy namespace and remove ntfy from kai-server restic backup configuration without leaving ser8 alerting pointed at a retired service.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Advisory recommendation** This should stay centered on infrastructure#500 for the live kai-server teardown, but it has one real cross-repo dependency: `coilyco-bridge/deploy` still treats ntfy as an alert delivery target for ser8 observability. The infrastructure work should not delete the kai-server ntfy service until that deploy-side ntfy bridge is removed or explicitly accepted as dead. **Decisions already settled** * Retire `repo-recall`, `ntfy`, `lunch-money`, and `coilysiren-eco-spec-tracker` from kai-server. * Do not delete `lunch-money-mcp`. `/substrate/deploy/services/lunch-money-mcp` is the newer deploy-repo bundle, serves `https://lunch-money.coilysiren.me/mcp`, and exposes the local harness at `http://kai-server:30080/mcp`. * Do not broaden the Eco cleanup to `coilysiren-eco-mcp-app` in this issue unless Kai explicitly expands scope. The deploy repo says eco-app replaces both predecessors, but this issue names only `coilysiren-eco-spec-tracker`. * Treat `coilyco-flight-deck/lunch-money-k8s#22` as the source-side retirement track for the old standalone chart/docs. Do not duplicate that cleanup here. **Repo evidence** * `deploy/lunch-money/`, `scripts/k8s/lunch_money.py`, the `Makefile` `lunch-money` target, and `.ward/ward.yaml` still define the old standalone `lunch-money` deployment path in this repo. * `scripts/restic-backup.sh`, `docs/restic-backups.md`, and `docs/FEATURES.md` still include `ntfy` in the PVC backup set. * `terraform/tailscale/services.yaml` still mints service identities for `repo-recall` and `eco-spec`. `scripts/k8s/terraform_tailscale_merge.py` also carries those names in its historical migration lists, so an engineer should inspect whether those references are still live code or only migration history before editing. * The deploy repo no longer has `services/repo-recall`, but this repo still has stale pointer docs at `docs/repo-recall-deploy.md` and `docs/repo-recall-deploy-ops.md`. * `/substrate/deploy/services/eco-app/README.md` and `deploy/main.yml` confirm the fused `eco-app` replaces `coilysiren-eco-spec-tracker` and retires the `eco-spec` tailnet device. * `/substrate/deploy/services/lunch-money-mcp/README.md` confirms the replacement Lunch Money surface is the deploy-repo bundle, not the older Helm release. * `/substrate/deploy/services/ser8-observability` still includes an `ntfy-alertmanager` bridge and `vmalert-values.yml` still sends to it. That is the main blocker to deleting ntfy cleanly. **Open questions for the implementer** * Is Telegram-only alerting now confirmed on ser8 after removing the ntfy bridge? If not, do that first in `coilyco-bridge/deploy`. * Do any of the four target namespaces have PVCs? `ntfy` is expected to, because restic still includes it. The other three may be disposable, but the issue should record the live PVC inventory before deletion. * Are any ExternalSecrets still syncing secrets only for these retired workloads? Likely candidates are `/coilysiren/repo-recall/*`, `/coilysiren/eco-spec/ts-authkey`, `/sentry-dsn/eco-spec-tracker`, and the old `lunch-money-token` ExternalSecret. Do not delete SSM values in this issue unless Kai explicitly wants secret retirement too. * Does `lunch-money-mcp` pass its readiness checks before deleting the old `lunch-money` namespace? Use the deploy repo's documented checks for the metadata endpoint, protected MCP endpoint, and namespace pods. **Recommended execution order** 1. Land the deploy-repo ntfy bridge removal first, then verify ser8 alerting is Telegram-only. 2. On kai-server, capture before evidence: namespace list, namespace memory rollup, and per-namespace inventory for workloads, ingress/service/tailscale sidecars, PVCs, ExternalSecrets, Secrets, and Helm releases. 3. For each namespace with PVCs, either take or verify a restic snapshot, or explicitly record that the service is disposable. 4. Delete the live resources namespace by namespace. Prefer Helm uninstall where a release exists, then remove remaining namespace resources. 5. Remove infrastructure paths that could recreate them: old lunch-money deploy path and ward verb, ntfy backup inclusion/docs, stale Tailscale service entries for `repo-recall` and `eco-spec`, and stale repo-recall pointer docs if they no longer point to a real service. 6. Post after evidence: `kubectl get ns`, namespace memory rollup, and proof that none of `repo-recall`, `ntfy`, `lunch-money`, or `coilysiren-eco-spec-tracker` appears in container memory rollups. **Main risks** * Deleting ntfy before removing the ser8 bridge can make Alertmanager carry a dead receiver and hide notification regressions behind retry noise. * Deleting PVC-backed namespaces without snapshot evidence can lose the only local state for ntfy or any unexpectedly stateful legacy app. * Removing `repo-recall` may break old agent-context paths if anything still resolves its MCP URL from SSM. The runtime namespace can go, but leave SSM cleanup to a separate explicit secret-retirement decision. * Editing Tailscale Terraform state-backed service lists needs care. Remove desired resources through the current `terraform/tailscale` stack, do not hand-edit state or revive the retired merge flow. ### Remove legacy ntfy alertmanager bridge before kai-server ntfy teardown Upstream dependency for `coilyco-flight-deck/infrastructure#500`. Infrastructure#500 plans to delete the legacy kai-server `ntfy` namespace and remove ntfy from restic backup coverage. Before that deletion, this repo should stop treating ntfy as a live alert delivery path. Evidence from the deploy repo: * `services/ser8-observability/deploy/vmalert-values.yml` still sends Alertmanager notifications to `http://ntfy-alertmanager.observability.svc:8000`. * `services/ser8-observability/deploy/ntfy-alertmanager.yml` still defines the bridge workload. * `services/ser8-observability/docs/ser8-k3s.md`, `docs/o11y.md`, `docs/o11y-sources.md`, `docs/playbook_kai_server_crashloop_pods.md`, and `services/ser8-observability/README.md` still describe ntfy as part of alert delivery or dual-shipping. Recommended scope: * Remove the ntfy webhook receiver and route from ser8 Alertmanager values, leaving Telegram as the alert path. * Remove the `ntfy-alertmanager` manifest if no longer applied. * Update the docs and runbooks so ntfy is not described as live or dual-shipping. * Roll out ser8 observability and verify at least one Telegram alert path or test notification succeeds. Downstream dependency: After this lands and is verified, infrastructure#500 can safely delete the kai-server `ntfy` namespace and remove ntfy from kai-server restic backup configuration without leaving ser8 alerting pointed at a retired service. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`