Deprovision legacy kai-server namespaces: repo-recall, ntfy, lunch-money, eco-spec-tracker #500
Labels
No labels
burndown-2026-06
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/advocate
role/director
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/science
role/sysadmin
state
ambient
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
coilyco-flight-deck/infrastructure#500
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Kai asked to deprovision these legacy kai-server workloads after a memory rollup still showed them live:
Decision on the ambiguous Lunch Money pair: retire the older standalone
lunch-moneynamespace, notlunch-money-mcp.lunch-money-mcpis the newer deploy-repo bundle (coilyco-bridge/deploy/services/lunch-money-mcp) andcoilyco-flight-deck/lunch-money-k8s#22already tracks source-side retirement of the older standalone deploy.Please deprovision the live legacy workloads safely:
repo-recall- live namespace still present even thoughcoilyco-bridge/deploy#48closed and the deploy repo no longer hasservices/repo-recall.ntfy- remove the legacy ntfy namespace/service if Telegram cutover is complete. Also remove remaining backup/timer/docs references that still treat ntfy as a live PVC/service.lunch-money- retire the older standalone Helm deploy and namespace after verifyinglunch-money-mcpis the serving replacement.coilysiren-eco-spec-tracker- retire the predecessor namespace now thatservices/eco-appis the fused Eco app. The deploy docs already say eco-app replacescoilysiren-eco-mcp-appandcoilysiren-eco-spec-tracker; this issue only names the spec-tracker namespace Kai requested.Safety requirements:
deploy/lunch-money/,scripts/k8s/lunch_money.py,Makefiletargetlunch-money,scripts/restic-backup.shRESTIC_BACKUP_NAMESPACESincludingntfy, and Tailscale service entries forrepo-recall/eco-spec.coilyco-flight-deck/lunch-money-k8s#22for source-side chart/docs retirement rather than duplicating that work here.kubectl get nsand namespace memory rollup in the issue.Acceptance: the four legacy namespaces above no longer appear in
kubectl top pod -A --containersnamespace rollups, and tracked infra no longer recreates them.WARD-RESERVATION: held 🔒
reservation details
Holder: container
engineer-codex-infrastructure-500on hostkais-macbook-pro-2.local.Reserved by
ward agent --harness codex(reserved 2026-07-09T21:10:15Z). Concurrentward agentruns are blocked until it finishes or the reservation goes stale (1h TTL).--forceoverrides.Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).
run seed context — what this run is carrying (ward#609)
coilyco-flight-deck/infrastructure#500· branchissue-500· harnesscodex· workflowdirect-to-mainengineer-codex-infrastructure-500· wardv0.501.0· dispatched2026-07-09T21:10:15ZStatic container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.501.0).
— Codex, via
ward agent🔎 ward agent advisor
ward agent advisor --harness codexran a one-shot standard research pass on this question:Advisory recommendation
This should stay centered on infrastructure#500 for the live kai-server teardown, but it has one real cross-repo dependency:
coilyco-bridge/deploystill treats ntfy as an alert delivery target for ser8 observability. The infrastructure work should not delete the kai-server ntfy service until that deploy-side ntfy bridge is removed or explicitly accepted as dead.Decisions already settled
repo-recall,ntfy,lunch-money, andcoilysiren-eco-spec-trackerfrom kai-server.lunch-money-mcp./substrate/deploy/services/lunch-money-mcpis the newer deploy-repo bundle, serveshttps://lunch-money.coilysiren.me/mcp, and exposes the local harness athttp://kai-server:30080/mcp.coilysiren-eco-mcp-appin this issue unless Kai explicitly expands scope. The deploy repo says eco-app replaces both predecessors, but this issue names onlycoilysiren-eco-spec-tracker.coilyco-flight-deck/lunch-money-k8s#22as the source-side retirement track for the old standalone chart/docs. Do not duplicate that cleanup here.Repo evidence
deploy/lunch-money/,scripts/k8s/lunch_money.py, theMakefilelunch-moneytarget, and.ward/ward.yamlstill define the old standalonelunch-moneydeployment path in this repo.scripts/restic-backup.sh,docs/restic-backups.md, anddocs/FEATURES.mdstill includentfyin the PVC backup set.terraform/tailscale/services.yamlstill mints service identities forrepo-recallandeco-spec.scripts/k8s/terraform_tailscale_merge.pyalso carries those names in its historical migration lists, so an engineer should inspect whether those references are still live code or only migration history before editing.services/repo-recall, but this repo still has stale pointer docs atdocs/repo-recall-deploy.mdanddocs/repo-recall-deploy-ops.md./substrate/deploy/services/eco-app/README.mdanddeploy/main.ymlconfirm the fusedeco-appreplacescoilysiren-eco-spec-trackerand retires theeco-spectailnet device./substrate/deploy/services/lunch-money-mcp/README.mdconfirms the replacement Lunch Money surface is the deploy-repo bundle, not the older Helm release./substrate/deploy/services/ser8-observabilitystill includes anntfy-alertmanagerbridge andvmalert-values.ymlstill sends to it. That is the main blocker to deleting ntfy cleanly.Open questions for the implementer
coilyco-bridge/deploy.ntfyis expected to, because restic still includes it. The other three may be disposable, but the issue should record the live PVC inventory before deletion./coilysiren/repo-recall/*,/coilysiren/eco-spec/ts-authkey,/sentry-dsn/eco-spec-tracker, and the oldlunch-money-tokenExternalSecret. Do not delete SSM values in this issue unless Kai explicitly wants secret retirement too.lunch-money-mcppass its readiness checks before deleting the oldlunch-moneynamespace? Use the deploy repo's documented checks for the metadata endpoint, protected MCP endpoint, and namespace pods.Recommended execution order
repo-recallandeco-spec, and stale repo-recall pointer docs if they no longer point to a real service.kubectl get ns, namespace memory rollup, and proof that none ofrepo-recall,ntfy,lunch-money, orcoilysiren-eco-spec-trackerappears in container memory rollups.Main risks
repo-recallmay break old agent-context paths if anything still resolves its MCP URL from SSM. The runtime namespace can go, but leave SSM cleanup to a separate explicit secret-retirement decision.terraform/tailscalestack, do not hand-edit state or revive the retired merge flow.Remove legacy ntfy alertmanager bridge before kai-server ntfy teardown
Upstream dependency for
coilyco-flight-deck/infrastructure#500.Infrastructure#500 plans to delete the legacy kai-server
ntfynamespace and remove ntfy from restic backup coverage. Before that deletion, this repo should stop treating ntfy as a live alert delivery path.Evidence from the deploy repo:
services/ser8-observability/deploy/vmalert-values.ymlstill sends Alertmanager notifications tohttp://ntfy-alertmanager.observability.svc:8000.services/ser8-observability/deploy/ntfy-alertmanager.ymlstill defines the bridge workload.services/ser8-observability/docs/ser8-k3s.md,docs/o11y.md,docs/o11y-sources.md,docs/playbook_kai_server_crashloop_pods.md, andservices/ser8-observability/README.mdstill describe ntfy as part of alert delivery or dual-shipping.Recommended scope:
ntfy-alertmanagermanifest if no longer applied.Downstream dependency:
After this lands and is verified, infrastructure#500 can safely delete the kai-server
ntfynamespace and remove ntfy from kai-server restic backup configuration without leaving ser8 alerting pointed at a retired service.Researched and posted automatically by
ward agent advisor --harness codex(ward#179). This is one-shot research, not a carried change - verify before acting on it.— Codex, via
ward agentFour of five done. One is not, and it is not a cleanup any more.
Gone:
repo-recall,ntfy,coilysiren-eco-spec-tracker. None appear inkubectl get ns.lunch-money-mcpcorrectly survives, per the decision recorded in the body.Still running: the standalone
lunch-moneynamespace, which this issue decided to retire.That is not a stalled teardown. It is a healthy workload that has been restarted as recently as 24 days ago and has been serving for 94. The memory rollup that triggered this issue showed it at 2Mi, which is why it read as a leftover, but something is keeping it alive.
So the question is no longer "finish the deprovision", it is "was retiring this still the right call". Deleting a running service on the strength of a decision made when it looked dormant is the kind of thing worth re-confirming rather than executing from an old issue. Leaving open and re-scoped.
Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901).
Closing as substantially done. repo-recall, ntfy, and eco-spec-tracker are gone and lunch-money-mcp correctly survives. Whether to retire the still-healthy standalone lunch-money namespace is a fresh decision, not the leftover of this cleanup, and belongs in its own issue.
This was
priority/P3, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless.If this is live work, reopen it. The whole set is recoverable with
state:closed label:burndown-2026-08.