coilyco-flight-deck/infrastructure
Watch
2
0
Code Issues 123 Pull requests Projects Releases Packages Wiki Activity Actions

terraform/aws-inventory: decide whether comfyui.coilysiren.me should exist #914

New issue
Closed
opened 2026-08-25 01:06:54 +00:00 by coilyco-ops · 3 comments
coilyco-ops commented 2026-08-25 01:06:54 +00:00
Member
Copy link

Found while landing #911. Not caused by that change, and deliberately not fixed there, because a DNS mail change is the wrong blast radius to carry a state repair.

The hazard

just terraform-aws-inventory apply plans 3 to destroy today. Verified 2026-08-24 against the live state with a full terraform plan:

Plan: 2 to add, 1 to change, 3 to destroy.

The three destroys:

  • aws_route53_record.home_a["eco-jobs-tracker"]
  • aws_route53_record.home_a["eco-mcp"]
  • aws_route53_record.home_a["grafana"]

All three resolve live right now. dig +short @8.8.8.8 <host>.coilysiren.me A returns the home public IP for each. An apply removes them from DNS.

Why they drifted

They were dropped from the home_a for_each map during the ExternalDNS migration (#339) without a matching terraform state rm. Terraform still holds them in state, so absence from the map reads as "delete this".

ExternalDNS did not adopt them either. docs/external-dns.md says ownership is tracked in companion extdns-* TXT records, and none exists:

dig +short @8.8.8.8 extdns-a-eco-jobs-tracker.coilysiren.me TXT   # empty
dig +short @8.8.8.8 extdns-a-eco-mcp.coilysiren.me TXT            # empty
dig +short @8.8.8.8 extdns-a-grafana.coilysiren.me TXT            # empty

So each record is currently owned by nothing: out of the terraform config, not in the ExternalDNS registry, still serving traffic.

Second, smaller item

aws_route53_record.comfyui plans as create. The CNAME is in main.tf but absent from state, and comfyui.coilysiren.me does not resolve today. The record appears never to have been applied.

What resolving this needs

Two decisions, per host, that an operator has to make rather than an agent guess:

  1. Whether each of eco-jobs-tracker, eco-mcp, grafana should still resolve at all. If yes, decide the owner: back into the home_a map, or migrated to ExternalDNS with the label, hostname annotation, and the target annotation that keeps it on the home public IP rather than the tailnet IP. If no, an apply deleting it is correct and this becomes a no-op.
  2. Whether comfyui.coilysiren.me is still wanted. If yes, the create is correct and the record just needs applying.

Until one of those lands, every apply against this module must use -target. #911 was landed that way.

Done means

just terraform-aws-inventory plan is clean, or its remaining diff is intentional and reviewed, and no live A record is scheduled for deletion by accident.

Found while landing #911. Not caused by that change, and deliberately not fixed there, because a DNS mail change is the wrong blast radius to carry a state repair. ## The hazard `just terraform-aws-inventory apply` plans **3 to destroy** today. Verified 2026-08-24 against the live state with a full `terraform plan`: ``` Plan: 2 to add, 1 to change, 3 to destroy. ``` The three destroys: * `aws_route53_record.home_a["eco-jobs-tracker"]` * `aws_route53_record.home_a["eco-mcp"]` * `aws_route53_record.home_a["grafana"]` All three resolve live right now. `dig +short @8.8.8.8 <host>.coilysiren.me A` returns the home public IP for each. An apply removes them from DNS. ## Why they drifted They were dropped from the `home_a` `for_each` map during the ExternalDNS migration (#339) without a matching `terraform state rm`. Terraform still holds them in state, so absence from the map reads as "delete this". ExternalDNS did not adopt them either. `docs/external-dns.md` says ownership is tracked in companion `extdns-*` TXT records, and none exists: ``` dig +short @8.8.8.8 extdns-a-eco-jobs-tracker.coilysiren.me TXT # empty dig +short @8.8.8.8 extdns-a-eco-mcp.coilysiren.me TXT # empty dig +short @8.8.8.8 extdns-a-grafana.coilysiren.me TXT # empty ``` So each record is currently owned by nothing: out of the terraform config, not in the ExternalDNS registry, still serving traffic. ## Second, smaller item `aws_route53_record.comfyui` plans as **create**. The CNAME is in `main.tf` but absent from state, and `comfyui.coilysiren.me` does not resolve today. The record appears never to have been applied. ## What resolving this needs Two decisions, per host, that an operator has to make rather than an agent guess: 1. Whether each of `eco-jobs-tracker`, `eco-mcp`, `grafana` should still resolve at all. If yes, decide the owner: back into the `home_a` map, or migrated to ExternalDNS with the label, hostname annotation, and the `target` annotation that keeps it on the home public IP rather than the tailnet IP. If no, an apply deleting it is correct and this becomes a no-op. 2. Whether `comfyui.coilysiren.me` is still wanted. If yes, the create is correct and the record just needs applying. Until one of those lands, **every apply against this module must use `-target`**. #911 was landed that way. ## Done means `just terraform-aws-inventory plan` is clean, or its remaining diff is intentional and reviewed, and no live A record is scheduled for deletion by accident.
coilyco-ops commented 2026-08-25 01:07:05 +00:00
Author
Member
Copy link

Correction to the last line of the body: #911's apply has not run yet, it is pending operator approval. Its plan is targeted and clean at 1 to add, 1 to change, 0 to destroy, so the -target rule stated above is what #911 will be applied under, not something already done.

Correction to the last line of the body: #911's apply has not run yet, it is pending operator approval. Its plan is targeted and clean at `1 to add, 1 to change, 0 to destroy`, so the `-target` rule stated above is what #911 will be applied under, not something already done.
coilyco-ops commented 2026-08-25 02:48:08 +00:00
Author
Member
Copy link

Deleted. Kai's call. Apply complete! Resources: 0 added, 0 changed, 3 destroyed.

Evidence gathered before deleting

Four independent confirmations that all three were dead:

  • scripts/k8s/retire-zombie-namespaces.sh:148 explicitly instructs an operator to delete the eco-mcp and eco-jobs-tracker records by hand. The deletion was already the intended end state, it just never happened.
  • That script's retired-namespace list carries coilysiren-eco-mcp-app and coilysiren-eco-spec-tracker, the k8s namespaces behind those two hostnames. (eco-spec-tracker is the old internal name for eco-jobs-tracker, per its repo digest.)
  • All three served CN=TRAEFIK DEFAULT CERT and returned 404. No ingress, no cert-manager certificate, nothing configured for those hosts.
  • The control, eco-app.coilysiren.me, served a real CN=eco-app.coilysiren.me Let's Encrypt certificate and returned 200. The Eco SPA, MCP, jobs, and worker service were fused into it, which is why the two Eco names were retired. grafana predates SigNoz becoming the canonical observability pane.

After-state

Authoritative (ns-1394.awsdns-46.org) and public (8.8.8.8) both empty:

eco-jobs-tracker.coilysiren.me  A  ->  (none)
eco-mcp.coilysiren.me           A  ->  (none)
grafana.coilysiren.me           A  ->  (none)

Survivors intact, all still on the home IP:

eco  eco-app  factorio  galaxy-gen

eco-app.coilysiren.me still returns 200.

The module is no longer dangerous

The full untargeted plan is now:

Plan: 1 to add, 0 to change, 0 to destroy.

Zero destroys. The hazard this issue was opened for is closed. An untargeted apply can no longer delete a live record.

#919 landed alongside, removing the now-stale hand-delete instruction from the retirement script.

One item left, and it needs a decision

aws_route53_record.comfyui still plans as a create. It is the whole remaining diff.

What I found:

  • comfyui.coilysiren.me does not resolve and never has. The CNAME has been in main.tf without ever being applied.
  • The comfyui namespace is in retire-zombie-namespaces.sh's retired list, and that script separately cleans up ComfyUI strays left in the forgejo namespace.
  • The CNAME target is a tailnet hostname, comfyui.tail09a41b.ts.net., not the cluster. So retiring the k8s namespace does not by itself prove the tailnet ComfyUI is gone.
  • tailscale status on this host shows no comfyui peer, but that is not conclusive, since peer visibility depends on ACLs.
  • I could not enumerate tailnet devices to settle it. just list-tailscale-devices needs TAILSCALE_API_KEY or the OAuth pair in the shell, and docs/tailscale.md says admin creds stay out of SSM by design. That is an attended check.

Leaning toward removing the CNAME from main.tf, since the k8s side was retired and the record has never existed. If ComfyUI returns on the tailnet it is a one-line re-add. But that is a decision rather than a cleanup, so leaving this open for Kai.

Deleted. Kai's call. `Apply complete! Resources: 0 added, 0 changed, 3 destroyed.` ## Evidence gathered before deleting Four independent confirmations that all three were dead: * `scripts/k8s/retire-zombie-namespaces.sh:148` explicitly instructs an operator to delete the `eco-mcp` and `eco-jobs-tracker` records by hand. The deletion was already the intended end state, it just never happened. * That script's retired-namespace list carries `coilysiren-eco-mcp-app` and `coilysiren-eco-spec-tracker`, the k8s namespaces behind those two hostnames. (`eco-spec-tracker` is the old internal name for `eco-jobs-tracker`, per its repo digest.) * All three served `CN=TRAEFIK DEFAULT CERT` and returned `404`. No ingress, no cert-manager certificate, nothing configured for those hosts. * The control, `eco-app.coilysiren.me`, served a real `CN=eco-app.coilysiren.me` Let's Encrypt certificate and returned `200`. The Eco SPA, MCP, jobs, and worker service were fused into it, which is why the two Eco names were retired. `grafana` predates SigNoz becoming the canonical observability pane. ## After-state Authoritative (`ns-1394.awsdns-46.org`) and public (`8.8.8.8`) both empty: ``` eco-jobs-tracker.coilysiren.me A -> (none) eco-mcp.coilysiren.me A -> (none) grafana.coilysiren.me A -> (none) ``` Survivors intact, all still on the home IP: ``` eco eco-app factorio galaxy-gen ``` `eco-app.coilysiren.me` still returns `200`. ## The module is no longer dangerous The full untargeted plan is now: ``` Plan: 1 to add, 0 to change, 0 to destroy. ``` Zero destroys. The hazard this issue was opened for is closed. An untargeted apply can no longer delete a live record. #919 landed alongside, removing the now-stale hand-delete instruction from the retirement script. ## One item left, and it needs a decision `aws_route53_record.comfyui` still plans as a create. It is the whole remaining diff. What I found: * `comfyui.coilysiren.me` does not resolve and never has. The CNAME has been in `main.tf` without ever being applied. * The `comfyui` namespace is in `retire-zombie-namespaces.sh`'s retired list, and that script separately cleans up ComfyUI strays left in the `forgejo` namespace. * The CNAME target is a tailnet hostname, `comfyui.tail09a41b.ts.net.`, not the cluster. So retiring the k8s namespace does not by itself prove the tailnet ComfyUI is gone. * `tailscale status` on this host shows no `comfyui` peer, but that is not conclusive, since peer visibility depends on ACLs. * I could not enumerate tailnet devices to settle it. `just list-tailscale-devices` needs `TAILSCALE_API_KEY` or the OAuth pair in the shell, and `docs/tailscale.md` says admin creds stay out of SSM by design. That is an attended check. Leaning toward removing the CNAME from `main.tf`, since the k8s side was retired and the record has never existed. If ComfyUI returns on the tailnet it is a one-line re-add. But that is a decision rather than a cleanup, so leaving this open for Kai.
coilyco-ops changed title from terraform/aws-inventory carries live-DNS-destroying drift: an untargeted apply deletes three A records to terraform/aws-inventory: decide whether comfyui.coilysiren.me should exist 2026-08-25 02:48:16 +00:00
coilyco-ops commented 2026-08-25 04:35:04 +00:00
Author
Member
Copy link

Done, both halves. Closing.

DNS side

aws_route53_record.comfyui removed in #920, squashed to a8751f7 on main. The module now plans with zero resource changes against live state. Only the computed outputs move, route53_records from 17 to 15.

Tower side

tailscale serve --bg --https=443 http://127.0.0.1:8188 is running on kai-tower-3026.

Before-state captured first as a rollback point. The node already had a raw TCP forward and nothing else:

{"TCP": {"8188": {"TCPForward": "127.0.0.1:8188"}}}

That is passthrough with no TLS termination, which is why 8188 was reachable but 443 was closed.

After, verified from a separate tailnet host:

  • HTTPS on 443 returns 200, and /system_stats reports ComfyUI 0.29.0 on win32.
  • The certificate is a real Let's Encrypt leaf for the tower's own tailnet name, issuer C=US, O=Let's Encrypt, CN=YE1, valid 2026-08-25 to 2026-11-23. curl accepts it with no -k, so it is browser-trusted.
  • The pre-existing :8188 path still returns 200. Nothing that worked before was broken.

Rollback if ever needed is tailscale serve --https=443 off, which leaves the original TCP forward intact.

Why the CNAME could not have been repointed

Recorded for the next person who wonders. Three things were broken, and a repoint only fixes the first.

  1. The target comfyui.<tailnet>.ts.net was a Tailscale sidecar device belonging to the k3s ComfyUI pod, joined with /coilysiren/comfyui/ts-authkey. retire-zombie-namespaces.sh retired that namespace and the device went with it.
  2. This tailnet's .ts.net names are not in public DNS. Checked against 8.8.8.8: neither the dead sidecar name nor the tower's own name resolves. A public CNAME into one dead-ends unless the client's resolver hands .ts.net to MagicDNS, which is client-side split-DNS behavior rather than something the record can guarantee. Most likely why this record was written and never applied.
  3. Nothing on the tower held a certificate for a coilysiren.me name, and 443 and 80 were both closed. The original browser-trusted URL came from cert-manager issuing a Let's Encrypt certificate inside k3s with the sidecar carrying traffic, and that retired with the namespace.

tailscale serve reaches the same goal through the mechanism designed for it, with no Route53 record and no renewal to babysit.

Whole issue, closed out

  • Three orphaned A records deleted, 0 added, 0 changed, 3 destroyed.
  • Stale hand-delete instruction removed from the retirement script (#919).
  • comfyui CNAME removed, ComfyUI published over HTTPS instead (#920).
  • An untargeted just terraform-aws-inventory apply is now safe. The -target discipline this issue forced is no longer load-bearing.

Optional follow-up, not filed

The serve config lives in tailscaled's own state, so it survives reboots without help. It would not survive a host rebuild. If that matters, it belongs in a converge script beside scripts/converge-ollama-windows.ps1. Not filing an issue for it, since the durability that actually matters day to day is already there.

Done, both halves. Closing. ## DNS side `aws_route53_record.comfyui` removed in #920, squashed to `a8751f7` on `main`. The module now plans with **zero resource changes** against live state. Only the computed outputs move, `route53_records` from 17 to 15. ## Tower side `tailscale serve --bg --https=443 http://127.0.0.1:8188` is running on `kai-tower-3026`. Before-state captured first as a rollback point. The node already had a raw TCP forward and nothing else: ```json {"TCP": {"8188": {"TCPForward": "127.0.0.1:8188"}}} ``` That is passthrough with no TLS termination, which is why 8188 was reachable but 443 was closed. After, verified from a separate tailnet host: * HTTPS on 443 returns `200`, and `/system_stats` reports ComfyUI 0.29.0 on win32. * The certificate is a real Let's Encrypt leaf for the tower's own tailnet name, issuer `C=US, O=Let's Encrypt, CN=YE1`, valid 2026-08-25 to 2026-11-23. `curl` accepts it with no `-k`, so it is browser-trusted. * The pre-existing `:8188` path still returns `200`. Nothing that worked before was broken. Rollback if ever needed is `tailscale serve --https=443 off`, which leaves the original TCP forward intact. ## Why the CNAME could not have been repointed Recorded for the next person who wonders. Three things were broken, and a repoint only fixes the first. 1. The target `comfyui.<tailnet>.ts.net` was a Tailscale **sidecar** device belonging to the k3s ComfyUI pod, joined with `/coilysiren/comfyui/ts-authkey`. `retire-zombie-namespaces.sh` retired that namespace and the device went with it. 2. This tailnet's `.ts.net` names are not in public DNS. Checked against `8.8.8.8`: neither the dead sidecar name nor the tower's own name resolves. A public CNAME into one dead-ends unless the client's resolver hands `.ts.net` to MagicDNS, which is client-side split-DNS behavior rather than something the record can guarantee. Most likely why this record was written and never applied. 3. Nothing on the tower held a certificate for a `coilysiren.me` name, and 443 and 80 were both closed. The original browser-trusted URL came from cert-manager issuing a Let's Encrypt certificate inside k3s with the sidecar carrying traffic, and that retired with the namespace. `tailscale serve` reaches the same goal through the mechanism designed for it, with no Route53 record and no renewal to babysit. ## Whole issue, closed out * Three orphaned A records deleted, `0 added, 0 changed, 3 destroyed`. * Stale hand-delete instruction removed from the retirement script (#919). * `comfyui` CNAME removed, ComfyUI published over HTTPS instead (#920). * An untargeted `just terraform-aws-inventory apply` is now safe. The `-target` discipline this issue forced is no longer load-bearing. ## Optional follow-up, not filed The serve config lives in tailscaled's own state, so it survives reboots without help. It would not survive a host rebuild. If that matters, it belongs in a converge script beside `scripts/converge-ollama-windows.ps1`. Not filing an issue for it, since the durability that actually matters day to day is already there.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
aos/claude/wz68-mirror-push
aos/claude/wz68-mirror-watch
aos/claude/wz68-bot-identity
aos/claude/wz68-bot-handle
aos/claude/wz68
aos/claude/ff54-retire-coilyco-ai-hook
aos/claude/ff54-retire-issue-refs
aos/claude/ue86
aos/claude/yq86
aos/claude/ser8-image-build-929
ops/zombie-retire-secret-strays
ops/917-alert-unit-name
ops/917-restic-failure-alert
ops/914-drop-comfyui-cname
ops/914-drop-stale-dns-instruction
ops/retire-coilysiren-me-mail
ops/911-dns-mail-auth
aos/claude/fj88
ops/393-aos-pin-rollout
ops/708-o11y-telegram-per-host-cache
chore/claude-permissions-allow-all
fix/comfyui-preflight-whatif
fix/comfyui-supervisor-job-object
fix/forgejo-creds-windows-ssm
fix/comfyui-orphan-on-refused-restart
fix/windows-projects-root
fix/comfyui-restart-race
windows-tasks-s4u
windows-attribution-bash-shim-python
fix/comfyui-whatif-verification
windows-terminal-kai-tower-3026
fix/setup-git-lfs-windows-path
chore/drop-ward-yaml
fix/terraform-docs-action-arg
fix/mint-key-orphan
feat/public-asset-host
aos/claude/ka54-failclosed
aos/claude/ka54-cleanup
aos/claude/ka54
aos/claude/uq98-unattended-forgejo
aos/claude/uq98-disk-domains
aos/claude/xk58
aos/claude/aw85-bands
aos/claude/ym96-docs-band
aos/claude/wt57-alerts-to-mapper
aos/claude/xu59-role-trailer
aos/claude/ym96-verb-convert
aos/claude/ym96-token-cleanup
aos/claude/ym96-label-sync
aos/claude/wt57-atlas-cleanup
fix/mirror-unskip-sirens-echo
aos/claude/uz86-session-name
retire-code-fleet-watch-830
codex/alacritty-link-rollout
chore/mcp-beaver-fleet-watch
aos/claude/attribution-interpreter
aos/claude/mg96-fm
aos/claude/js44
coilyco-ops/deprovision-tangled-knot
aos/claude/mg96
aos/claude/zv96
outdated-branch-815
aos/claude/agent-proxy-publisher-runner
aos/claude/git-attribution-pin-interpreter
aos/claude/restore-tag-quarantine
aos/claude/mirror-workflows-permission
aos/claude/forgejo-16
aos/claude/dind-mtu
aos/claude/precommit-egress-proxy
aos/claude/qa57-branch-protection
chore/egress-proxy-open
chore/egress-allow-code-forgejo
aos/claude/qa57
chore/aos-precommit-v0.18-bump
aos/claude/ck67
infra/claude/ser8-runner-force-pull
aos/claude/runner-scratch-10gi
aos/claude/runner-recycle-selector
aos/claude/xk86
aos/codex/20260806t002312z-7943-00371332
codex/sirens-echo-deepseek-tailnet-20260805
codex/daily-aos-sync
issue-756-remove-role-provider-symlink
aos/codex/sirens-echo-runner-rename
aos/codex/20260731t171534z-9817-e7959fba
issue-697-aos-check-config
issue-694-aos-convergence
ops/remove-mac-models-launchd-20260729
ops/remove-mac-models-20260729
tailscale-proxy-converge-self-heal
issue-587
backup/issue-785-first-person
issue-785-first-person
ops/galaxy-gen-registry-rotate
ops/website-deploy-runner
codex/history-derived-skill-20260728
ops/k3s-pod-capacity-config
ops/publisher-runners
codex/code-fleet-watch-v2
codex/code-fleet-watch
recovery/2026-07-28-triaged-branch-archive
recovery/2026-07-27-local-work
recovery/machine-exit-20260727
issue-653-registry-retirement
issue-593
fix/aws-iam-ssm-import
feat/warded-ssm-identity
issue-500
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
burndown-2026-08
Closed in the 2026-08-26 backlog burn-down. Reopen freely: state:closed label:burndown-2026-08 recovers the whole set.

  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
qa-fixture
Disposable issue admitted to the bounded Ward QA verification lane.

  
role/devrel
requires work from the Developer Advocate seat

  
role/eval
requires work from the Agent Evaluation Engineer seat

  
role/exec
requires work from the exec role

  
role/frontend
requires work from the Frontend Design Engineer seat

  
role/gamedev
requires work from the Game Developer seat

  
role/human
requires a person, and specifically not an agent seat

  
role/platform
requires work from the Developer Platform Engineer seat

  
role/qa
requires work from the QA role

  
role/sysadmin
requires work from the Systems Administrator seat

  
role/tpm
requires work from the Technical Program Manager seat

No labels
burndown-2026-06
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/devrel
role/eval
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/sysadmin
role/tpm
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/infrastructure#914
No description provided.