coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

Retire hosted Sentry → GlitchTip (phase-3 DSN cutover) #395

New issue
Open
opened 2026-06-24 03:38:28 +00:00 by coilyco-ops · 2 comments
coilyco-ops commented 2026-06-24 03:38:28 +00:00
Member
Copy link

Retire hosted Sentry → GlitchTip (phase-3 DSN cutover)

GlitchTip phase 1 is up and tailnet-reachable (https://glitchtip.<tailnet>.ts.net, HTTP 200, smoke-verified 2026-06-18). This issue tracks the actual Sentry retirement. It supersedes the stale "5 app repos" line in docs/glitchtip-deploy-plan.md (phase 3) — that list is wrong: coily is archived and isn't a Sentry producer, eco-spec-tracker + eco-telemetry were fused into eco-app, and the list missed galaxy-gen + website.

Mechanism (not per-repo code)

Producers read SENTRY_DSN from an SSM /sentry-dsn/* SecureString via an ExternalSecret; the SDK is Sentry-API-compatible. Cutover per producer = overwrite its SSM /sentry-dsn/* value with the GlitchTip DSN, refresh the ExternalSecret, restart the pod. No app-repo PR needed.

Blocker (human, UI)

Create the GlitchTip org + one project per producer to mint the DSNs (and rotate the still-unrotated admin password in SSM /glitchtip/admin-password). GlitchTip currently has 0 orgs/projects. Nothing downstream can proceed until the DSNs exist.

Live producers to cut (one at a time)

  • backend — SSM /sentry-dsn/backend
  • eco-app (fused) — SSM /sentry-dsn/eco-mcp-app
  • galaxy-gen (errors-only) — SSM /sentry-dsn/galaxy-gen
  • website (errors-only) — SSM /sentry-dsn/website
  • kai-server thermal-heartbeat host cron — SSM /sentry-dsn/kai-server

Retire stale params (no live producer)

  • /sentry-dsn/eco-spec-tracker (folded into eco-app)
  • /sentry-dsn/eco-telemetry (folded into eco-app)
  • /sentry/projects (already unconsumed, safe to delete)

Teardown (externally visible — staged, human pulls the trigger)

  • Confirm all live producers emitting to GlitchTip (fire a test error per project, confirm it lands).
  • Decommission the hosted Sentry org/projects.
  • Update docs/glitchtip-deploy-plan.md phase 3 + deploy repo docs/o11y-sources.md / docs/o11y.md once Sentry is gone.

Related: GlitchTip deploy #137, ser8 move #320.

## Retire hosted Sentry → GlitchTip (phase-3 DSN cutover) GlitchTip phase 1 is up and tailnet-reachable (`https://glitchtip.<tailnet>.ts.net`, HTTP 200, smoke-verified 2026-06-18). This issue tracks the actual Sentry retirement. It **supersedes the stale "5 app repos" line** in `docs/glitchtip-deploy-plan.md` (phase 3) — that list is wrong: `coily` is archived and isn't a Sentry producer, `eco-spec-tracker` + `eco-telemetry` were fused into `eco-app`, and the list missed `galaxy-gen` + `website`. ### Mechanism (not per-repo code) Producers read `SENTRY_DSN` from an SSM `/sentry-dsn/*` SecureString via an ExternalSecret; the SDK is Sentry-API-compatible. **Cutover per producer = overwrite its SSM `/sentry-dsn/*` value with the GlitchTip DSN, refresh the ExternalSecret, restart the pod.** No app-repo PR needed. ### Blocker (human, UI) Create the GlitchTip org + one project per producer to mint the DSNs (and rotate the still-unrotated admin password in SSM `/glitchtip/admin-password`). GlitchTip currently has 0 orgs/projects. Nothing downstream can proceed until the DSNs exist. ### Live producers to cut (one at a time) - [ ] `backend` — SSM `/sentry-dsn/backend` - [ ] `eco-app` (fused) — SSM `/sentry-dsn/eco-mcp-app` - [ ] `galaxy-gen` (errors-only) — SSM `/sentry-dsn/galaxy-gen` - [ ] `website` (errors-only) — SSM `/sentry-dsn/website` - [ ] kai-server `thermal-heartbeat` host cron — SSM `/sentry-dsn/kai-server` ### Retire stale params (no live producer) - [ ] `/sentry-dsn/eco-spec-tracker` (folded into eco-app) - [ ] `/sentry-dsn/eco-telemetry` (folded into eco-app) - [ ] `/sentry/projects` (already unconsumed, safe to delete) ### Teardown (externally visible — staged, human pulls the trigger) - [ ] Confirm all live producers emitting to GlitchTip (fire a test error per project, confirm it lands). - [ ] Decommission the hosted Sentry org/projects. - [ ] Update `docs/glitchtip-deploy-plan.md` phase 3 + deploy repo `docs/o11y-sources.md` / `docs/o11y.md` once Sentry is gone. Related: GlitchTip deploy #137, ser8 move #320.
coilyco-ops commented 2026-06-24 05:38:11 +00:00
Author
Member
Copy link

Triage: P2 / consult - Sentry->GlitchTip cutover gated on a human UI step (create GlitchTip org+projects) then externally-visible teardown.

(goose-style pass; claude as the judgment engine, applied 2026-06-23)

<!-- goose-triage --> **Triage:** `P2` / `consult` - Sentry->GlitchTip cutover gated on a human UI step (create GlitchTip org+projects) then externally-visible teardown. (goose-style pass; claude as the judgment engine, applied 2026-06-23)
coilyco-ops commented 2026-07-10 13:19:49 +00:00
Author
Member
Copy link

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Advisory recommendation

Keep this as one infrastructure issue. The cutover is mostly an SSM/runtime operation, not app code, but it has two hidden gates that should be explicit before anyone starts overwriting DSNs:

  • The GlitchTip UI gate is still first: rotate /glitchtip/admin-password, create the coilysiren org, then create one real project each for backend, eco-app, galaxy-gen, website, and kai-server.
  • Public reachability is a gate for browser producers. website is a Gatsby site using @sentry/gatsby, and galaxy-gen has a browser SDK. They cannot use a tailnet-only DSN from user browsers. Do not cut those two until https://glitchtip.coilysiren.me is live, or until there is a deliberate alternate public ingest path.

Decisions found

  • The stale phase-3 producer list in docs/glitchtip-deploy-plan-design.md should be treated as superseded by this issue. The current live list is backend, fused eco-app, galaxy-gen, website, and kai-server thermal-heartbeat.
  • backend and eco-app already consume SENTRY_DSN through ExternalSecrets backed by /sentry-dsn/backend and /sentry-dsn/eco-mcp-app. No app repo PR is needed for those.
  • thermal-heartbeat has two Sentry-facing paths: /sentry-dsn/kai-server for breach events, and /kai-server/thermal-heartbeat-cron-url for Sentry cron check-ins. Hosted Sentry cannot be fully retired until the cron monitor path has a GlitchTip uptime heartbeat replacement or another explicit owner.
  • GlitchTip phase 1 is genuinely complete: the docs say the smoke org/project was deleted afterward, so the current 0 orgs/projects state is expected, not a failed deploy.

Options

  • Option A: wait for public GlitchTip, then cut everything. Simplest mental model and required for browser producers. Risk: in-cluster producers may fail if glitchtip.coilysiren.me hairpins through the home WAN path. Verify from an app pod before using one public DSN host everywhere.
  • Option B: staged endpoint strategy. Use a reachable internal or tailnet host for server-side producers, use the public host for browser producers, and keep the project ids/public keys the same. This is more operationally fiddly, but avoids blocking backend and eco-app on public ingress.
  • Option C: cut only server-side producers now, defer browser producers and teardown. Fastest useful progress, but hosted Sentry remains live until website, galaxy-gen, and the thermal cron monitor are solved.

Recommendation: use Option C first, then finish with Option A or B after public reachability is verified. Start with backend because it has a known forced-exception smoke path (GET /explode in the repo digest), then eco-app, then kai-server breach events. Leave website and galaxy-gen until the public GlitchTip hostname is live.

Suggested cutover runbook per producer

  1. Create the GlitchTip project and copy its DSN.
  2. Decide the DSN hostname based on where the producer runs: pod-internal or tailnet for server-side producers, public glitchtip.coilysiren.me for browser producers.
  3. Overwrite the matching /sentry-dsn/* SecureString only when ready to cut that one producer.
  4. Force the ExternalSecret refresh instead of waiting for the 1h interval, then restart the pod or host timer.
  5. Fire a producer-specific test error and confirm it lands in the right GlitchTip project.
  6. Confirm the old hosted Sentry project stops receiving new events before moving to the next producer.

Risks and open questions

  • DSN hostname reachability is the biggest technical risk. A DSN minted while GLITCHTIP_DOMAIN points at the tailnet URL is not suitable for browser-side SDKs. A public DSN may not be suitable from pods unless hairpin or internal routing is verified.
  • Do not decommission hosted Sentry while /kai-server/thermal-heartbeat-cron-url still points at Sentry. GlitchTip supports heartbeat-style uptime monitors, but that is not the same API as Sentry cron check-ins, so the monitor URL needs its own migration decision.
  • Backend and eco-app docs describe Sentry as an error/log sink in places. The phase-3 acceptance test should verify the exact signal Kai cares about, not just that one exception appears.
  • Keep hosted Sentry org/projects until every live producer has a confirmed GlitchTip test event and at least one quiet period in Sentry. The SSM parameter version history gives some rollback help, but deleting hosted projects removes the cleanest rollback target.
  • After all live producers are cut, delete only the confirmed stale params first: /sentry-dsn/eco-spec-tracker, /sentry-dsn/eco-telemetry, and /sentry/projects. Audit /sentry/personal-auth-token separately during final hosted-Sentry teardown rather than silently deleting it in this pass.

Documentation cleanup

Once the cutover is proven and hosted Sentry is actually gone, update:

  • docs/glitchtip-deploy-plan.md phase 3 and the stale list in docs/glitchtip-deploy-plan-design.md.
  • deploy repo docs/o11y-sources.md and docs/o11y.md, which still describe Sentry as live for the old producer set.
  • the thermal-heartbeat docs if its cron monitor moves from Sentry to GlitchTip uptime heartbeat or another alerting path.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Advisory recommendation** Keep this as one infrastructure issue. The cutover is mostly an SSM/runtime operation, not app code, but it has two hidden gates that should be explicit before anyone starts overwriting DSNs: * The GlitchTip UI gate is still first: rotate `/glitchtip/admin-password`, create the `coilysiren` org, then create one real project each for `backend`, `eco-app`, `galaxy-gen`, `website`, and `kai-server`. * Public reachability is a gate for browser producers. `website` is a Gatsby site using `@sentry/gatsby`, and `galaxy-gen` has a browser SDK. They cannot use a tailnet-only DSN from user browsers. Do not cut those two until `https://glitchtip.coilysiren.me` is live, or until there is a deliberate alternate public ingest path. **Decisions found** * The stale phase-3 producer list in `docs/glitchtip-deploy-plan-design.md` should be treated as superseded by this issue. The current live list is `backend`, fused `eco-app`, `galaxy-gen`, `website`, and kai-server `thermal-heartbeat`. * `backend` and `eco-app` already consume `SENTRY_DSN` through ExternalSecrets backed by `/sentry-dsn/backend` and `/sentry-dsn/eco-mcp-app`. No app repo PR is needed for those. * `thermal-heartbeat` has two Sentry-facing paths: `/sentry-dsn/kai-server` for breach events, and `/kai-server/thermal-heartbeat-cron-url` for Sentry cron check-ins. Hosted Sentry cannot be fully retired until the cron monitor path has a GlitchTip uptime heartbeat replacement or another explicit owner. * GlitchTip phase 1 is genuinely complete: the docs say the smoke org/project was deleted afterward, so the current 0 orgs/projects state is expected, not a failed deploy. **Options** * **Option A: wait for public GlitchTip, then cut everything.** Simplest mental model and required for browser producers. Risk: in-cluster producers may fail if `glitchtip.coilysiren.me` hairpins through the home WAN path. Verify from an app pod before using one public DSN host everywhere. * **Option B: staged endpoint strategy.** Use a reachable internal or tailnet host for server-side producers, use the public host for browser producers, and keep the project ids/public keys the same. This is more operationally fiddly, but avoids blocking backend and eco-app on public ingress. * **Option C: cut only server-side producers now, defer browser producers and teardown.** Fastest useful progress, but hosted Sentry remains live until `website`, `galaxy-gen`, and the thermal cron monitor are solved. Recommendation: use Option C first, then finish with Option A or B after public reachability is verified. Start with `backend` because it has a known forced-exception smoke path (`GET /explode` in the repo digest), then `eco-app`, then kai-server breach events. Leave `website` and `galaxy-gen` until the public GlitchTip hostname is live. **Suggested cutover runbook per producer** 1. Create the GlitchTip project and copy its DSN. 2. Decide the DSN hostname based on where the producer runs: pod-internal or tailnet for server-side producers, public `glitchtip.coilysiren.me` for browser producers. 3. Overwrite the matching `/sentry-dsn/*` SecureString only when ready to cut that one producer. 4. Force the ExternalSecret refresh instead of waiting for the 1h interval, then restart the pod or host timer. 5. Fire a producer-specific test error and confirm it lands in the right GlitchTip project. 6. Confirm the old hosted Sentry project stops receiving new events before moving to the next producer. **Risks and open questions** * DSN hostname reachability is the biggest technical risk. A DSN minted while `GLITCHTIP_DOMAIN` points at the tailnet URL is not suitable for browser-side SDKs. A public DSN may not be suitable from pods unless hairpin or internal routing is verified. * Do not decommission hosted Sentry while `/kai-server/thermal-heartbeat-cron-url` still points at Sentry. GlitchTip supports heartbeat-style uptime monitors, but that is not the same API as Sentry cron check-ins, so the monitor URL needs its own migration decision. * Backend and eco-app docs describe Sentry as an error/log sink in places. The phase-3 acceptance test should verify the exact signal Kai cares about, not just that one exception appears. * Keep hosted Sentry org/projects until every live producer has a confirmed GlitchTip test event and at least one quiet period in Sentry. The SSM parameter version history gives some rollback help, but deleting hosted projects removes the cleanest rollback target. * After all live producers are cut, delete only the confirmed stale params first: `/sentry-dsn/eco-spec-tracker`, `/sentry-dsn/eco-telemetry`, and `/sentry/projects`. Audit `/sentry/personal-auth-token` separately during final hosted-Sentry teardown rather than silently deleting it in this pass. **Documentation cleanup** Once the cutover is proven and hosted Sentry is actually gone, update: * `docs/glitchtip-deploy-plan.md` phase 3 and the stale list in `docs/glitchtip-deploy-plan-design.md`. * deploy repo `docs/o11y-sources.md` and `docs/o11y.md`, which still describe Sentry as live for the old producer set. * the thermal-heartbeat docs if its cron monitor moves from Sentry to GlitchTip uptime heartbeat or another alerting path. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it. <!-- ward-agent-reply --> <!-- ward-agent-signature --> — Codex, via `ward agent`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#395
No description provided.