coilyco-flight-deck/infrastructure

Fork 0

Ser8 ops stack readiness and operationalization campaign #518

New issue

Open

opened 2026-07-10 02:04:18 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-10 02:04:18 +00:00

Member

Goal

Make the Ser8 ops plane operational enough that Kai can rely on it for daily operations without remembering which pane is live, which URL is stale, or which part still needs manual babysitting.

This is a long-horizon coordination issue. The director should build and maintain a readiness matrix, fix every reversible repo-owned gap, and file child issues for anything that needs a human UI step, destructive host action, or a separate repo owner.

Current picture from the 2026-07-10 scout

Ser8 is the canonical ops plane. The intended pane split is:

Metrics - VictoriaMetrics and vmui on Ser8.
Traces and logs - SigNoz on Ser8.
Errors - GlitchTip as the self-hosted Sentry-compatible sink.
Alerting - vmalert plus Alertmanager, moving toward Telegram and away from ntfy.
Reachability - Gatus and the Ser8 watchtower gauges.
Backups - restic from kai-server to Ser8, with restore-drill follow-through still operationally important.
MCP and operator access - Signoz MCP and related read-only surfaces need live probes, not just deployed manifests.
Human-facing private URLs - Ser8 and tailnet-only UIs need friendly, documented, private names where they are daily tools.

The main open anchors found during scouting:

coilyco-flight-deck/infrastructure#385 - continuous profiling pane, Pyroscope standalone on Ser8.
coilyco-flight-deck/infrastructure#395 - hosted Sentry to GlitchTip DSN cutover, gated on GlitchTip org/project setup.
coilyco-flight-deck/infrastructure#369 - ntfy teardown and Telegram alert consolidation.
coilyco-bridge/deploy#125 - Signoz MCP live NodePort repair after failed rollout.
coilyco-bridge/deploy#110 - friendly private aliases for tailnet-only UIs, including Ser8 ops surfaces.
coilyco-bridge/deploy#114 - product-level resource visibility for legacy observability namespace contents.
coilyco-flight-deck/infrastructure#403 - deploy safety harness, relevant if Ser8 work keeps requiring babysat applies.

Closed or absorbed work should not be re-opened just because it is part of the story. Use closed issues as evidence, then verify live state.

Director work

Read the owning docs first: deploy/docs/o11y.md, deploy/docs/o11y-sources.md, deploy/services/ser8-observability/docs/ser8-o11y-contract.md, and the related issue threads above.
Build a readiness matrix with one row per pane: metrics, traces, logs, errors, alerting, reachability, backups, MCP access, private UI aliases, resource visibility, and deploy safety.
For each row, record current state as green, yellow, or red, with the exact evidence source used.
Fix repo-owned gaps directly in the owning repo when the fix is reversible and durable: manifests, Helm values, dashboards, pipelines, docs, runbooks, workflows, probes, or guarded scripts.
File child issues for any blocker that cannot be carried in the director run. Examples: GlitchTip UI org/project creation, destructive host or disk mutation, missing credentials, or cross-repo work outside the current grant.
Prefer small landed slices over one giant branch. Each child issue should have a concrete done-condition and the owning repo named.

Mutation boundary

Use live inspection to prove state, but make durable changes through the owning repo. Do not hot-edit cluster state when the correct fix is a tracked manifest, values file, parser, workflow, runbook, or guarded rollout step.

Any live mutation must be through documented guarded verbs or an explicitly documented operator path. Do not copy secret values, opaque tokens, passwords, or private host identifiers into issue comments.

Done-condition

This issue is done when the Ser8 readiness matrix is complete and every row is one of:

Green - verified live with a current probe or query, and docs point to the right surface.
Yellow - usable with a linked child issue for the remaining improvement.
Red - blocked by a linked child issue that names the human action or external dependency.

Do not close this issue after the first fixed component. Close it only when there is no untriaged Ser8 ops gap left in the matrix.

## Goal Make the Ser8 ops plane operational enough that Kai can rely on it for daily operations without remembering which pane is live, which URL is stale, or which part still needs manual babysitting. This is a long-horizon coordination issue. The director should build and maintain a readiness matrix, fix every reversible repo-owned gap, and file child issues for anything that needs a human UI step, destructive host action, or a separate repo owner. ## Current picture from the 2026-07-10 scout Ser8 is the canonical ops plane. The intended pane split is: * Metrics - VictoriaMetrics and vmui on Ser8. * Traces and logs - SigNoz on Ser8. * Errors - GlitchTip as the self-hosted Sentry-compatible sink. * Alerting - vmalert plus Alertmanager, moving toward Telegram and away from ntfy. * Reachability - Gatus and the Ser8 watchtower gauges. * Backups - restic from kai-server to Ser8, with restore-drill follow-through still operationally important. * MCP and operator access - Signoz MCP and related read-only surfaces need live probes, not just deployed manifests. * Human-facing private URLs - Ser8 and tailnet-only UIs need friendly, documented, private names where they are daily tools. The main open anchors found during scouting: * `coilyco-flight-deck/infrastructure#385` - continuous profiling pane, Pyroscope standalone on Ser8. * `coilyco-flight-deck/infrastructure#395` - hosted Sentry to GlitchTip DSN cutover, gated on GlitchTip org/project setup. * `coilyco-flight-deck/infrastructure#369` - ntfy teardown and Telegram alert consolidation. * `coilyco-bridge/deploy#125` - Signoz MCP live NodePort repair after failed rollout. * `coilyco-bridge/deploy#110` - friendly private aliases for tailnet-only UIs, including Ser8 ops surfaces. * `coilyco-bridge/deploy#114` - product-level resource visibility for legacy observability namespace contents. * `coilyco-flight-deck/infrastructure#403` - deploy safety harness, relevant if Ser8 work keeps requiring babysat applies. Closed or absorbed work should not be re-opened just because it is part of the story. Use closed issues as evidence, then verify live state. ## Director work * Read the owning docs first: `deploy/docs/o11y.md`, `deploy/docs/o11y-sources.md`, `deploy/services/ser8-observability/docs/ser8-o11y-contract.md`, and the related issue threads above. * Build a readiness matrix with one row per pane: metrics, traces, logs, errors, alerting, reachability, backups, MCP access, private UI aliases, resource visibility, and deploy safety. * For each row, record current state as green, yellow, or red, with the exact evidence source used. * Fix repo-owned gaps directly in the owning repo when the fix is reversible and durable: manifests, Helm values, dashboards, pipelines, docs, runbooks, workflows, probes, or guarded scripts. * File child issues for any blocker that cannot be carried in the director run. Examples: GlitchTip UI org/project creation, destructive host or disk mutation, missing credentials, or cross-repo work outside the current grant. * Prefer small landed slices over one giant branch. Each child issue should have a concrete done-condition and the owning repo named. ## Mutation boundary Use live inspection to prove state, but make durable changes through the owning repo. Do not hot-edit cluster state when the correct fix is a tracked manifest, values file, parser, workflow, runbook, or guarded rollout step. Any live mutation must be through documented guarded verbs or an explicitly documented operator path. Do not copy secret values, opaque tokens, passwords, or private host identifiers into issue comments. ## Done-condition This issue is done when the Ser8 readiness matrix is complete and every row is one of: * Green - verified live with a current probe or query, and docs point to the right surface. * Yellow - usable with a linked child issue for the remaining improvement. * Red - blocked by a linked child issue that names the human action or external dependency. Do not close this issue after the first fixed component. Close it only when there is no untriaged Ser8 ops gap left in the matrix.

coilyco-ops added the

interactive

labels

2026-07-10 02:04:18 +00:00

coilyco-ops added

consult

and removed

interactive

labels

2026-07-10 08:59:49 +00:00

coilyco-ops commented

2026-07-10 12:17:42 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Advisory Recommendation
Keep infrastructure#518 as the parent readiness matrix and do not close it until each row has current evidence plus either a green state or a linked blocker. The campaign should avoid reopening closed work just because it is part of the story. Use closed issues as evidence, then verify live state.

Best Path

Parent issue owns the matrix, state colors, and evidence links.
Repo-owned durable fixes should be small patches in the owning repo, mostly docs, manifests, Helm values, probes, and runbooks.
Existing child issues already cover most blockers: infrastructure#385, infrastructure#395, infrastructure#369, infrastructure#403, deploy#125, deploy#110, and deploy#114.
New child work is only needed if live backup verification confirms the restic target is down or if the matrix finds an unowned Ser8 gap.

Readiness Matrix Draft

Metrics - green - VictoriaMetrics vmui is documented at ser8:30428. A live VM query against the Ser8 tailnet FQDN returned HTTP 200 and up query success with one result.
Traces - yellow - SigNoz UI is live at Ser8 and returned HTTP 200. The remaining gap is proving live trace ingest with a query or MCP probe.
Logs - yellow - SigNoz UI is live and the parser repo exists, but log-producers.md says only Ser8 cluster pods are collected today. Kai-server pods and journald are still producer-cutover work.
Errors - red - GlitchTip login returned HTTP 200, but infrastructure#395 says there are still zero orgs/projects and hosted Sentry remains the live sink. Human UI setup and DSN cutover block operational readiness.
Alerting - yellow - vmalert/Alertmanager are configured and ALERTS query returned zero active alerts. Telegram is present in values, but deploy still carries the ntfy bridge and docs still describe dual-shipping. Finish infrastructure#369 before calling this green.
Reachability - yellow - Gatus is live on Ser8. Its API returned six endpoint statuses, all successful, and the watchtower metric exists in VictoriaMetrics. Docs in deploy/docs/o11y*.md still say Gatus is planned or not live, so this is not green until docs match reality.
Backups - red until rechecked - infrastructure#320 closed with restic wiring landed, but its own closing comment says the first restore drill remains the rough edge. A tailnet HTTP probe to the rest-server port returned connection refused from this surface. The director should verify systemd state and last snapshots from the hosts, then file a new infra child issue if the target is truly down.
MCP access - red - deploy#125 is open and repeatedly failed post-close verification. The SigNoz MCP NodePort still refuses MCP initialize, while sibling MCP probes passed.
Private UI aliases - yellow - deploy#110 has a good advisory decision record for names and rollout order. Implementation is still pending, especially for Ser8 NodePort UIs.
Resource visibility - yellow - deploy#114 has the right recommendation: do not split the legacy observability namespace yet, use an interim product rollup query, then add durable labels during deploy#51 migration.
Deploy safety - yellow - infrastructure#403 correctly scopes the harness. It is not required to finish every pane, but it is the highest-leverage way to stop Ser8 work from needing babysat applies.

Decisions To Record On The Issue

Ser8 is the canonical ops plane. Kai-server observability is legacy except for GlitchTip until it moves.
Pane discipline should stay strict: VictoriaMetrics for metrics, SigNoz for traces/logs, GlitchTip for errors, Gatus/watchtower for reachability.
Pyroscope remains the profiling choice from infrastructure#385; Coroot is only a reconsideration option, not a reason to restart the decision.
Friendly names should follow deploy#110: metrics, signoz, glitchtip, and reachability as private-first surfaces.
Backup readiness requires restore evidence, not only installed timers.

Open Questions

Can Ser8 and kai-server systemd prove restic backup, rest-server, and prune timers are active, and can restic snapshots show a recent successful backup?
Has a restore drill ever been run after infrastructure#320 landed?
Which live producers already send traces to Ser8 SigNoz, as distinct from merely having the UI up?
Should Gatus docs be patched immediately in deploy, or should that wait for a broader reachability docs pass?
Does Alertmanager Telegram have a successful delivery test after removing the ntfy bridge?

Risks

Calling UI HTTP 200 green too early would hide missing ingest, missing projects, or broken MCP access.
Backups are the highest operational risk because a closed issue plus untested restore can create false confidence.
Ser8 docs are already stale around Gatus and ntfy, so Kai may keep opening stale URLs unless docs are corrected.
MCP and private alias work lives in coilyco-bridge/deploy, so the parent issue must track cross-repo blockers without duplicating them.

Recommendation
Post the matrix above to infrastructure#518, then have the director work row by row. Start with the reversible documentation mismatches and live probes: update Gatus/o11y docs, verify restic host state, and keep deploy#125 moving because MCP access is the clearest red cross-repo blocker. Do not close #518 after the first green pane. Close it only when every row is green or has a linked blocker with a concrete done-condition.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Advisory Recommendation** Keep `infrastructure#518` as the parent readiness matrix and do not close it until each row has current evidence plus either a green state or a linked blocker. The campaign should avoid reopening closed work just because it is part of the story. Use closed issues as evidence, then verify live state. **Best Path** * Parent issue owns the matrix, state colors, and evidence links. * Repo-owned durable fixes should be small patches in the owning repo, mostly docs, manifests, Helm values, probes, and runbooks. * Existing child issues already cover most blockers: `infrastructure#385`, `infrastructure#395`, `infrastructure#369`, `infrastructure#403`, `deploy#125`, `deploy#110`, and `deploy#114`. * New child work is only needed if live backup verification confirms the restic target is down or if the matrix finds an unowned Ser8 gap. **Readiness Matrix Draft** * **Metrics - green** - VictoriaMetrics vmui is documented at `ser8:30428`. A live VM query against the Ser8 tailnet FQDN returned HTTP 200 and `up` query success with one result. * **Traces - yellow** - SigNoz UI is live at Ser8 and returned HTTP 200. The remaining gap is proving live trace ingest with a query or MCP probe. * **Logs - yellow** - SigNoz UI is live and the parser repo exists, but `log-producers.md` says only Ser8 cluster pods are collected today. Kai-server pods and journald are still producer-cutover work. * **Errors - red** - GlitchTip login returned HTTP 200, but `infrastructure#395` says there are still zero orgs/projects and hosted Sentry remains the live sink. Human UI setup and DSN cutover block operational readiness. * **Alerting - yellow** - vmalert/Alertmanager are configured and `ALERTS` query returned zero active alerts. Telegram is present in values, but `deploy` still carries the ntfy bridge and docs still describe dual-shipping. Finish `infrastructure#369` before calling this green. * **Reachability - yellow** - Gatus is live on Ser8. Its API returned six endpoint statuses, all successful, and the watchtower metric exists in VictoriaMetrics. Docs in `deploy/docs/o11y*.md` still say Gatus is planned or not live, so this is not green until docs match reality. * **Backups - red until rechecked** - `infrastructure#320` closed with restic wiring landed, but its own closing comment says the first restore drill remains the rough edge. A tailnet HTTP probe to the rest-server port returned connection refused from this surface. The director should verify systemd state and last snapshots from the hosts, then file a new infra child issue if the target is truly down. * **MCP access - red** - `deploy#125` is open and repeatedly failed post-close verification. The SigNoz MCP NodePort still refuses MCP initialize, while sibling MCP probes passed. * **Private UI aliases - yellow** - `deploy#110` has a good advisory decision record for names and rollout order. Implementation is still pending, especially for Ser8 NodePort UIs. * **Resource visibility - yellow** - `deploy#114` has the right recommendation: do not split the legacy `observability` namespace yet, use an interim product rollup query, then add durable labels during `deploy#51` migration. * **Deploy safety - yellow** - `infrastructure#403` correctly scopes the harness. It is not required to finish every pane, but it is the highest-leverage way to stop Ser8 work from needing babysat applies. **Decisions To Record On The Issue** * Ser8 is the canonical ops plane. Kai-server `observability` is legacy except for GlitchTip until it moves. * Pane discipline should stay strict: VictoriaMetrics for metrics, SigNoz for traces/logs, GlitchTip for errors, Gatus/watchtower for reachability. * Pyroscope remains the profiling choice from `infrastructure#385`; Coroot is only a reconsideration option, not a reason to restart the decision. * Friendly names should follow `deploy#110`: `metrics`, `signoz`, `glitchtip`, and `reachability` as private-first surfaces. * Backup readiness requires restore evidence, not only installed timers. **Open Questions** * Can Ser8 and kai-server systemd prove restic backup, rest-server, and prune timers are active, and can `restic snapshots` show a recent successful backup? * Has a restore drill ever been run after `infrastructure#320` landed? * Which live producers already send traces to Ser8 SigNoz, as distinct from merely having the UI up? * Should Gatus docs be patched immediately in `deploy`, or should that wait for a broader reachability docs pass? * Does Alertmanager Telegram have a successful delivery test after removing the ntfy bridge? **Risks** * Calling UI HTTP 200 green too early would hide missing ingest, missing projects, or broken MCP access. * Backups are the highest operational risk because a closed issue plus untested restore can create false confidence. * Ser8 docs are already stale around Gatus and ntfy, so Kai may keep opening stale URLs unless docs are corrected. * MCP and private alias work lives in `coilyco-bridge/deploy`, so the parent issue must track cross-repo blockers without duplicating them. **Recommendation** Post the matrix above to `infrastructure#518`, then have the director work row by row. Start with the reversible documentation mismatches and live probes: update Gatus/o11y docs, verify restic host state, and keep `deploy#125` moving because MCP access is the clearest red cross-repo blocker. Do not close `#518` after the first green pane. Close it only when every row is green or has a linked blocker with a concrete done-condition. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-10 18:12:52 +00:00

[epic] Eliminate the recurring post-bounce fleet full-stop (self-recovery + drift gate + blast-radius) #541