coilyco-flight-deck/infrastructure

Fork 0

Stand up a continuous-profiling pane (Pyroscope standalone) on ser8 #385

New issue

Open

opened 2026-06-19 17:27:24 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-06-19 17:27:24 +00:00

Owner

The o11y stack has no perf-profiling pane today. The 2026-06 platform survey is done and a pick is made; this tracks the deploy. Usage/decision is captured in the o11y-profiler skill (agentic-os-kai) and the gap is listed in coilyco-bridge/deploy/docs/o11y.md.

Pick

Grafana Pyroscope v2.x, standalone. Wins on the deciding axis (flame-graph + diff-view UX: the queryless Profiles Drilldown, diff normalized by percent-of-total). Key fact: standalone Pyroscope serves its own UI on :4040, no Grafana required, so the no-Grafana-on-ser8 cost is zero. Apache-2.0, healthy cadence (v2.1.0, 2026).
Instrumentation: Python via the pyroscope-io SDK (not eBPF - CPython eBPF unwinding is still fragile in 2026), Go via eBPF/pprof, Node via SDK. Python-heavy stack is why the SDK path matters.
Runner-up: Parca (pure eBPF, self-contained FrostDB, no object store) if zero-instrumentation ever beats diff UX.

First step

helm repo add grafana https://grafana.github.io/helm-charts, install grafana/pyroscope chart 2.x in monolithic mode (replicaCount=1) on ser8.
Storage to a small in-cluster MinIO (matches v2's object-store-native design) or a local-path PVC. Retention 7-14 days (profiles are dense).
Tailnet-only :4040 (ts-proxy pattern, same as signoz/glitchtip). No public exposure.
Instrument one Python service first, confirm the diff view is legible, then roll wider.

Parked (re-check, could change the pick)

OTel profiling signal - Alpha as of 2026-03, GA targeted ~Q3 2026 (not committed). If a good self-hosted OSS backend ingests OTLP profiles post-GA, the calculus shifts toward an OTel-native pipeline. Re-check at GA.
SigNoz-native profiling - would be the dream fit (already deployed, single pane with traces/logs) but maintainer confirms off the near-term roadmap (Dec 2025, SigNoz#5641). Do not block on it; recheck the changelog in ~6 months.

Sits alongside the other ser8 o11y build-outs: Gatus #232, GlitchTip-to-ser8 #320.

The o11y stack has no perf-profiling pane today. The 2026-06 platform survey is done and a pick is made; this tracks the deploy. Usage/decision is captured in the `o11y-profiler` skill (agentic-os-kai) and the gap is listed in `coilyco-bridge/deploy/docs/o11y.md`. ## Pick - **Grafana Pyroscope v2.x, standalone.** Wins on the deciding axis (flame-graph + diff-view UX: the queryless Profiles Drilldown, diff normalized by percent-of-total). Key fact: standalone Pyroscope serves its own UI on `:4040`, **no Grafana required**, so the no-Grafana-on-ser8 cost is zero. Apache-2.0, healthy cadence (v2.1.0, 2026). - **Instrumentation:** Python via the `pyroscope-io` SDK (not eBPF - CPython eBPF unwinding is still fragile in 2026), Go via eBPF/pprof, Node via SDK. Python-heavy stack is why the SDK path matters. - **Runner-up: Parca** (pure eBPF, self-contained FrostDB, no object store) if zero-instrumentation ever beats diff UX. ## First step 1. `helm repo add grafana https://grafana.github.io/helm-charts`, install `grafana/pyroscope` chart 2.x in **monolithic** mode (`replicaCount=1`) on ser8. 2. Storage to a small in-cluster MinIO (matches v2's object-store-native design) or a local-path PVC. Retention 7-14 days (profiles are dense). 3. Tailnet-only `:4040` (ts-proxy pattern, same as signoz/glitchtip). No public exposure. 4. Instrument one Python service first, confirm the diff view is legible, then roll wider. ## Parked (re-check, could change the pick) - **OTel profiling signal** - Alpha as of 2026-03, GA targeted ~Q3 2026 (not committed). If a good self-hosted OSS backend ingests OTLP profiles post-GA, the calculus shifts toward an OTel-native pipeline. Re-check at GA. - **SigNoz-native profiling** - would be the dream fit (already deployed, single pane with traces/logs) but maintainer confirms off the near-term roadmap (Dec 2025, SigNoz#5641). Do not block on it; recheck the changelog in ~6 months. Sits alongside the other ser8 o11y build-outs: Gatus #232, GlitchTip-to-ser8 #320.

coilysiren commented

2026-06-23 05:37:48 +00:00

Author

Owner

or perhaps: https://coroot.com/

coilyco-ops added the

interactive

labels

2026-06-24 05:38:20 +00:00

coilyco-ops commented

2026-06-24 05:38:21 +00:00

Member

Triage: P3 / interactive - Pyroscope standalone; needs a live helm install on ser8 + service instrumentation.

(goose-style pass; claude as the judgment engine, applied 2026-06-23)

**Triage:** `P3` / `interactive` - Pyroscope standalone; needs a live helm install on ser8 + service instrumentation. (goose-style pass; claude as the judgment engine, applied 2026-06-23)

coilyco-ops added

consult

and removed

interactive

labels

2026-07-10 09:00:04 +00:00

coilyco-ops commented

2026-07-10 13:24:47 +00:00

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Advisory recommendation: keep the issue pointed at Grafana Pyroscope v2.x standalone. The Coroot comment is worth recording as an alternative, but it does not change the pick for this narrow pane: Pyroscope gives the purpose-built profiles UI and Python SDK path this stack needs, while Coroot would overlap with the existing VictoriaMetrics + SigNoz pane split and leans more toward whole-platform eBPF observability.

Key decisions:

Deploy now, do not wait for OTel Profiles. OpenTelemetry Profiles entered public Alpha on 2026-03-26, so it is real but still not the stable backend contract to build this pane around yet. Re-check after GA.
Use Pyroscope single-binary / monolithic mode on ser8. Grafana's Helm docs list the single-instance install as the right path when only one Pyroscope instance is needed, and the in-cluster query URL is on :4040.
Treat “no Grafana required” as true, with one nuance. Grafana docs distinguish standalone Pyroscope OSS UI from Grafana Profiles Drilldown. The standalone UI supports viewing, comparing, and diffing profiles, so ser8 does not need Grafana, but the engineer should validate the standalone comparison/diff UX rather than assuming the Grafana plugin UI is present.
Start with local-path PVC, not MinIO. The chart supports filesystem storage by default and ser8 is a single-node k3s ops plane. A local-path PVC is less moving infrastructure for the first pane. MinIO only earns its keep if Pyroscope later needs object-store semantics that survive beyond one node or line up with a broader backup/restore story.
Set retention explicitly. Pyroscope defaults are easy to misread: query lookback defaults to 1w, while compactor block retention defaults to disabled. Pick 7d first, or 14d if weekly before/after deploy diffs matter, and set both query lookback and block retention to match.
Instrument one Python service first. Prefer a service that already has meaningful traffic and SigNoz traces, such as backend or eco-jobs-tracker, then expand. Use pyroscope-io SDK first. Keep eBPF/Alloy as a later path for Go or broad node coverage.

Implementation shape I would recommend:

Helm chart grafana/pyroscope pinned to the current 2.x chart, observed as 2.1.0 in the Grafana Helm index on 2026-07-10.
Namespace observability on the ser8 context.
pyroscope.replicaCount: 1, architecture.microservices.enabled: false.
pyroscope.persistence.enabled: true, storageClassName: local-path, initial PVC around 20Gi unless a render shows a better chart-specific convention.
Disable bundled Alloy for the first Python-SDK push path unless the engineer intentionally wants annotation scraping immediately. The 2.1.0 values enable Alloy by default, which is unnecessary moving surface for the first canary.
Tailnet-only UI via the existing standalone ts-serve pattern, pyroscope.<tailnet>.ts.net -> Pyroscope service :4040. Add a NodePort sibling only if a host-userspace producer needs localhost access, matching the existing ts-proxy plus NodePort for host access trap note.

Open questions before build:

Which repo owns the Pyroscope chart values for ser8 now: this infra repo, or coilyco-bridge/deploy beside services/ser8-observability? The docs currently describe ser8 o11y from the deploy repo, while this issue lives in infrastructure.
Which first Python app should be the canary? The best canary is the one with live traffic, an easy rollback, and existing trace/log correlation.
Does ser8 currently have the secret plumbing needed for a ts-pyroscope auth key, or should the rollout follow the ser8 pattern of resolving SSM on the control host and creating the Kubernetes Secret imperatively?
Is 7d enough retention, or does Kai want 14d for deployment-to-deployment comparison? Make this explicit because Pyroscope's defaults do not encode the issue's 7-14d intent.

Risks:

Chart defaults can produce an apparently working but non-durable install if persistence and retention are not set deliberately.
Pyroscope :4040 is both UI and ingest surface. Tailnet-only is probably enough for this homelab, but labels can leak service names, function names, repo paths, and accidental high-cardinality identifiers.
Python SDK overhead and fidelity need a canary. Keep sampling defaults conservative, make profiling easy to disable by env/config, and confirm the flame graph is legible before a wider rollout.
The issue body's “Profiles Drilldown” wording can lead an engineer toward Grafana-specific assumptions. For ser8, the acceptance test should be the standalone OSS Pyroscope UI.
Coroot is a valid future reconsideration if the priority flips to zero-instrumentation node-wide profiling, but that is a different product decision than “add a profiling pane beside existing SigNoz/VictoriaMetrics/GlitchTip.”

Suggested done condition: Pyroscope pod ready on ser8, UI reachable only over tailnet, retention/persistence rendered in tracked config, one Python service pushes profiles, a before/after or two-range comparison is readable in the standalone UI, and docs/o11y.md / o11y-sources.md / the profiler skill are updated to mark the pane live.

Sources checked: Pyroscope Helm docs, Pyroscope UI docs, Pyroscope profile type support, Pyroscope Python SDK docs, Pyroscope config reference, OpenTelemetry Profiles Alpha announcement, Coroot profiling docs, plus the local ser8 o11y docs under coilyco-bridge/deploy and this repo's k3s deploy notes.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Advisory recommendation:** keep the issue pointed at **Grafana Pyroscope v2.x standalone**. The Coroot comment is worth recording as an alternative, but it does not change the pick for this narrow pane: Pyroscope gives the purpose-built profiles UI and Python SDK path this stack needs, while Coroot would overlap with the existing VictoriaMetrics + SigNoz pane split and leans more toward whole-platform eBPF observability. **Key decisions:** * **Deploy now, do not wait for OTel Profiles.** OpenTelemetry Profiles entered public Alpha on 2026-03-26, so it is real but still not the stable backend contract to build this pane around yet. Re-check after GA. * **Use Pyroscope single-binary / monolithic mode on ser8.** Grafana's Helm docs list the single-instance install as the right path when only one Pyroscope instance is needed, and the in-cluster query URL is on `:4040`. * **Treat “no Grafana required” as true, with one nuance.** Grafana docs distinguish standalone Pyroscope OSS UI from Grafana Profiles Drilldown. The standalone UI supports viewing, comparing, and diffing profiles, so ser8 does not need Grafana, but the engineer should validate the standalone comparison/diff UX rather than assuming the Grafana plugin UI is present. * **Start with local-path PVC, not MinIO.** The chart supports filesystem storage by default and ser8 is a single-node k3s ops plane. A local-path PVC is less moving infrastructure for the first pane. MinIO only earns its keep if Pyroscope later needs object-store semantics that survive beyond one node or line up with a broader backup/restore story. * **Set retention explicitly.** Pyroscope defaults are easy to misread: query lookback defaults to `1w`, while compactor block retention defaults to disabled. Pick 7d first, or 14d if weekly before/after deploy diffs matter, and set both query lookback and block retention to match. * **Instrument one Python service first.** Prefer a service that already has meaningful traffic and SigNoz traces, such as `backend` or `eco-jobs-tracker`, then expand. Use `pyroscope-io` SDK first. Keep eBPF/Alloy as a later path for Go or broad node coverage. **Implementation shape I would recommend:** * Helm chart `grafana/pyroscope` pinned to the current 2.x chart, observed as `2.1.0` in the Grafana Helm index on 2026-07-10. * Namespace `observability` on the ser8 context. * `pyroscope.replicaCount: 1`, `architecture.microservices.enabled: false`. * `pyroscope.persistence.enabled: true`, `storageClassName: local-path`, initial PVC around 20Gi unless a render shows a better chart-specific convention. * Disable bundled Alloy for the first Python-SDK push path unless the engineer intentionally wants annotation scraping immediately. The 2.1.0 values enable Alloy by default, which is unnecessary moving surface for the first canary. * Tailnet-only UI via the existing standalone `ts-serve` pattern, `pyroscope.<tailnet>.ts.net` -> Pyroscope service `:4040`. Add a NodePort sibling only if a host-userspace producer needs localhost access, matching the existing `ts-proxy plus NodePort for host access` trap note. **Open questions before build:** * Which repo owns the Pyroscope chart values for ser8 now: this infra repo, or `coilyco-bridge/deploy` beside `services/ser8-observability`? The docs currently describe ser8 o11y from the deploy repo, while this issue lives in infrastructure. * Which first Python app should be the canary? The best canary is the one with live traffic, an easy rollback, and existing trace/log correlation. * Does ser8 currently have the secret plumbing needed for a `ts-pyroscope` auth key, or should the rollout follow the ser8 pattern of resolving SSM on the control host and creating the Kubernetes Secret imperatively? * Is 7d enough retention, or does Kai want 14d for deployment-to-deployment comparison? Make this explicit because Pyroscope's defaults do not encode the issue's 7-14d intent. **Risks:** * Chart defaults can produce an apparently working but non-durable install if persistence and retention are not set deliberately. * Pyroscope `:4040` is both UI and ingest surface. Tailnet-only is probably enough for this homelab, but labels can leak service names, function names, repo paths, and accidental high-cardinality identifiers. * Python SDK overhead and fidelity need a canary. Keep sampling defaults conservative, make profiling easy to disable by env/config, and confirm the flame graph is legible before a wider rollout. * The issue body's “Profiles Drilldown” wording can lead an engineer toward Grafana-specific assumptions. For ser8, the acceptance test should be the standalone OSS Pyroscope UI. * Coroot is a valid future reconsideration if the priority flips to zero-instrumentation node-wide profiling, but that is a different product decision than “add a profiling pane beside existing SigNoz/VictoriaMetrics/GlitchTip.” **Suggested done condition:** Pyroscope pod ready on ser8, UI reachable only over tailnet, retention/persistence rendered in tracked config, one Python service pushes profiles, a before/after or two-range comparison is readable in the standalone UI, and `docs/o11y.md` / `o11y-sources.md` / the profiler skill are updated to mark the pane live. **Sources checked:** [Pyroscope Helm docs](https://grafana.com/docs/pyroscope/latest/deploy-kubernetes/helm/), [Pyroscope UI docs](https://grafana.com/docs/pyroscope/latest/view-and-analyze-profile-data/pyroscope-ui/), [Pyroscope profile type support](https://grafana.com/docs/pyroscope/latest/configure-client/profile-types/), [Pyroscope Python SDK docs](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/python/), [Pyroscope config reference](https://grafana.com/docs/pyroscope/latest/configure-server/reference-configuration-parameters/), [OpenTelemetry Profiles Alpha announcement](https://opentelemetry.io/blog/2026/profiles-alpha/), [Coroot profiling docs](https://docs.coroot.com/profiling/), plus the local ser8 o11y docs under `coilyco-bridge/deploy` and this repo's k3s deploy notes. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`