Stand up a self-hosted health-check service (Gatus), tailnet-reachable #232

New issue

Closed

opened 2026-06-05 22:08:08 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-05 22:08:08 +00:00

Owner

What's missing

There is no continuous health-check / uptime monitor for the production floor today. When forgejo, gauntlet, eco-jobs-tracker, or one of the eco-side services goes down, the first signal is Kai noticing manually or a downstream agent erroring out. No paging, no historical uptime record, no SLO surface.

The k3s cluster on kai-server has liveness/readiness probes inside each workload, but those are local to the pod. They do not answer "is the public endpoint actually reachable from outside the cluster?" and they do not answer "is the tailnet-only endpoint reachable from a probe living inside the tailnet?".

Why now

Forgejo is the canonical commit host (issue #220, #215 show runner-side flakiness has already burned hours that a passive monitor would have caught earlier).
The o2r relay and the agent-channel CLI now route real work between machines. A silent o2r outage breaks dispatch end-to-end.
The eco surface (eco-jobs-tracker, eco-mcp-app, eco-telemetry) is starting to have real users.
Adding more services without a uniform "is it up?" surface scales linearly into pain.

Desired shape

A self-hosted health-check service running inside the k3s cluster, deployed via the existing Ansible + k3s path (same shape as the other services in infrastructure/). Config-as-code over a UI-driven tool, so check definitions live in this repo and follow the same review path as everything else.

Probes need to cover:

Public HTTP(S) - forgejo.coilysiren.me, eco-jobs-tracker, galaxy-gen, website, any other *.coilysiren.me surface.
Internal k3s services - probed from inside the cluster via Service DNS, catching the case where the pod is healthy but the ingress is broken.
Tailnet-only endpoints (the stretch goal) - anything bound to 100.x.x.x or *.ts.net that should never be public. Because the probe runs on kai-server which is already a tailnet node, this is reachable for free, no sidecar gymnastics required.

Alerting routes through whatever notification surface fits cleanly with the existing stack (Discord webhook is the lowest-friction first hop given coily ops discord is already wired; Slack / email can come later).

Candidate stacks

Ranked by fit with the IaC / config-as-code posture:

Gatus - declarative YAML, single binary, native Discord/Slack/PagerDuty alerting, ships a Helm chart. Best fit for "checks live in this repo" discipline. Recommended starting point.
Uptime Kuma - more featureful UI, but config is database-backed, which fights the GitOps posture. Good fallback if Gatus turns out to be too thin.
Healthchecks.io (self-hosted) - push-based dead-man's-switch model. Useful as a complement to Gatus for cron-style jobs (ansible-freshen, tap-bump, daily memory sweeps), not as the primary uptime monitor.

The recommendation is Gatus as the primary, with Healthchecks.io added later when there is a real cron job that needs dead-man coverage.

Tailscale reach: the easy path

Because kai-server is already a tailnet node, a Pod scheduled on it inherits no tailnet reach by default (k3s networks the pod, not the host). Two clean options:

Run the Gatus pod with hostNetwork: true so it shares kai-server's network namespace and sees the tailnet directly. Simplest, no sidecar.
Tailscale operator + ProxyGroup (the supported path), letting Gatus stay in the pod network and have the operator inject tailnet reach. More moving parts, but reusable for future workloads that need tailnet egress.

Option 1 is the right shape for a single-node cluster running one health-check pod. Revisit option 2 only if a second workload ever needs the same trick.

Open questions

Where alerts land first (Discord webhook channel name? new channel?).
Whether the dashboard should be exposed publicly behind auth, or kept tailnet-only.
Whether to fold this in under the existing Ansible role pattern or carve out a new role.

Done definition

Gatus deployed via Ansible + k3s, config checked into infrastructure/.
At least the public surfaces from production_floor.md covered, plus forgejo and o2r.
At least one tailnet-only probe proving the stretch goal works.
Alerts firing to a real notification channel on synthetic failure.

Filed by Codex during a routing handoff from Kai's Kapwing workspace.

## What's missing There is no continuous health-check / uptime monitor for the production floor today. When forgejo, gauntlet, eco-jobs-tracker, or one of the eco-side services goes down, the first signal is Kai noticing manually or a downstream agent erroring out. No paging, no historical uptime record, no SLO surface. The k3s cluster on kai-server has liveness/readiness probes inside each workload, but those are local to the pod. They do not answer **"is the public endpoint actually reachable from outside the cluster?"** and they do not answer **"is the tailnet-only endpoint reachable from a probe living inside the tailnet?"**. ## Why now - Forgejo is the canonical commit host (issue #220, #215 show runner-side flakiness has already burned hours that a passive monitor would have caught earlier). - The o2r relay and the agent-channel CLI now route real work between machines. A silent o2r outage breaks dispatch end-to-end. - The eco surface (eco-jobs-tracker, eco-mcp-app, eco-telemetry) is starting to have real users. - Adding more services without a uniform "is it up?" surface scales linearly into pain. ## Desired shape A self-hosted health-check service running inside the k3s cluster, deployed via the existing Ansible + k3s path (same shape as the other services in `infrastructure/`). Config-as-code over a UI-driven tool, so check definitions live in this repo and follow the same review path as everything else. Probes need to cover: - **Public HTTP(S)** - forgejo.coilysiren.me, eco-jobs-tracker, galaxy-gen, website, any other `*.coilysiren.me` surface. - **Internal k3s services** - probed from inside the cluster via Service DNS, catching the case where the pod is healthy but the ingress is broken. - **Tailnet-only endpoints** (the stretch goal) - anything bound to `100.x.x.x` or `*.ts.net` that should never be public. Because the probe runs on kai-server which is already a tailnet node, this is reachable for free, no sidecar gymnastics required. Alerting routes through whatever notification surface fits cleanly with the existing stack (Discord webhook is the lowest-friction first hop given `coily ops discord` is already wired; Slack / email can come later). ## Candidate stacks Ranked by fit with the IaC / config-as-code posture: - **Gatus** - declarative YAML, single binary, native Discord/Slack/PagerDuty alerting, ships a Helm chart. Best fit for "checks live in this repo" discipline. Recommended starting point. - **Uptime Kuma** - more featureful UI, but config is database-backed, which fights the GitOps posture. Good fallback if Gatus turns out to be too thin. - **Healthchecks.io (self-hosted)** - push-based dead-man's-switch model. Useful as a *complement* to Gatus for cron-style jobs (ansible-freshen, tap-bump, daily memory sweeps), not as the primary uptime monitor. The recommendation is Gatus as the primary, with Healthchecks.io added later when there is a real cron job that needs dead-man coverage. ## Tailscale reach: the easy path Because kai-server is already a tailnet node, a Pod scheduled on it inherits no tailnet reach by default (k3s networks the pod, not the host). Two clean options: 1. **Run the Gatus pod with `hostNetwork: true`** so it shares kai-server's network namespace and sees the tailnet directly. Simplest, no sidecar. 2. **Tailscale operator + ProxyGroup** (the supported path), letting Gatus stay in the pod network and have the operator inject tailnet reach. More moving parts, but reusable for future workloads that need tailnet egress. Option 1 is the right shape for a single-node cluster running one health-check pod. Revisit option 2 only if a second workload ever needs the same trick. ## Open questions - Where alerts land first (Discord webhook channel name? new channel?). - Whether the dashboard should be exposed publicly behind auth, or kept tailnet-only. - Whether to fold this in under the existing Ansible role pattern or carve out a new role. ## Done definition - Gatus deployed via Ansible + k3s, config checked into `infrastructure/`. - At least the public surfaces from `production_floor.md` covered, plus forgejo and o2r. - At least one tailnet-only probe proving the stretch goal works. - Alerts firing to a real notification channel on synthetic failure. Filed by Codex during a routing handoff from Kai's Kapwing workspace.

coilysiren added the

label

2026-06-17 07:05:19 +00:00

coilysiren commented

2026-06-17 08:22:37 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:22:37 +00:00