coilyco-flight-deck/infrastructure

Fork 0

Full-mesh reachability healthcheck (N×N connectivity matrix) across the fleet #255

New issue

Open

opened 2026-06-07 03:25:08 +00:00 by coilysiren · 5 comments

coilysiren commented

2026-06-07 03:25:08 +00:00

Owner

Problem

The kais-macbook-pro -> kai-server outage (debug channel ZHQK, fleet fix #253) was only found because Kai happened to try kubectl and hit a Bad Gateway. The fault then took a multi-host, multi-agent debug session to localize. Nothing on the fleet was continuously asserting "node A can reach node B on the real data path." We discover reachability faults reactively, by tripping over them.

Worse, the fault was the kind a naive healthcheck misses: userspace Tailscale ping worked (~3ms) while the kernel TCP/ICMP path black-holed. A check that only does a tailscale ping or a DERP-routed probe would have stayed green through the entire outage.

Ask

Stand up a standing full-mesh reachability healthcheck across the fleet: every node probes every other node, N×(N-1) directed pairs, on a cadence, emitting a pass/fail per pair into observability (SigNoz / VictoriaMetrics already on kai-server).

On the term — there isn't one universal name. Candidates, roughly synonymous:

full-mesh reachability checks / mesh healthchecks (SRE-common, e.g. smokeping mesh, cloudprober)
all-pairs connectivity matrix / reachability matrix (the N×N output)
blackbox mesh probing (Prometheus blackbox_exporter framing)

Proposing "full-mesh reachability healthcheck" for the noun and "connectivity matrix" for the rendered N×N view. Open to renaming.

Design constraints (learned from ZHQK)

Probe the kernel data path, not userspace. Each pair must exercise real TCP connect (nc -z) and/or kernel ICMP and a real ssh exec, NOT just tailscale ping. The whole point is to catch the userspace-green / kernel-dead split that defined this outage.
Multiple probe types per pair — at minimum ICMP, TCP to a known port (22 / 6443), and an actual remote exec. A single probe type hides partial faults (recall: ping by name worked while ssh by socket did not).
Directed, not symmetric — A->B can fail while B->A is fine (the fault was mac-local egress). Probe both directions independently.
Render the matrix — a node × node grid (green/red per cell) so a glance localizes the fault to a row (one bad source), a column (one unreachable target), or a cell (one bad pair).
Alert on transitions, with a short flap window so a node reboot (expected k3s/apiserver startup gap, see the diagnose concept in ZHQK) doesn't page.

Building blocks already in hand

The ZHQK debug harness (coilyco-flight-deck/otel-a2a-relay-ansible-debug: scripts/step.py probe battery — ping / dig / nc-ssh / nc-api / ssh-exec — plus baseline/bless diffing) is essentially the single-pair version of this. The mesh check is that battery, fanned across all ordered host pairs, on a schedule, with results to metrics instead of an o2r channel. Rollout is an ansible role per the authoring-vs-rollout split (tool authored in its home repo, fan-out lives in ansible).

Relationship to #253

#253 fixes the specific dual-Tailscale-install root cause. This issue is the detection layer: had a mesh healthcheck been running, ZHQK would have opened with "kais-macbook-pro row is red on TCP/ssh but green on userspace ping" instead of an hour of localization. Complementary, not a duplicate.

Filed from the ZHQK debrief (channel closed RESOLVED 2026-06-07).

## Problem The kais-macbook-pro -> kai-server outage (debug channel ZHQK, fleet fix #253) was only found because Kai happened to try `kubectl` and hit a Bad Gateway. The fault then took a multi-host, multi-agent debug session to localize. Nothing on the fleet was *continuously* asserting "node A can reach node B on the real data path." We discover reachability faults reactively, by tripping over them. Worse, the fault was the kind a naive healthcheck misses: **userspace Tailscale ping worked (~3ms) while the kernel TCP/ICMP path black-holed.** A check that only does a `tailscale ping` or a DERP-routed probe would have stayed green through the entire outage. ## Ask Stand up a standing **full-mesh reachability healthcheck** across the fleet: every node probes every other node, N×(N-1) directed pairs, on a cadence, emitting a pass/fail per pair into observability (SigNoz / VictoriaMetrics already on kai-server). On the term — there isn't one universal name. Candidates, roughly synonymous: * **full-mesh reachability checks** / **mesh healthchecks** (SRE-common, e.g. smokeping mesh, cloudprober) * **all-pairs connectivity matrix** / **reachability matrix** (the N×N output) * **blackbox mesh probing** (Prometheus blackbox_exporter framing) Proposing **"full-mesh reachability healthcheck"** for the noun and **"connectivity matrix"** for the rendered N×N view. Open to renaming. ## Design constraints (learned from ZHQK) * **Probe the kernel data path, not userspace.** Each pair must exercise real TCP connect (`nc -z`) and/or kernel ICMP and a real `ssh exec`, NOT just `tailscale ping`. The whole point is to catch the userspace-green / kernel-dead split that defined this outage. * **Multiple probe types per pair** — at minimum ICMP, TCP to a known port (22 / 6443), and an actual remote exec. A single probe type hides partial faults (recall: ping by name worked while ssh by socket did not). * **Directed, not symmetric** — A->B can fail while B->A is fine (the fault was mac-local egress). Probe both directions independently. * **Render the matrix** — a node × node grid (green/red per cell) so a glance localizes the fault to a row (one bad source), a column (one unreachable target), or a cell (one bad pair). * **Alert on transitions**, with a short flap window so a node reboot (expected k3s/apiserver startup gap, see the diagnose concept in ZHQK) doesn't page. ## Building blocks already in hand The ZHQK debug harness (`coilyco-flight-deck/otel-a2a-relay-ansible-debug`: `scripts/step.py` probe battery — ping / dig / nc-ssh / nc-api / ssh-exec — plus baseline/bless diffing) is essentially the single-pair version of this. The mesh check is that battery, fanned across all ordered host pairs, on a schedule, with results to metrics instead of an o2r channel. Rollout is an ansible role per the authoring-vs-rollout split (tool authored in its home repo, fan-out lives in ansible). ## Relationship to #253 \#253 fixes the *specific* dual-Tailscale-install root cause. This issue is the *detection* layer: had a mesh healthcheck been running, ZHQK would have opened with "kais-macbook-pro row is red on TCP/ssh but green on userspace ping" instead of an hour of localization. Complementary, not a duplicate. Filed from the ZHQK debrief (channel closed RESOLVED 2026-06-07).

coilysiren commented

2026-06-07 03:46:41 +00:00

Author

Owner

Seed: per-target ssh user map

The ser8 capture exposed that ssh user is per-target, not fleet-uniform (otel-a2a-relay-ansible-debug#10). Recording the map as it stands so the mesh check ships with it rather than re-discovering host-by-host:

kai-server -> kai (confirmed - ssh-exec green)
ser8 -> coilysiren (confirmed - ssh-exec green)
kai-desktop-tower (windows native) -> firem (per Kai; UNVERIFIED - host offline last seen 17h)
kai-desktop-tower-wsl -> kai (per Kai, "I think"; UNVERIFIED - host offline)

Verify the two tower rows by probing once those hosts are back online. Usernames are meaningful names, not opaque, so the map can live in committed inventory (not SSM) - same lookup shape as the per-target address, just a second column. A mesh ssh-exec probe must key the user off the target, defaulting to nothing (a wrong/empty user reads as a false RED, exactly what #10 hit on ser8).

## Seed: per-target ssh user map The ser8 capture exposed that ssh user is per-target, not fleet-uniform (otel-a2a-relay-ansible-debug#10). Recording the map as it stands so the mesh check ships with it rather than re-discovering host-by-host: * `kai-server` -> **kai** (confirmed - ssh-exec green) * `ser8` -> **coilysiren** (confirmed - ssh-exec green) * `kai-desktop-tower` (windows native) -> **firem** (per Kai; UNVERIFIED - host offline last seen 17h) * `kai-desktop-tower-wsl` -> **kai** (per Kai, "I think"; UNVERIFIED - host offline) Verify the two tower rows by probing once those hosts are back online. Usernames are meaningful names, not opaque, so the map can live in committed inventory (not SSM) - same lookup shape as the per-target address, just a second column. A mesh ssh-exec probe must key the user off the target, defaulting to nothing (a wrong/empty user reads as a false RED, exactly what #10 hit on ser8).

coilysiren added the

label

2026-06-17 07:05:38 +00:00

coilysiren commented

2026-06-17 08:22:35 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:22:36 +00:00

closed this issue
added the
burndown-2026-06
label

coilyco-ops reopened this issue

2026-06-24 04:35:37 +00:00

coilyco-ops commented

2026-06-24 04:35:38 +00:00

Member

Reopening (the burndown-2026-06 close invited "reopen if this resurfaces" - it has).

Near-term scope: just the ser8 <-> kai-server pair

Rather than open with the full N x (N-1) fleet mesh, the first deliverable is the 2-node, 2-directed-pair case: kai-server -> ser8 and ser8 -> kai-server, rendered as a simple red/green list (the degenerate 2x2 of the connectivity matrix). The full-fleet mesh stays the eventual goal; this is the first pair to light up and proves the probe battery + render + alert path end to end.

Why these two first: they're the always-on Linux boxes carrying the observability + CI load, the ssh-user map for both is already confirmed green in the seed comment above (kai-server -> kai, ser8 -> coilysiren), and a fault between them is the highest-blast-radius reachability failure on the fleet.

Design constraints from the parent issue carry unchanged and are exactly what makes this non-trivial even for two nodes:

Kernel data path, not userspace - real TCP nc -z to 22 + 6443 and a real ssh-exec, NOT tailscale ping. The whole reason this issue exists is the ZHQK userspace-green / kernel-dead split.
Both directions independently - A->B can fail while B->A is fine.
Multiple probe types per pair - ICMP + TCP(22) + TCP(6443) + ssh-exec, so a partial fault can't hide behind one green probe.
Emit pass/fail per (src, dst, probe) into the observability stack already on these boxes, alert on transitions with a short flap window so an expected k3s/apiserver reboot gap doesn't page.

Generalizing from this 2-node case to the full fleet matrix is then "fan the same battery across all ordered host pairs" - the rollout (ansible role per the authoring-vs-rollout split) and the tower rows (still UNVERIFIED per the seed map) come after the pair is green and trustworthy.

Reopening (the `burndown-2026-06` close invited "reopen if this resurfaces" - it has). ## Near-term scope: just the ser8 <-> kai-server pair Rather than open with the full N x (N-1) fleet mesh, the first deliverable is the 2-node, 2-directed-pair case: **kai-server -> ser8** and **ser8 -> kai-server**, rendered as a simple red/green list (the degenerate 2x2 of the connectivity matrix). The full-fleet mesh stays the eventual goal; this is the first pair to light up and proves the probe battery + render + alert path end to end. Why these two first: they're the always-on Linux boxes carrying the observability + CI load, the ssh-user map for both is already confirmed green in the seed comment above (`kai-server -> kai`, `ser8 -> coilysiren`), and a fault between them is the highest-blast-radius reachability failure on the fleet. Design constraints from the parent issue carry unchanged and are exactly what makes this non-trivial even for two nodes: * **Kernel data path, not userspace** - real TCP `nc -z` to 22 + 6443 and a real ssh-exec, NOT `tailscale ping`. The whole reason this issue exists is the ZHQK userspace-green / kernel-dead split. * **Both directions independently** - A->B can fail while B->A is fine. * **Multiple probe types per pair** - ICMP + TCP(22) + TCP(6443) + ssh-exec, so a partial fault can't hide behind one green probe. * **Emit pass/fail per (src, dst, probe)** into the observability stack already on these boxes, **alert on transitions** with a short flap window so an expected k3s/apiserver reboot gap doesn't page. Generalizing from this 2-node case to the full fleet matrix is then "fan the same battery across all ordered host pairs" - the rollout (ansible role per the authoring-vs-rollout split) and the tower rows (still UNVERIFIED per the seed map) come after the pair is green and trustworthy.

coilyco-ops commented

2026-06-24 04:46:02 +00:00

Member

Architecture: per-node probe container -> central SigNoz aggregation

Settling the shape (SRE review):

Producer: an isolated probe container on every node

Each node runs a small, isolated container that probes itself -> every other node (its own outbound row of the matrix), on a cadence. Reachability is directed and must originate at the source: the ZHQK fault was mac-local egress, and a central poller can only ever measure "central -> X", never "A -> B". So the probe lives on each node by necessity, not preference.

The container is dumb: it runs the battery, emits a result, and forgets it. No storage, no UI, no SigNoz on the edge node.

Aggregator: SigNoz on ser8 only

Each probe container is just an OTLP exporter pointing at the existing collector on ser8 (signoz-otel-collector...:4318, or the tailnet endpoint). It emits a gauge per probe, e.g. fleet_reachability{src=<self>, dst=<peer>, probe=<type>} = 1|0. SigNoz aggregates every node's row into the full matrix, renders it, and alerts on transitions -> Telegram (per #369). We do not run SigNoz on every node; SigNoz is the single central sink and pane.

Probe set (ssh-exec REMOVED)

Probes per directed pair: ICMP, TCP connect to 22, TCP connect to 6443. ssh-exec is dropped from the requirements.

This is safe: the whole point was catching the userspace-green / kernel-dead split, and a kernel-path TCP connect (nc -z to 22/6443) already exercises the real kernel data path that black-holed during ZHQK - a TCP connect would have gone RED while userspace tailscale ping stayed green. ssh-exec was the redundant belt-and-suspenders layer on top of that. Dropping it also removes the worst part of the design: no ssh keys / per-target user map need to be distributed into every node's probe container. The seed ssh-user map comment above is now informational only, not a build requirement.

Hard constraint: host network namespace

The container must run with host networking (network: host / host netns), NOT a bridged container with its own netns. The probe has to traverse the host's real tailscale interface and kernel routes; a NAT'd bridge can route differently and hand back false greens. This is load-bearing for catching the exact fault class.

Rollout: ansible role, not a DaemonSet

kai-server and ser8 are separate k3s clusters (and the towers are Windows/WSL), so no single DaemonSet spans the fleet. Per the authoring-vs-rollout split, the probe tool/container is authored in its home repo and a host-level container fanned by an ansible role converges it onto every node. (k8s-native DaemonSet-per-cluster is a possible later optimization, but ansible host-container is the uniform path across mixed hosts.)

Near-term scope (unchanged)

First deliverable is still just the ser8 <-> kai-server pair (2 nodes, 2 directed rows) - the degenerate case that proves producer -> OTLP -> SigNoz aggregation -> render -> alert end to end, before fanning to the full mesh.

Still open: the rendering UI

The aggregation surface is decided (SigNoz). The rendering of the green/red matrix within/around SigNoz is still being chosen - tracked as the remaining open decision, comment to follow.

## Architecture: per-node probe container -> central SigNoz aggregation Settling the shape (SRE review): ### Producer: an isolated probe container on every node Each node runs a small, isolated container that probes **itself -> every other node** (its own outbound row of the matrix), on a cadence. Reachability is directed and **must originate at the source**: the ZHQK fault was mac-local *egress*, and a central poller can only ever measure "central -> X", never "A -> B". So the probe lives on each node by necessity, not preference. The container is dumb: it runs the battery, emits a result, and forgets it. No storage, no UI, no SigNoz on the edge node. ### Aggregator: SigNoz on ser8 only Each probe container is just an **OTLP exporter** pointing at the existing collector on ser8 (`signoz-otel-collector...:4318`, or the tailnet endpoint). It emits a gauge per probe, e.g. `fleet_reachability{src=<self>, dst=<peer>, probe=<type>} = 1|0`. SigNoz aggregates every node's row into the full matrix, renders it, and alerts on transitions -> Telegram (per #369). We do **not** run SigNoz on every node; SigNoz is the single central sink and pane. ### Probe set (ssh-exec REMOVED) Probes per directed pair: **ICMP**, **TCP connect to 22**, **TCP connect to 6443**. ssh-exec is **dropped from the requirements.** This is safe: the whole point was catching the userspace-green / kernel-dead split, and a **kernel-path TCP connect** (`nc -z` to 22/6443) already exercises the real kernel data path that black-holed during ZHQK - a TCP connect would have gone RED while userspace tailscale ping stayed green. ssh-exec was the redundant belt-and-suspenders layer on top of that. Dropping it also removes the worst part of the design: no ssh keys / per-target user map need to be distributed into every node's probe container. The seed ssh-user map comment above is now informational only, not a build requirement. ### Hard constraint: host network namespace The container must run with **host networking (`network: host` / host netns)**, NOT a bridged container with its own netns. The probe has to traverse the host's *real* tailscale interface and kernel routes; a NAT'd bridge can route differently and hand back false greens. This is load-bearing for catching the exact fault class. ### Rollout: ansible role, not a DaemonSet kai-server and ser8 are **separate k3s clusters** (and the towers are Windows/WSL), so no single DaemonSet spans the fleet. Per the authoring-vs-rollout split, the probe tool/container is authored in its home repo and a **host-level container fanned by an ansible role** converges it onto every node. (k8s-native DaemonSet-per-cluster is a possible later optimization, but ansible host-container is the uniform path across mixed hosts.) ### Near-term scope (unchanged) First deliverable is still just the ser8 <-> kai-server pair (2 nodes, 2 directed rows) - the degenerate case that proves producer -> OTLP -> SigNoz aggregation -> render -> alert end to end, before fanning to the full mesh. ### Still open: the rendering UI The aggregation surface is decided (SigNoz). The *rendering* of the green/red matrix within/around SigNoz is still being chosen - tracked as the remaining open decision, comment to follow.

coilyco-ops commented

2026-06-24 04:51:36 +00:00

Member

Availability requirement + UI/tooling decision: buy (Gatus), don't build

New hard requirement: must operate when SigNoz is down

The monitor cannot be hosted on the thing it monitors. SigNoz lives on ser8; a SigNoz-only UI/alert path goes dark exactly when ser8 is the fault - the one outage you most need to see. So live status, the UI, and alerting must all survive SigNoz/ser8 being down. SigNoz is demoted to a best-effort aggregator, never the source of truth for live state.

Decision: buy, don't build - Gatus, one instance per node

Dropping ssh-exec (previous comment) is what makes "buy" viable: every remaining probe is ICMP + TCP-connect, which off-the-shelf status-page tools do natively, so no custom probe code is needed. Gatus satisfies every requirement:

Per-node isolated container - one Gatus per node, each configured to probe every peer (its own directed row). Runs with host netns + cap_net_raw so ICMP/TCP traverse the host's real tailscale interface and kernel routes (the load-bearing constraint - a bridged netns can hand back false greens).
Probes - icmp://<peer>, tcp://<peer>:22, tcp://<peer>:6443, with conditions ([CONNECTED] == true, [RESPONSE_TIME] < N). Kernel-path TCP connect is what catches the ZHQK userspace-green/kernel-dead split. (Gatus also supports ssh:// if we ever want exec back - no custom code, just keys in config.)
Tier 1, always-on (source of truth) - Gatus's built-in status page per node IS the "simple red/green list". Fully self-contained: own UI, own storage, zero dependency on ser8. During a ser8 outage you open any surviving node's Gatus page and read its row directly.
Tier 1 alerting (dead-man path) - Gatus native Telegram alerting -> coilyco_ops_bot (#369), with failure-threshold (flap window) + send-on-resolved (recovery). Fires off-fleet, independent of ser8, so "ser8/SigNoz is down" actually pages.
Tier 2, best-effort aggregate - Gatus exposes Prometheus /metrics (set metrics: true). The existing vmagent (already scraping in the fleet) / SigNoz collector scrapes each node's Gatus for the cross-node N x N pane + history when SigNoz is up. This supersedes the "custom container pushes OTLP to SigNoz" detail from the architecture comment above - aggregation is pull-scrape of Gatus /metrics, not a bespoke OTLP exporter.

Why not Uptime Kuma

Capable and prettier, but config is web-UI/clickops with sqlite state - not config-as-code. Fanning a uniform per-node config across the fleet via ansible wants Gatus's single declarative YAML, generated from inventory.

Rollout (unchanged shape)

Ansible role fans the Gatus container + a per-node YAML (peer list generated from inventory) onto every node. Separate k3s clusters + Windows/WSL towers => host-level container via ansible, not a DaemonSet. Authoring-vs-rollout split honored: tool is upstream Gatus, the role is the fleet rollout.

Net stack

per-node Gatus (host netns) -> built-in red/green page + direct Telegram alerts [always on] ; Gatus /metrics -> vmagent/SigNoz scrape -> aggregated matrix + history [when ser8 up]. No custom probe code, no single point of failure for live status.

## Availability requirement + UI/tooling decision: buy (Gatus), don't build ### New hard requirement: must operate when SigNoz is down The monitor cannot be hosted on the thing it monitors. SigNoz lives on ser8; a SigNoz-only UI/alert path goes dark exactly when ser8 is the fault - the one outage you most need to see. So live status, the UI, and alerting must all survive SigNoz/ser8 being down. SigNoz is demoted to a **best-effort aggregator**, never the source of truth for live state. ### Decision: buy, don't build - Gatus, one instance per node Dropping ssh-exec (previous comment) is what makes "buy" viable: every remaining probe is ICMP + TCP-connect, which off-the-shelf status-page tools do natively, so no custom probe code is needed. [Gatus](https://github.com/TwiN/gatus) satisfies every requirement: * **Per-node isolated container** - one Gatus per node, each configured to probe every *peer* (its own directed row). Runs with **host netns + `cap_net_raw`** so ICMP/TCP traverse the host's real tailscale interface and kernel routes (the load-bearing constraint - a bridged netns can hand back false greens). * **Probes** - `icmp://<peer>`, `tcp://<peer>:22`, `tcp://<peer>:6443`, with conditions (`[CONNECTED] == true`, `[RESPONSE_TIME] < N`). Kernel-path TCP connect is what catches the ZHQK userspace-green/kernel-dead split. (Gatus also supports `ssh://` if we ever want exec back - no custom code, just keys in config.) * **Tier 1, always-on (source of truth)** - Gatus's built-in status page per node IS the "simple red/green list". Fully self-contained: own UI, own storage, zero dependency on ser8. During a ser8 outage you open any surviving node's Gatus page and read its row directly. * **Tier 1 alerting (dead-man path)** - Gatus native Telegram alerting -> `coilyco_ops_bot` (#369), with `failure-threshold` (flap window) + `send-on-resolved` (recovery). Fires off-fleet, independent of ser8, so "ser8/SigNoz is down" actually pages. * **Tier 2, best-effort aggregate** - Gatus exposes Prometheus `/metrics` (set `metrics: true`). The existing **vmagent** (already scraping in the fleet) / SigNoz collector scrapes each node's Gatus for the cross-node N x N pane + history when SigNoz is up. **This supersedes** the "custom container pushes OTLP to SigNoz" detail from the architecture comment above - aggregation is pull-scrape of Gatus `/metrics`, not a bespoke OTLP exporter. ### Why not Uptime Kuma Capable and prettier, but config is web-UI/clickops with sqlite state - not config-as-code. Fanning a uniform per-node config across the fleet via ansible wants Gatus's single declarative YAML, generated from inventory. ### Rollout (unchanged shape) Ansible role fans the Gatus container + a per-node YAML (peer list generated from inventory) onto every node. Separate k3s clusters + Windows/WSL towers => host-level container via ansible, not a DaemonSet. Authoring-vs-rollout split honored: tool is upstream Gatus, the role is the fleet rollout. ### Net stack per-node Gatus (host netns) -> built-in red/green page + direct Telegram alerts [always on] ; Gatus /metrics -> vmagent/SigNoz scrape -> aggregated matrix + history [when ser8 up]. No custom probe code, no single point of failure for live status.

coilysiren referenced this issue from a commit

2026-06-24 05:47:38 +00:00

feat(ansible): fleet-reachability Gatus rollout role for ser8 + kai-server

coilyco-ops referenced this issue

2026-06-24 06:44:14 +00:00

Host-level Gatus rollout to kai-tower-3026 (Windows-native + WSL, no-page) #398

coilyco-ops added the

headless

label

2026-07-09 21:46:32 +00:00

coilyco-ops added

interactive