kai-server exports no hostmetrics to SigNoz, so the busiest node has no CPU, load, or disk baseline #841

New issue

Open

opened 2026-08-15 16:53:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 16:53:08 +00:00

Owner

Summary

ser8 reports host-level metrics to SigNoz. kai-server does not. kai-server is the node that runs k3s, Forgejo, the Eco server, and every runner pool, so it is the one node where a baseline actually matters, and it is the one node that has none.

Evidence

Querying system.cpu.load_average.5m grouped by host.name over 2 days returns exactly one series:

host.name = "ser8"   (25 buckets, values 0.72 to 6.24)

There is no kai-server series. Filtering host.name = 'kai-server' returns an empty result set.

The metrics catalog matching cpu contains only:

node_stats.kubelet.cpu.usage
node_stats.kubernetes.namespace.cpu.usage
system.cpu.load_average.1m, .5m, .15m
system.cpu.time

The node_stats.* metrics come from the node-stats-mcp exporter and are Kubernetes-scoped. The system.* hostmetrics are ser8 only.

Why it matters

This blocked a live investigation today. kai-server's k3s-server process was measured at ~4.6 cores, and there was no way to tell whether that is a regression or the normal steady state for this node. The only available readings were instantaneous point samples taken during the incident.

The same gap applies to disk throughput and IO pressure. Both were measured by hand from /proc counters across three samples because no time series exists.

Ask

Converge the SigNoz hostmetrics receiver onto kai-server the same way it is configured for ser8, so that CPU, load average, memory, disk IO, and filesystem metrics land with host.name = "kai-server".

Acceptance

system.cpu.load_average.5m grouped by host.name returns a kai-server series, and disk IO counters for kai-server are queryable over at least a 7 day window.

## Summary ser8 reports host-level metrics to SigNoz. kai-server does not. kai-server is the node that runs k3s, Forgejo, the Eco server, and every runner pool, so it is the one node where a baseline actually matters, and it is the one node that has none. ## Evidence Querying `system.cpu.load_average.5m` grouped by `host.name` over 2 days returns exactly one series: ``` host.name = "ser8" (25 buckets, values 0.72 to 6.24) ``` There is no `kai-server` series. Filtering `host.name = 'kai-server'` returns an empty result set. The metrics catalog matching `cpu` contains only: - `node_stats.kubelet.cpu.usage` - `node_stats.kubernetes.namespace.cpu.usage` - `system.cpu.load_average.1m`, `.5m`, `.15m` - `system.cpu.time` The `node_stats.*` metrics come from the node-stats-mcp exporter and are Kubernetes-scoped. The `system.*` hostmetrics are ser8 only. ## Why it matters This blocked a live investigation today. kai-server's `k3s-server` process was measured at ~4.6 cores, and there was no way to tell whether that is a regression or the normal steady state for this node. The only available readings were instantaneous point samples taken during the incident. The same gap applies to disk throughput and IO pressure. Both were measured by hand from `/proc` counters across three samples because no time series exists. ## Ask Converge the SigNoz hostmetrics receiver onto kai-server the same way it is configured for ser8, so that CPU, load average, memory, disk IO, and filesystem metrics land with `host.name = "kai-server"`. ## Acceptance `system.cpu.load_average.5m` grouped by `host.name` returns a `kai-server` series, and disk IO counters for kai-server are queryable over at least a 7 day window.

coilyco-ops referenced this issue

2026-08-15 20:40:25 +00:00

kai-server: k3s-server process sustains ~4.6 cores with no identified cause #839

coilyco-ops referenced this issue

2026-08-15 20:47:22 +00:00

kai-server: k3s-server process sustains ~4.6 cores with no identified cause #839

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:09 +00:00

coilyco-ops added the