Fleet git-state o11y: detect and alert on unpushed, dirty, and stranded work

coilysiren commented

2026-06-12 11:54:12 +00:00

Owner

Kai loses track of unpushed work routinely. 2026-06-12 alone produced four specimens: an atlas commit stranded on kai-server (1 ahead, 5 behind, existed nowhere else), the agentic-os GitHub mirror silently 5 commits stale, kai-server checkouts reporting ahead counts against a deleted origin tracking ref, and a dirty voice-flow-learning-loop tree.

Design (devex o11y, no new UI - rides the ser8 o11y pipeline):

Collector: a sweep on each always-on box (or ser8 ssh-sweeping) walks ~/projects///, fetches before measuring (stale refs lie), and emits per-repo gauges: dirty file count, ahead/behind canonical, ahead/behind mirror, stale-tracking-ref flag, last-commit age. kai-server's coilysiren-pull-all.timer and mirror timers are adjacent machinery that measures nothing today.
Storage: gauges remote-write into VictoriaMetrics on ser8 like any other metric.
Alerting: vmalert rule fires ntfy when unpushed or dirty work crosses ~24h, one deduped daily digest line. Losing track is a forgetting failure, so the surface is push, not a dashboard.
Ad-hoc: vmui queries. Optional later: ahead/behind badges on atlas cards (atlas#5 already adds last-commit freshness), enrichment of an existing surface.

Related: #309 push mirrors delete the forgejo-to-github staleness class entirely and should land regardless. Sequencing: needs the ser8 metrics pipeline (o11y burn-down items 2-4) as the sink.

Kai loses track of unpushed work routinely. 2026-06-12 alone produced four specimens: an atlas commit stranded on kai-server (1 ahead, 5 behind, existed nowhere else), the agentic-os GitHub mirror silently 5 commits stale, kai-server checkouts reporting ahead counts against a deleted origin tracking ref, and a dirty voice-flow-learning-loop tree. Design (devex o11y, no new UI - rides the ser8 o11y pipeline): - Collector: a sweep on each always-on box (or ser8 ssh-sweeping) walks ~/projects/*/*/, fetches before measuring (stale refs lie), and emits per-repo gauges: dirty file count, ahead/behind canonical, ahead/behind mirror, stale-tracking-ref flag, last-commit age. kai-server's coilysiren-pull-all.timer and mirror timers are adjacent machinery that measures nothing today. - Storage: gauges remote-write into VictoriaMetrics on ser8 like any other metric. - Alerting: vmalert rule fires ntfy when unpushed or dirty work crosses ~24h, one deduped daily digest line. Losing track is a forgetting failure, so the surface is push, not a dashboard. - Ad-hoc: vmui queries. Optional later: ahead/behind badges on atlas cards (atlas#5 already adds last-commit freshness), enrichment of an existing surface. Related: #309 push mirrors delete the forgejo-to-github staleness class entirely and should land regardless. Sequencing: needs the ser8 metrics pipeline (o11y burn-down items 2-4) as the sink.

coilysiren added the

P2

label

2026-06-17 07:07:01 +00:00

coilysiren added

P3

and removed

P2

labels

2026-06-17 08:39:24 +00:00

coilysiren referenced this issue from a commit

2026-06-17 21:00:49 +00:00