interactive: restore kai-server thermal heartbeat after checkout path move #645

New issue

Closed

opened 2026-07-25 14:38:29 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-25 14:38:29 +00:00

Owner

Problem

The new deploy#209 freshness probe exposed a live host failure on kai-server.

Observed on 2026-07-25:

get_configured_freshness finds /var/lib/node-exporter/textfile/thermal.prom, but reports it stale by about 47 days.
thermal-heartbeat.timer is enabled, active, and firing every minute.
thermal-heartbeat.service fails every tick with status=200/CHDIR.
The installed unit still points at /home/kai/projects/coilysiren/infrastructure, which no longer exists.
The canonical checkout exists at /home/kai/projects/coilyco-flight-deck/infrastructure.
The tracked systemd/thermal-heartbeat.service already contains the corrected canonical paths.

Remediation boundary

Applying the tracked unit requires attended sudo. The current ops session has no passwordless systemd or file-install authority, so it made no host mutation.

Use the existing tracked installer from the canonical checkout. It already installs the corrected unit, reloads systemd, enables the timer, and preserves the secret-safe SSM flow:

scripts/thermal-heartbeat-install.sh

Do not retrieve or print the Sentry DSN or cron URL while repairing this.

Acceptance

The installed unit uses the canonical coilyco-flight-deck/infrastructure checkout.
thermal-heartbeat.service completes successfully.
thermal.prom advances at the timer cadence.
node-stats get_configured_freshness reports thermal-heartbeat fresh within its 120-second threshold.
The node-exporter textfile series remains available.
The issue records the attended apply and live verification evidence.

Discovered while verifying coilyco-bridge/deploy#209.

## Problem The new deploy#209 freshness probe exposed a live host failure on kai-server. Observed on 2026-07-25: * `get_configured_freshness` finds `/var/lib/node-exporter/textfile/thermal.prom`, but reports it stale by about 47 days. * `thermal-heartbeat.timer` is enabled, active, and firing every minute. * `thermal-heartbeat.service` fails every tick with `status=200/CHDIR`. * The installed unit still points at `/home/kai/projects/coilysiren/infrastructure`, which no longer exists. * The canonical checkout exists at `/home/kai/projects/coilyco-flight-deck/infrastructure`. * The tracked `systemd/thermal-heartbeat.service` already contains the corrected canonical paths. ## Remediation boundary Applying the tracked unit requires attended sudo. The current ops session has no passwordless systemd or file-install authority, so it made no host mutation. Use the existing tracked installer from the canonical checkout. It already installs the corrected unit, reloads systemd, enables the timer, and preserves the secret-safe SSM flow: `scripts/thermal-heartbeat-install.sh` Do not retrieve or print the Sentry DSN or cron URL while repairing this. ## Acceptance * The installed unit uses the canonical `coilyco-flight-deck/infrastructure` checkout. * `thermal-heartbeat.service` completes successfully. * `thermal.prom` advances at the timer cadence. * node-stats `get_configured_freshness` reports `thermal-heartbeat` fresh within its 120-second threshold. * The node-exporter textfile series remains available. * The issue records the attended apply and live verification evidence. Discovered while verifying coilyco-bridge/deploy#209.

coilyco-ops

2026-07-29 03:46:31 +00:00

closed this issue
added the
autonomy
live-collab

priority
P4
labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#645

No description provided.

Rows
Columns