coilyco-flight-deck/infrastructure

Watch

kai-server privileged Ansible cannot converge unattended, and authored fixes are silently sitting unapplied #923

New issue

Open

opened 2026-08-25 05:47:29 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-25 05:47:29 +00:00

Owner

Found while working #917 and #864 on 2026-08-25. Both hit the same wall, which is why this is its own issue rather than a note on either.

The wall

Any privileged role targeting kai-server fails immediately:

$ just ansible-sync check fleet privileged tags=restic-backup
fatal: [kai-server]: FAILED! => {"changed": false, "msg": "Task failed: Missing sudo password"}

ser8 has passwordless sudo and converges clean in the same run. kai-server needs ask_pass and a human at a prompt.

Why it matters more than an inconvenience

The failure mode is not "the run errors and someone fixes it". It is that work gets authored, reviewed, merged, and then quietly never reaches the host, while the repository looks like it shipped. Two confirmed instances, found by accident within an hour of each other:

k3s-config pod resolver (#864). The role sets DNSStubListenerExtra on kai-server's own LAN address and points pods at that single nameserver, which fixes the DNSConfigForming warnings outright. On the host: config.yaml has no resolv-conf: key, /etc/rancher/k3s/resolv.conf does not exist, and dig @192.168.0.194 is connection-refused. Meanwhile the warning count reached 31,146 on coredns alone.
restic failure alerts (#917, landed in #921). Applied cleanly to ser8. kai-server, which runs restic-backup and is the host whose backup actually matters, is uncovered.

Neither was noticed by anything. Both were found by reading the host rather than the repo. There is no signal that distinguishes "converged" from "authored but never applied", which is the same silent-failure shape #917 was about.

What would resolve it

Not proposing a specific mechanism, since the tradeoffs are Kai's. The options are roughly:

Passwordless sudo for the converge account on kai-server, scoped the way ser8 already is. Simplest, and makes the two hosts consistent.
A narrowly scoped sudoers entry covering only what the privileged roles need, in the spirit of the existing kai-disk-pressure-report sudoers file.
Accept the attended model and add drift detection, so an unconverged host is visible without anyone going to look. just up-to-date check already exists as a read-only detector pattern in the fleet.

Option 3 is worth having regardless of 1 or 2, because it is the part that turns a silent gap into a reported one.

Done means

Either privileged roles converge kai-server without a human at a prompt, or something reports when kai-server's converged state has drifted from the repository. Right now neither is true, and the only way to find out is to SSH in and read files by hand.

Found while working #917 and #864 on 2026-08-25. Both hit the same wall, which is why this is its own issue rather than a note on either. ## The wall Any privileged role targeting kai-server fails immediately: ``` $ just ansible-sync check fleet privileged tags=restic-backup fatal: [kai-server]: FAILED! => {"changed": false, "msg": "Task failed: Missing sudo password"} ``` ser8 has passwordless sudo and converges clean in the same run. kai-server needs `ask_pass` and a human at a prompt. ## Why it matters more than an inconvenience The failure mode is not "the run errors and someone fixes it". It is that **work gets authored, reviewed, merged, and then quietly never reaches the host**, while the repository looks like it shipped. Two confirmed instances, found by accident within an hour of each other: * **`k3s-config` pod resolver (#864).** The role sets `DNSStubListenerExtra` on kai-server's own LAN address and points pods at that single nameserver, which fixes the `DNSConfigForming` warnings outright. On the host: `config.yaml` has no `resolv-conf:` key, `/etc/rancher/k3s/resolv.conf` does not exist, and `dig @192.168.0.194` is connection-refused. Meanwhile the warning count reached 31,146 on coredns alone. * **restic failure alerts (#917, landed in #921).** Applied cleanly to ser8. kai-server, which runs `restic-backup` and is the host whose backup actually matters, is uncovered. Neither was noticed by anything. Both were found by reading the host rather than the repo. There is no signal that distinguishes "converged" from "authored but never applied", which is the same silent-failure shape #917 was about. ## What would resolve it Not proposing a specific mechanism, since the tradeoffs are Kai's. The options are roughly: 1. **Passwordless sudo for the converge account on kai-server**, scoped the way ser8 already is. Simplest, and makes the two hosts consistent. 2. **A narrowly scoped sudoers entry** covering only what the privileged roles need, in the spirit of the existing `kai-disk-pressure-report` sudoers file. 3. **Accept the attended model and add drift detection**, so an unconverged host is visible without anyone going to look. `just up-to-date check` already exists as a read-only detector pattern in the fleet. Option 3 is worth having regardless of 1 or 2, because it is the part that turns a silent gap into a reported one. ## Done means Either privileged roles converge kai-server without a human at a prompt, or something reports when kai-server's converged state has drifted from the repository. Right now neither is true, and the only way to find out is to SSH in and read files by hand.

coilyco-ops referenced this issue

2026-08-25 06:11:09 +00:00

Both k3s nodes silently drop a nameserver from every pod, ~65k warnings and counting #811

coilyco-ops referenced this issue

2026-08-25 06:11:12 +00:00

interactive: apply and verify k3s pod resolver #688

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:03 +00:00

coilyco-ops commented

2026-08-25 09:08:43 +00:00

Author

Owner

Correcting this issue's central claim. I filed it saying nothing reports the gap. Something does, it has been right for three weeks, and nobody has read it. That is a different problem with a different fix.

The detector already exists

fleet-converge compares each host's last successful full-apply marker against the current privileged
Git-tree fingerprint, which docs/fleet-converge.md documents under Privileged status. On kai-server right now:

~/.local/state/fleet-converge/privileged-status.current   kai-server=missing
~/.local/state/fleet-converge/privileged-pending          fingerprint=b550993a…  hosts=kai-server
~/.local/state/fleet-converge/last-privileged-apply       does not exist

All three files last written 2026-08-04. missing means no successful untagged full apply has ever written the marker on that host, so kai-server has been correctly flagged as carrying pending root-level work for three weeks.

It was right about the restic failure-alert rollout and the k3s pod resolver before either was authored, because it is fingerprint-based rather than per-role.

So the real gap is delivery, not detection

Three things make a correct finding invisible:

The timer is absent by default. fleet_converge_enabled: false is the role default, so fleet-converge.timer does not exist on kai-server. The detector only advances when someone runs systemctl --user start fleet-converge.service by hand.
Its output is a user journal and two state files. journalctl --user -u fleet-converge.service currently returns -- No entries --. Nothing pushes, nothing alerts, nothing fails a check.
A stale marker looks identical to a healthy one unless you check the mtime. Three-week-old files reporting missing read the same as fresh ones.

This is the same shape as everything else found on 2026-08-25: coilyco-bridge/deploy#791 (a rollout orphaning a release with no signal), deploy#793 (a tool returning empty results while logging outcome: ok), and the ComfyUI ExternalSecret that reported SecretSynced True for 46 days while delivering a credential to nobody. The mechanism works and reports into a void.

Which reorders the options in the body

Option 3, drift detection, is already built. It does not need building, it needs somewhere to arrive. That is much cheaper than what I originally described.

Concretely, in rough order of effort:

Route the existing finding somewhere read. The estate already has a Telegram alert path, and restic-failure-alert (#921) is a worked example of a systemd unit posting to it. A privileged-status.current of missing or stale is exactly as alertable as a failed backup.
Enable the timer so the marker advances without a human. fleet_converge_enabled: true in inventory, which the doc notes is a deliberate configuration change rather than a runtime toggle.
Passwordless sudo for the converge account on kai-server, matching ser8, which removes the prompt that causes the pending state in the first place.

1 and 3 are independent and both worth having. 1 makes the gap visible; 3 stops it recurring. 2 is what makes 1 fire more than once.

One caveat on reading the marker

Two things it does not mean, worth stating because both would mislead:

A tagged run applies real changes but deliberately does not write the marker. A host can be genuinely current on the role you care about and still report missing.
missing does not name the unapplied roles. Checking a specific one still means asking the host, as in systemctl show restic-backup.service -p OnFailure --value.

Why this is not in docs/

I tried to document it in docs/fleet-converge.md and could not. That file is at 7,962 of 8,000 characters, 38 characters of headroom, and docs/ holds 41 pages against a 40-page cap. The catalog-doc-size hook rejects both a new page and an inline addition, and it explicitly anticipates the workaround:

splitting one doc into two to clear the size cap trades one violation for another

So this content lives here until #855 and #856 free a slot. Trimming another author's dense prose to make room for mine seemed the wrong trade to make unilaterally. Flagging it because the next person with something to document will hit the same wall, and the caps are currently a hard blocker on new infrastructure documentation rather than a nudge.

**Correcting this issue's central claim.** I filed it saying nothing reports the gap. Something does, it has been right for three weeks, and nobody has read it. That is a different problem with a different fix. ## The detector already exists `fleet-converge` compares each host's last successful full-apply marker against the current privileged Git-tree fingerprint, which `docs/fleet-converge.md` documents under **Privileged status**. On kai-server right now: ``` ~/.local/state/fleet-converge/privileged-status.current kai-server=missing ~/.local/state/fleet-converge/privileged-pending fingerprint=b550993a… hosts=kai-server ~/.local/state/fleet-converge/last-privileged-apply does not exist ``` All three files last written **2026-08-04**. `missing` means no successful untagged full apply has *ever* written the marker on that host, so kai-server has been correctly flagged as carrying pending root-level work for three weeks. It was right about the restic failure-alert rollout and the k3s pod resolver before either was authored, because it is fingerprint-based rather than per-role. ## So the real gap is delivery, not detection Three things make a correct finding invisible: * **The timer is absent by default.** `fleet_converge_enabled: false` is the role default, so `fleet-converge.timer` does not exist on kai-server. The detector only advances when someone runs `systemctl --user start fleet-converge.service` by hand. * **Its output is a user journal and two state files.** `journalctl --user -u fleet-converge.service` currently returns `-- No entries --`. Nothing pushes, nothing alerts, nothing fails a check. * **A stale marker looks identical to a healthy one** unless you check the mtime. Three-week-old files reporting `missing` read the same as fresh ones. This is the same shape as everything else found on 2026-08-25: `coilyco-bridge/deploy#791` (a rollout orphaning a release with no signal), `deploy#793` (a tool returning empty results while logging `outcome: ok`), and the ComfyUI ExternalSecret that reported `SecretSynced True` for 46 days while delivering a credential to nobody. **The mechanism works and reports into a void.** ## Which reorders the options in the body Option 3, drift detection, is **already built**. It does not need building, it needs somewhere to arrive. That is much cheaper than what I originally described. Concretely, in rough order of effort: 1. **Route the existing finding somewhere read.** The estate already has a Telegram alert path, and `restic-failure-alert` (#921) is a worked example of a systemd unit posting to it. A `privileged-status.current` of `missing` or `stale` is exactly as alertable as a failed backup. 2. **Enable the timer** so the marker advances without a human. `fleet_converge_enabled: true` in inventory, which the doc notes is a deliberate configuration change rather than a runtime toggle. 3. **Passwordless sudo for the converge account on kai-server**, matching ser8, which removes the prompt that causes the pending state in the first place. 1 and 3 are independent and both worth having. 1 makes the gap visible; 3 stops it recurring. 2 is what makes 1 fire more than once. ## One caveat on reading the marker Two things it does not mean, worth stating because both would mislead: * A **tagged** run applies real changes but deliberately does not write the marker. A host can be genuinely current on the role you care about and still report `missing`. * `missing` does not name the unapplied roles. Checking a specific one still means asking the host, as in `systemctl show restic-backup.service -p OnFailure --value`. ## Why this is not in docs/ I tried to document it in `docs/fleet-converge.md` and could not. That file is at **7,962 of 8,000 characters**, 38 characters of headroom, and `docs/` holds **41 pages against a 40-page cap**. The `catalog-doc-size` hook rejects both a new page and an inline addition, and it explicitly anticipates the workaround: > splitting one doc into two to clear the size cap trades one violation for another So this content lives here until `#855` and `#856` free a slot. Trimming another author's dense prose to make room for mine seemed the wrong trade to make unilaterally. Flagging it because the next person with something to document will hit the same wall, and the caps are currently a hard blocker on new infrastructure documentation rather than a nudge.

coilyco-ops referenced this issue

2026-08-25 09:13:43 +00:00

CoreDNS SERVFAIL window took out CI, Flux, and external-secrets (08:29 to 08:54 UTC, 2026-08-25) #926

coilyco-ops referenced this issue

2026-08-25 09:13:55 +00:00

Both k3s nodes silently drop a nameserver from every pod, ~65k warnings and counting #811

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:53 +00:00

coilyco-ops referenced this issue