ward exec sandbox fails on ser8 (apparmor_restrict_unprivileged_userns) — fleet-converge silently failing #370

New issue

Open

opened 2026-06-18 01:33:14 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-18 01:33:14 +00:00

Owner

ward exec's sandbox fails on ser8, so every audited fleet verb on the ops plane is broken — including the fleet-converge timer, which has almost certainly been failing silently since ser8 came up.

Symptom

$ ward exec ansible-sync action=apply hosts=ser8
ward: exec ansible-sync in .../infrastructure
ward: sandbox: make-rprivate: permission denied
exit status 1

Surfaced while activating the Telegram fleet producers (#369), but it is not specific to that verb — it hits any ward exec on ser8.

Root cause

ser8 runs with AppArmor's unprivileged-userns restriction on:

/proc/sys/kernel/apparmor_restrict_unprivileged_userns = 1 (Ubuntu 24.04 default hardening)
(kernel.unprivileged_userns_clone = 1 and user.max_user_namespaces = 235662 are both fine — it is AppArmor, not the classic toggles)

ward exec creates an unprivileged user+mount namespace for its sandbox and runs mount --make-rprivate; AppArmor denies the userns to ward (a Homebrew binary with no AppArmor profile), so make-rprivate returns EPERM.

Blast radius

fleet-converge (fleet-converge.sh runs ward exec ansible-sync action=apply every 30 min on ser8) has been failing on every tick. It pushes a high alert on failure — but coily/ntfy were down, so the alerts went nowhere. Double-silent.
Any other ward exec verb invoked on ser8 (interactive or timer).
kai-server is unaffected (different host / no such restriction), which is why the fleet-converge target is kai-server and the failure stayed hidden.

Fix options

AppArmor profile for ward (preferred, keeps the host hardened). Ship a profile permitting userns for the ward/coily binary path, loaded via an ansible role so it converges fleet-wide on Ubuntu 24.04 hosts. Hardening stays on for everything else.
Relax the restriction on ser8. kernel.apparmor_restrict_unprivileged_userns=0 via a persisted /etc/sysctl.d/ drop-in (ansible-managed). One line, un-breaks everything, but weakens a host hardening (unprivileged userns is an exploit primitive). Posture call.
Sandbox fallback in ward. ward could detect the EPERM on make-rprivate and degrade to an unsandboxed (audited-only) exec with a warning, instead of hard-failing. Belongs upstream in ward; tracks separately.

Also worth fixing regardless

fleet-converge failing silently for weeks is the exact "auto-converge that fails quietly is the disease it cures" failure mode its own docs warn about. Once Telegram alerting lands (#369), a failed converge will at least be loud. Consider a vmalert rule on a fleet-converge heartbeat/last-success timestamp so a wedged timer pages on its own.

Found during the ntfy->Telegram migration (#369). Host: ser8 (Beelink SER8, Ubuntu 24.04).

`ward exec`'s sandbox fails on ser8, so every audited fleet verb on the ops plane is broken — including the `fleet-converge` timer, which has almost certainly been failing silently since ser8 came up. ## Symptom ``` $ ward exec ansible-sync action=apply hosts=ser8 ward: exec ansible-sync in .../infrastructure ward: sandbox: make-rprivate: permission denied exit status 1 ``` Surfaced while activating the Telegram fleet producers (#369), but it is not specific to that verb — it hits any `ward exec` on ser8. ## Root cause ser8 runs with AppArmor's unprivileged-userns restriction on: - `/proc/sys/kernel/apparmor_restrict_unprivileged_userns = 1` (Ubuntu 24.04 default hardening) - (`kernel.unprivileged_userns_clone = 1` and `user.max_user_namespaces = 235662` are both fine — it is AppArmor, not the classic toggles) `ward exec` creates an unprivileged user+mount namespace for its sandbox and runs `mount --make-rprivate`; AppArmor denies the userns to `ward` (a Homebrew binary with no AppArmor profile), so `make-rprivate` returns EPERM. ## Blast radius - **fleet-converge** (`fleet-converge.sh` runs `ward exec ansible-sync action=apply` every 30 min on ser8) has been failing on every tick. It pushes a `high` alert on failure — but coily/ntfy were down, so the alerts went nowhere. Double-silent. - Any other `ward exec` verb invoked on ser8 (interactive or timer). - kai-server is unaffected (different host / no such restriction), which is why the fleet-converge target is `kai-server` and the failure stayed hidden. ## Fix options 1. **AppArmor profile for `ward` (preferred, keeps the host hardened).** Ship a profile permitting `userns` for the `ward`/`coily` binary path, loaded via an ansible role so it converges fleet-wide on Ubuntu 24.04 hosts. Hardening stays on for everything else. 2. **Relax the restriction on ser8.** `kernel.apparmor_restrict_unprivileged_userns=0` via a persisted `/etc/sysctl.d/` drop-in (ansible-managed). One line, un-breaks everything, but weakens a host hardening (unprivileged userns is an exploit primitive). Posture call. 3. **Sandbox fallback in ward.** ward could detect the EPERM on `make-rprivate` and degrade to an unsandboxed (audited-only) exec with a warning, instead of hard-failing. Belongs upstream in ward; tracks separately. ## Also worth fixing regardless fleet-converge failing silently for weeks is the exact "auto-converge that fails quietly is the disease it cures" failure mode its own docs warn about. Once Telegram alerting lands (#369), a failed converge will at least be loud. Consider a vmalert rule on a fleet-converge heartbeat/last-success timestamp so a wedged timer pages on its own. Found during the ntfy->Telegram migration (#369). Host: ser8 (Beelink SER8, Ubuntu 24.04).

coilysiren added the

interactive

labels

2026-06-18 06:18:17 +00:00