coilyco-flight-deck/infrastructure

Watch

ward exec sandbox fails on ser8 (apparmor_restrict_unprivileged_userns) — fleet-converge silently failing #370

New issue

Closed

opened 2026-06-18 01:33:14 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-18 01:33:14 +00:00

Owner

ward exec's sandbox fails on ser8, so every audited fleet verb on the ops plane is broken — including the fleet-converge timer, which has almost certainly been failing silently since ser8 came up.

Symptom

$ ward exec ansible-sync action=apply hosts=ser8
ward: exec ansible-sync in .../infrastructure
ward: sandbox: make-rprivate: permission denied
exit status 1

Surfaced while activating the Telegram fleet producers (#369), but it is not specific to that verb — it hits any ward exec on ser8.

Root cause

ser8 runs with AppArmor's unprivileged-userns restriction on:

/proc/sys/kernel/apparmor_restrict_unprivileged_userns = 1 (Ubuntu 24.04 default hardening)
(kernel.unprivileged_userns_clone = 1 and user.max_user_namespaces = 235662 are both fine — it is AppArmor, not the classic toggles)

ward exec creates an unprivileged user+mount namespace for its sandbox and runs mount --make-rprivate; AppArmor denies the userns to ward (a Homebrew binary with no AppArmor profile), so make-rprivate returns EPERM.

Blast radius

fleet-converge (fleet-converge.sh runs ward exec ansible-sync action=apply every 30 min on ser8) has been failing on every tick. It pushes a high alert on failure — but coily/ntfy were down, so the alerts went nowhere. Double-silent.
Any other ward exec verb invoked on ser8 (interactive or timer).
kai-server is unaffected (different host / no such restriction), which is why the fleet-converge target is kai-server and the failure stayed hidden.

Fix options

AppArmor profile for ward (preferred, keeps the host hardened). Ship a profile permitting userns for the ward/coily binary path, loaded via an ansible role so it converges fleet-wide on Ubuntu 24.04 hosts. Hardening stays on for everything else.
Relax the restriction on ser8. kernel.apparmor_restrict_unprivileged_userns=0 via a persisted /etc/sysctl.d/ drop-in (ansible-managed). One line, un-breaks everything, but weakens a host hardening (unprivileged userns is an exploit primitive). Posture call.
Sandbox fallback in ward. ward could detect the EPERM on make-rprivate and degrade to an unsandboxed (audited-only) exec with a warning, instead of hard-failing. Belongs upstream in ward; tracks separately.

Also worth fixing regardless

fleet-converge failing silently for weeks is the exact "auto-converge that fails quietly is the disease it cures" failure mode its own docs warn about. Once Telegram alerting lands (#369), a failed converge will at least be loud. Consider a vmalert rule on a fleet-converge heartbeat/last-success timestamp so a wedged timer pages on its own.

Found during the ntfy->Telegram migration (#369). Host: ser8 (Beelink SER8, Ubuntu 24.04).

`ward exec`'s sandbox fails on ser8, so every audited fleet verb on the ops plane is broken — including the `fleet-converge` timer, which has almost certainly been failing silently since ser8 came up. ## Symptom ``` $ ward exec ansible-sync action=apply hosts=ser8 ward: exec ansible-sync in .../infrastructure ward: sandbox: make-rprivate: permission denied exit status 1 ``` Surfaced while activating the Telegram fleet producers (#369), but it is not specific to that verb — it hits any `ward exec` on ser8. ## Root cause ser8 runs with AppArmor's unprivileged-userns restriction on: - `/proc/sys/kernel/apparmor_restrict_unprivileged_userns = 1` (Ubuntu 24.04 default hardening) - (`kernel.unprivileged_userns_clone = 1` and `user.max_user_namespaces = 235662` are both fine — it is AppArmor, not the classic toggles) `ward exec` creates an unprivileged user+mount namespace for its sandbox and runs `mount --make-rprivate`; AppArmor denies the userns to `ward` (a Homebrew binary with no AppArmor profile), so `make-rprivate` returns EPERM. ## Blast radius - **fleet-converge** (`fleet-converge.sh` runs `ward exec ansible-sync action=apply` every 30 min on ser8) has been failing on every tick. It pushes a `high` alert on failure — but coily/ntfy were down, so the alerts went nowhere. Double-silent. - Any other `ward exec` verb invoked on ser8 (interactive or timer). - kai-server is unaffected (different host / no such restriction), which is why the fleet-converge target is `kai-server` and the failure stayed hidden. ## Fix options 1. **AppArmor profile for `ward` (preferred, keeps the host hardened).** Ship a profile permitting `userns` for the `ward`/`coily` binary path, loaded via an ansible role so it converges fleet-wide on Ubuntu 24.04 hosts. Hardening stays on for everything else. 2. **Relax the restriction on ser8.** `kernel.apparmor_restrict_unprivileged_userns=0` via a persisted `/etc/sysctl.d/` drop-in (ansible-managed). One line, un-breaks everything, but weakens a host hardening (unprivileged userns is an exploit primitive). Posture call. 3. **Sandbox fallback in ward.** ward could detect the EPERM on `make-rprivate` and degrade to an unsandboxed (audited-only) exec with a warning, instead of hard-failing. Belongs upstream in ward; tracks separately. ## Also worth fixing regardless fleet-converge failing silently for weeks is the exact "auto-converge that fails quietly is the disease it cures" failure mode its own docs warn about. Once Telegram alerting lands (#369), a failed converge will at least be loud. Consider a vmalert rule on a fleet-converge heartbeat/last-success timestamp so a wedged timer pages on its own. Found during the ntfy->Telegram migration (#369). Host: ser8 (Beelink SER8, Ubuntu 24.04).

coilysiren added the

priority

autonomy

live-collab

labels

2026-06-18 06:18:17 +00:00

coilyco-ops removed the

autonomy

live-collab

label

2026-07-16 01:05:47 +00:00

coilyco-ops added

and removed

labels

2026-07-29 03:45:48 +00:00

coilyco-ops added the

role/platform

role/sysadmin

labels

2026-08-25 08:02:34 +00:00

coilyco-ops commented

2026-08-27 03:18:12 +00:00

Owner

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901).

Closing as superseded. The ward-apparmor role shipped the AppArmor profile this issue recommended and has been wired into sync.yml since 2026-07-12, and ward exec is retired in favour of just recipes, so fleet-converge no longer takes the sandbox path at all.

This was priority/P3, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless.

If this is live work, reopen it. The whole set is recoverable with state:closed label:burndown-2026-08.

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901). Closing as superseded. The `ward-apparmor` role shipped the AppArmor profile this issue recommended and has been wired into sync.yml since 2026-07-12, and `ward exec` is retired in favour of `just` recipes, so fleet-converge no longer takes the sandbox path at all. This was `priority/P3`, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless. If this is live work, reopen it. The whole set is recoverable with `state:closed label:burndown-2026-08`.

coilyco-ops

2026-08-27 03:18:15 +00:00

closed this issue
added the
burndown-2026-08
label

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#370

No description provided.

Rows
Columns