feat(ansible): provision docker-ce on kai-server, replacing snap docker (private /tmp blocks warded) #450

New issue

Closed

opened 2026-07-03 07:59:43 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-03 07:59:43 +00:00

Member

Why

kai-server's docker is the snap package (/snap/bin/docker, canonical rev 3505). Snap confinement gives the docker CLI a private /tmp and only the home interface, so any host-/tmp path handed to the CLI (--env-file, -v mounts, unix sockets) fails to open. This blocks every warded container mode on kai-server:

docker: open /tmp/ward-forgejo-env-2612683732: no such file or directory
exit status 125

Root cause + the four ward touchpoints are catalogued in coilyco-flight-deck/ward#557 (ward adds a fail-fast preflight there). This issue is the host remediation: swap snap docker for docker-ce so the CLI shares the host filesystem view.

No role owns docker today - dev-harness and signoz only command -v docker and assume it exists. Docker was hand-provisioned as snap. This adds the missing provisioning, the doctrinally-correct rollout home (authoring lives in ward; the fleet converge lives here).

Design

Mirror the existing snap-purge idiom already in playbooks/ser8-bootstrap.yml (it does snap remove --purge microk8s). This is the first apt + become: true provisioning role in the repo, so it sets the schema - ground it against ser8-bootstrap.yml and docs/ansible-safety-model.md.

New roles/docker (idempotent, become: true at role level):

Purge snap docker if present - snap remove --purge docker, guarded like the microk8s purge (tolerate 'not installed').
Add Docker's official apt repo + signing key (ansible.builtin.deb822_repository / apt_key + apt_repository).
Install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin.
Ensure dockerd enabled + started (ansible.builtin.systemd).
usermod -aG docker {{ ansible_user }} so kai runs docker without sudo (matches how snap docker was reachable as kai).
Assert docker CLI resolves to /usr/bin/docker, not /snap/bin/docker, at the end.

Wiring: gate to kai-server initially (when: inventory_hostname == 'kai-server'), ordered before dev-harness in playbooks/sync.yml so the docker-CLI check downstream sees docker-ce. Decide explicitly whether ser8 (k3s standby, does it run warded?) also wants docker-ce before widening the gate - default NO until confirmed, to avoid purging a working docker on a host that does not need the swap.

Docs: add docs/ansible-docker-role.md, link it from docs/ansible.md and docs/FEATURES.md (trifecta). Note the become/apt-first-instance in docs/ansible-safety-model.md if it establishes a new privileged pattern.

Verify

After the role lands, Kai runs the converge with sudo on kai-server, then command -v docker -> /usr/bin/docker and warded director surfaces a session cleanly (no exit-125). The converge is operator-run (sudo, host-mutating), not agent-run.

Immediate unblock (optional, before the role lands)

Manual on kai-server: sudo snap remove docker, install docker-ce from Docker's apt repo, sudo usermod -aG docker kai, re-login. But the role is the durable fix - prefer landing it.

Diagnosed from a director surface on kai-server, 2026-07-03. See ward#557.

## Why kai-server's `docker` is the **snap package** (`/snap/bin/docker`, canonical rev 3505). Snap confinement gives the docker CLI a **private `/tmp`** and only the `home` interface, so any host-`/tmp` path handed to the CLI (`--env-file`, `-v` mounts, unix sockets) fails to open. This blocks **every** warded container mode on kai-server: ``` docker: open /tmp/ward-forgejo-env-2612683732: no such file or directory exit status 125 ``` Root cause + the four ward touchpoints are catalogued in coilyco-flight-deck/ward#557 (ward adds a fail-fast preflight there). This issue is the **host remediation**: swap snap docker for docker-ce so the CLI shares the host filesystem view. No role owns docker today - `dev-harness` and `signoz` only `command -v docker` and assume it exists. Docker was hand-provisioned as snap. This adds the missing provisioning, the doctrinally-correct rollout home (authoring lives in ward; the fleet converge lives here). ## Design Mirror the existing snap-purge idiom already in `playbooks/ser8-bootstrap.yml` (it does `snap remove --purge microk8s`). This is the first apt + `become: true` provisioning role in the repo, so it sets the schema - ground it against `ser8-bootstrap.yml` and `docs/ansible-safety-model.md`. **New `roles/docker`** (idempotent, `become: true` at role level): 1. Purge snap docker if present - `snap remove --purge docker`, guarded like the microk8s purge (tolerate 'not installed'). 2. Add Docker's official apt repo + signing key (`ansible.builtin.deb822_repository` / `apt_key` + `apt_repository`). 3. Install `docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin`. 4. Ensure `dockerd` enabled + started (`ansible.builtin.systemd`). 5. `usermod -aG docker {{ ansible_user }}` so `kai` runs docker without sudo (matches how snap docker was reachable as kai). 6. Assert docker CLI resolves to `/usr/bin/docker`, not `/snap/bin/docker`, at the end. **Wiring**: gate to kai-server initially (`when: inventory_hostname == 'kai-server'`), ordered **before** `dev-harness` in `playbooks/sync.yml` so the docker-CLI check downstream sees docker-ce. Decide explicitly whether ser8 (k3s standby, does it run warded?) also wants docker-ce before widening the gate - default NO until confirmed, to avoid purging a working docker on a host that does not need the swap. **Docs**: add `docs/ansible-docker-role.md`, link it from `docs/ansible.md` and `docs/FEATURES.md` (trifecta). Note the become/apt-first-instance in `docs/ansible-safety-model.md` if it establishes a new privileged pattern. ## Verify After the role lands, Kai runs the converge with sudo on kai-server, then `command -v docker` -> `/usr/bin/docker` and `warded director` surfaces a session cleanly (no exit-125). The converge is operator-run (sudo, host-mutating), not agent-run. ## Immediate unblock (optional, before the role lands) Manual on kai-server: `sudo snap remove docker`, install docker-ce from Docker's apt repo, `sudo usermod -aG docker kai`, re-login. But the role is the durable fix - prefer landing it. Diagnosed from a director surface on kai-server, 2026-07-03. See ward#557.

coilyco-ops commented

2026-07-03 08:10:35 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-450 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T08:10:33Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-450` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T08:10:33Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-03 08:22:57 +00:00

feat(ansible): provision docker-ce on kai-server, replacing snap docker

coilysiren closed this issue

2026-07-03 08:22:57 +00:00

coilysiren commented

2026-07-03 08:24:43 +00:00

Owner

The role itself went smoothly. It maps almost one-to-one onto the snap-purge idiom in ser8-bootstrap.yml and the sibling docker-consuming roles, so the schema was already there to copy. What actually fought back was the pre-commit gate, not the feature. HEAD was already un-committable - the recent MSYS-path-conversion commits left 4-5 line comment blocks that trip the always_run code-comments 2-line cap, so nothing could land until I collapsed those in a separate hygiene commit. Then pylint's uv run wanted a Python 3.13 the container's read-only /opt/uv could not install, so I had to repoint UV_PYTHON_INSTALL_DIR at a writable dir to get the gate green. I also learned the code-comments YAML rule the hard way: it only scans tracked files, so my role's inline comments passed --all-files while untracked and then failed the instant I git added, which sent me folding them up into the header block.

Confidence in the code is decent but honestly untested. There is no ansible in this container and the converge is operator-run (sudo, host-mutating), so I could not exercise the role - the spots worth an eyeball are the deb822_repository entry and the suite/architecture templating. Worth filing: the two container-env frictions above (read-only /opt/uv breaking uv-run pylint, and HEAD landing with cap-busting comment blocks), and the still-open question of whether ser8 wants docker-ce before the kai-server gate widens - I kept it NO per the issue.

WARD-OUTCOME: done - roles/docker swaps snap docker for docker-ce on kai-server (purge, apt repo + keyring, install, dockerd, docker group, /usr/bin/docker assert), wired before dev-harness, landed on main. The role itself went smoothly. It maps almost one-to-one onto the snap-purge idiom in `ser8-bootstrap.yml` and the sibling docker-consuming roles, so the schema was already there to copy. What actually fought back was the pre-commit gate, not the feature. HEAD was already un-committable - the recent MSYS-path-conversion commits left 4-5 line comment blocks that trip the `always_run` code-comments 2-line cap, so nothing could land until I collapsed those in a separate hygiene commit. Then pylint's `uv run` wanted a Python 3.13 the container's read-only `/opt/uv` could not install, so I had to repoint `UV_PYTHON_INSTALL_DIR` at a writable dir to get the gate green. I also learned the code-comments YAML rule the hard way: it only scans tracked files, so my role's inline comments passed `--all-files` while untracked and then failed the instant I `git add`ed, which sent me folding them up into the header block. Confidence in the code is decent but honestly untested. There is no ansible in this container and the converge is operator-run (sudo, host-mutating), so I could not exercise the role - the spots worth an eyeball are the `deb822_repository` entry and the suite/architecture templating. Worth filing: the two container-env frictions above (read-only `/opt/uv` breaking uv-run pylint, and HEAD landing with cap-busting comment blocks), and the still-open question of whether ser8 wants docker-ce before the kai-server gate widens - I kept it NO per the issue.