docker role: add nvidia-container-toolkit for GPU-in-docker on kai-server (if/when wanted) #457

New issue

Open

opened 2026-07-03 19:14:11 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-03 19:14:11 +00:00

Member

Context

The docker-ce swap (infrastructure#450) purged snap docker, which had bundled a snap.docker.nvidia-container-toolkit.service. docker-ce installs no NVIDIA container runtime, so GPU-in-docker is not wired on kai-server after the swap.

Not an active regression

kai-server currently has no NVIDIA stack at all: nvidia-smi is absent and there are no /dev/nvidia* devices. The snap toolkit service existed but was inactive - there was no driver/GPU behind it. The fleet's actual GPU work runs on kai-tower via native ollama, not via docker on kai-server. So nothing GPU-shaped broke in the swap; this is a capability gap, not a outage.

The follow-up (low priority, gated on need)

If GPU-in-docker on kai-server is ever wanted (a CUDA container, GPU-backed model serving in a warded run, etc.), extend the ansible docker role to:

Ensure the NVIDIA driver is present (prereq - separate from the toolkit).
Add NVIDIA's apt repo + nvidia-container-toolkit package.
nvidia-ctk runtime configure --runtime=docker and restart dockerd so --gpus / runtime: nvidia work.

Gate it behind a docker_nvidia_enabled flag (default false), same shape as the docker role's existing docker_enabled gate, so it only fires on hosts that actually have a GPU. Until a GPU lands on kai-server, this stays parked.

Origin: surfaced during the docker-ce converge, infrastructure#450.

## Context The docker-ce swap (infrastructure#450) purged snap docker, which had bundled a `snap.docker.nvidia-container-toolkit.service`. docker-ce installs no NVIDIA container runtime, so GPU-in-docker is not wired on kai-server after the swap. ## Not an active regression kai-server currently has **no NVIDIA stack at all**: `nvidia-smi` is absent and there are no `/dev/nvidia*` devices. The snap toolkit service existed but was **inactive** - there was no driver/GPU behind it. The fleet's actual GPU work runs on kai-tower via native ollama, not via docker on kai-server. So nothing GPU-shaped broke in the swap; this is a capability gap, not a outage. ## The follow-up (low priority, gated on need) If GPU-in-docker on kai-server is ever wanted (a CUDA container, GPU-backed model serving in a warded run, etc.), extend the ansible `docker` role to: 1. Ensure the NVIDIA driver is present (prereq - separate from the toolkit). 2. Add NVIDIA's apt repo + `nvidia-container-toolkit` package. 3. `nvidia-ctk runtime configure --runtime=docker` and restart dockerd so `--gpus` / `runtime: nvidia` work. Gate it behind a `docker_nvidia_enabled` flag (default false), same shape as the docker role's existing `docker_enabled` gate, so it only fires on hosts that actually have a GPU. Until a GPU lands on kai-server, this stays parked. Origin: surfaced during the docker-ce converge, infrastructure#450.

coilysiren referenced this issue

2026-07-04 10:38:26 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #470

coilysiren referenced this issue

2026-07-04 10:38:33 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #471

coilyco-ops added the

label

2026-07-04 17:38:13 +00:00

coilyco-ops added the

consult

label

2026-07-07 04:08:42 +00:00

coilyco-ops added

interactive

and removed

consult

labels

2026-07-10 08:59:57 +00:00

coilyco-ops added the

consult

label

2026-07-10 11:58:02 +00:00

coilyco-ops commented

2026-07-10 13:13:09 +00:00

Author

Member

WARD-DISPATCH: failed ❌

failure details

This forwarded dispatch failed after the issue was already reserved.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/infrastructure#457 --harness codex --ward-version v0.580.0
Container: engineer-codex-infrastructure-457
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T131307Z-director-claude-sm47-coilyco-flight-deck-infrastructure-457.log
Failure: ward agent engineer --harness codex: refusing to dispatch the engineer role on coilyco-flight-deck/infrastructure#457: the issue is explicitly labeled interactive - remove that label or pass --force to override (consult/default issues dispatch normally)

Retry: choose another harness if the first one is down, or rerun with --force if the reservation is stale.

— Codex, via ward agent

WARD-DISPATCH: failed ❌ <details><summary>failure details</summary> This forwarded dispatch failed after the issue was already reserved. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/infrastructure#457 --harness codex --ward-version v0.580.0` Container: `engineer-codex-infrastructure-457` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T131307Z-director-claude-sm47-coilyco-flight-deck-infrastructure-457.log` Failure: `ward agent engineer --harness codex: refusing to dispatch the engineer role on coilyco-flight-deck/infrastructure#457: the issue is explicitly labeled interactive - remove that label or pass --force to override (consult/default issues dispatch normally)` Retry: choose another harness if the first one is down, or rerun with `--force` if the reservation is stale. </details>  — Codex, via `ward agent`

coilysiren removed the

consult

label

2026-07-10 23:32:55 +00:00