Agent pod fleet: SSH-able Claude / Codex / OpenAI runtimes on k3s #8

New issue

Closed

opened 2026-05-23 20:54:26 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:26 +00:00

Owner

Originally filed by @coilysiren on 2026-05-22T20:33:06Z - https://github.com/coilysiren/infrastructure/issues/283

Goal

Run Kai's Claude, Codex, and OpenAI agents as pods on the kai-server k3s cluster instead of natively on each host. Today each agent inherits its host's toolchain (brew on the Mac, linuxbrew on kai-server, Git Bash on Windows), so coily, paths, and tool versions drift per machine. A single pod image gives every agent an identical Linux userland no matter which node it lands on. Each pod sits on the tailnet, so ssh kai@agent-claude-<node> drops Kai into the agent's shell the same way ssh kai@kai-server does today.

This started as a "just run them in Docker" idea. Kubernetes is the better fit because the cluster already has the hard parts: the Tailscale operator (393d), ExternalSecrets (401d), and cert-manager are all running.

Shape

Namespace agents.
Three DaemonSets: agent-claude, agent-codex, agent-openai. One per runtime, so each is restarted, version-pinned, and SSH'd into independently. Not one DaemonSet with three containers.
nodeSelector: agent-host=true. A node opts into the fleet by carrying that label. This keeps the "one per node" model while decoupling rollout from node count. Label a node, pods appear.
tailscaled runs inside the agent container, not as a separate ts sidecar like deploy/repo-recall.yml. tailscale up --ssh then puts the agent's own shell on the tailnet. A sidecar would land SSH in the sidecar container, not the agent. The sidecar pattern is for exposing an HTTP service. The goal here is a shell.
State (the repo working tree) lives on a per-node hostPath. The nodes are durable physical machines, so work survives a pod restart by living on the node. DaemonSets have no volumeClaimTemplate anyway.
Secrets via ExternalSecrets from SSM, already the cluster pattern. No home-directory bind mount.

Hard constraint: only kai-server can host the fleet today

A tailnet-attached pod needs kernel-mode tailscale, which needs /dev/net/tun. The two WSL2-backed nodes (kai-macbook-pro-vm, kai-desktop-tower-wsl) don't expose that device to containerd. deploy/repo-recall.yml pins to kai-server for exactly this reason. So at rollout only kai-server carries agent-host=true and the fleet is three pods. Expanding to the other nodes (and whether that is even wanted, since they back machines Kai uses interactively) is tracked in #285.

Tailscale SSH and the UID question

tailscaled needs root for the netlink and TUN ioctls. The agent must write /home/kai/projects on the host, owned by kai (UID 1000 on kai-server). So the agent image carries a kai user at UID 1000, the entrypoint starts tailscaled as root, then runs the agent runtime as kai. Tailscale SSH execs the inbound session as kai via the tailnet ACL ssh rule. Files stay owned by 1000, tailscaled stays happy. This is the reason tailscaled can co-locate with the agent without the container-split that repo-recall.yml uses.

Auth keys

Mint one reusable, ephemeral, tag:agent auth key per runtime in terraform/tailscale-devices/, sync to SSM at /coilysiren/agents/<runtime>/ts-authkey. Reusable so a DaemonSet spanning nodes shares one key. Ephemeral so a dead pod deregisters instead of littering the tailnet. Tagged so a tailnet ACL can scope who reaches the agent pods (only Kai's devices).

Drafted manifest

agent-claude shown. agent-codex and agent-openai are identical bar the name and image. Land all three plus the namespace and RBAC in deploy/agents.yml.

# Agent pod fleet - SSH-able Claude runtime on k3s. One DaemonSet per
# runtime; this is the claude instance.
#
# tailscaled runs inside the agent container so `tailscale up --ssh`
# puts the agent's own shell on the tailnet: `ssh kai@agent-claude-<node>`
# lands where coily and the repos live, same feel as ssh'ing kai-server.
#
# DaemonSet + nodeSelector agent-host=true: label a node to opt it in.
# Only kai-server is labelled today - tailnet pods need kernel-mode
# tailscale (/dev/net/tun), which the WSL2 nodes don't expose. See the
# node-enablement follow-up issue.
#
# Apply: sudo k3s kubectl apply -f deploy/agents.yml  (confirm first)
---
apiVersion: v1
kind: Namespace
metadata:
  name: agents
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-claude
  namespace: agents
---
# In-container tailscaled persists its device identity into a k8s Secret
# (TS_KUBE_SECRET) so pod restarts reuse the node identity. Secret name
# is per-node so the DaemonSet can span nodes without collision, which is
# why the Role can't pin resourceNames - it's scoped to the dedicated
# `agents` namespace instead.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-ts-state
  namespace: agents
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-claude-ts-state
  namespace: agents
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: agent-ts-state
subjects:
  - kind: ServiceAccount
    name: agent-claude
    namespace: agents
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: agent-claude
  namespace: agents
spec:
  selector:
    matchLabels:
      app: agent-claude
  template:
    metadata:
      labels:
        app: agent-claude
    spec:
      serviceAccountName: agent-claude
      # Opt a node in: kubectl label node kai-server agent-host=true
      nodeSelector:
        agent-host: "true"
      containers:
        - name: agent
          image: ghcr.io/coilysiren/agent-claude:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Per-node tailnet identity + state secret, so the DaemonSet
            # can span nodes later without name collision.
            - name: TS_HOSTNAME
              value: "agent-claude-$(NODE_NAME)"
            - name: TS_KUBE_SECRET
              value: "agent-claude-ts-$(NODE_NAME)"
            - name: TS_AUTH_ONCE
              value: "true"
            - name: TS_EXTRA_ARGS
              value: "--ssh --accept-dns=false --advertise-tags=tag:agent"
            - name: TS_AUTHKEY
              valueFrom:
                secretKeyRef:
                  name: agent-claude-ts-authkey
                  key: authkey
          securityContext:
            # Kernel-mode tailscale needs a TUN device. NET_ADMIN + the
            # mounted /dev/net/tun is narrower than the full privileged:
            # true that repo-recall.yml needs for the WSL2 kernel. Verify
            # on kai-server's native 6.8 kernel; widen only if needed.
            capabilities:
              add: ["NET_ADMIN"]
          volumeMounts:
            - name: tun
              mountPath: /dev/net/tun
            - name: workspace
              mountPath: /home/kai/projects
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: "4"
              memory: 4Gi
      volumes:
        - name: tun
          hostPath:
            path: /dev/net/tun
            type: CharDevice
        # Per-node persistent workspace. hostPath, not a PVC: the nodes
        # are durable physical machines, so work survives a pod restart
        # on the node. Read-write - the agent edits repos.
        - name: workspace
          hostPath:
            path: /home/kai/projects
            type: Directory
---
# ts-authkey minted by terraform/tailscale-devices/ and synced to SSM.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: agent-claude-ts-authkey
  namespace: agents
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-parameter-store
    kind: ClusterSecretStore
  target:
    name: agent-claude-ts-authkey
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        authkey: "{{ .authkey }}"
  data:
    - secretKey: authkey
      remoteRef:
        key: /coilysiren/agents/claude/ts-authkey

Sub-tasks

Build the agent images (ghcr.io/coilysiren/agent-<runtime>): base userland, coily, the runtime CLI, tailscaled, a kai UID-1000 user, the drop-privileges entrypoint. Ship via a docker.yml GHA workflow like the other repos.
Mint three reusable-ephemeral tag:agent auth keys in terraform/tailscale-devices/, sync to SSM under /coilysiren/agents/<runtime>/ts-authkey.
Write deploy/agents.yml: namespace, RBAC, three DaemonSets, three ExternalSecrets.
Tailnet ACL: tag:agent reachable over SSH only from Kai's own devices.
Label kai-server agent-host=true, apply, verify ssh kai@agent-claude-kai-server.
Add a deploy-notes entry to docs/k3s-deploy-notes.md once it works.

Open decisions

Always-on vs on-demand. Three runtimes is three always-on pods per node, some burning tokens on heartbeat or proactive loops. Resident agents (OpenClaw heartbeat) want always-on. Interactive Claude/Codex sessions Kai SSHes into occasionally do not. The draft defaults to always-on DaemonSets as warm shells. Revisit if idle cost bites.
Where the image build lives. A new coilysiren/agent-images repo, or a directory in infrastructure. Leaning new repo to match the per-repo docker.yml pattern.
Workspace mount scope. The draft mounts all of /home/kai/projects read-write. Could scope per-runtime or use read-only plus an overlay if blast radius matters.

Out of scope

The interactive-vs-resident split is noted, not designed here.
Enabling the WSL2 nodes: #285.

_Originally filed by @coilysiren on 2026-05-22T20:33:06Z - [https://github.com/coilysiren/infrastructure/issues/283](https://github.com/coilysiren/infrastructure/issues/283)_ ## Goal Run Kai's Claude, Codex, and OpenAI agents as pods on the `kai-server` k3s cluster instead of natively on each host. Today each agent inherits its host's toolchain (brew on the Mac, linuxbrew on kai-server, Git Bash on Windows), so `coily`, paths, and tool versions drift per machine. A single pod image gives every agent an identical Linux userland no matter which node it lands on. Each pod sits on the tailnet, so `ssh kai@agent-claude-<node>` drops Kai into the agent's shell the same way `ssh kai@kai-server` does today. This started as a "just run them in Docker" idea. Kubernetes is the better fit because the cluster already has the hard parts: the Tailscale operator (393d), ExternalSecrets (401d), and cert-manager are all running. ## Shape - Namespace `agents`. - Three DaemonSets: `agent-claude`, `agent-codex`, `agent-openai`. One per runtime, so each is restarted, version-pinned, and SSH'd into independently. Not one DaemonSet with three containers. - `nodeSelector: agent-host=true`. A node opts into the fleet by carrying that label. This keeps the "one per node" model while decoupling rollout from node count. Label a node, pods appear. - `tailscaled` runs inside the agent container, not as a separate `ts` sidecar like `deploy/repo-recall.yml`. `tailscale up --ssh` then puts the agent's own shell on the tailnet. A sidecar would land SSH in the sidecar container, not the agent. The sidecar pattern is for exposing an HTTP service. The goal here is a shell. - State (the repo working tree) lives on a per-node `hostPath`. The nodes are durable physical machines, so work survives a pod restart by living on the node. DaemonSets have no `volumeClaimTemplate` anyway. - Secrets via ExternalSecrets from SSM, already the cluster pattern. No home-directory bind mount. ## Hard constraint: only kai-server can host the fleet today A tailnet-attached pod needs kernel-mode tailscale, which needs `/dev/net/tun`. The two WSL2-backed nodes (`kai-macbook-pro-vm`, `kai-desktop-tower-wsl`) don't expose that device to containerd. `deploy/repo-recall.yml` pins to `kai-server` for exactly this reason. So at rollout only `kai-server` carries `agent-host=true` and the fleet is three pods. Expanding to the other nodes (and whether that is even wanted, since they back machines Kai uses interactively) is tracked in #285. ## Tailscale SSH and the UID question `tailscaled` needs root for the netlink and TUN ioctls. The agent must write `/home/kai/projects` on the host, owned by `kai` (UID 1000 on kai-server). So the agent image carries a `kai` user at UID 1000, the entrypoint starts `tailscaled` as root, then runs the agent runtime as `kai`. Tailscale SSH execs the inbound session as `kai` via the tailnet ACL `ssh` rule. Files stay owned by 1000, `tailscaled` stays happy. This is the reason `tailscaled` can co-locate with the agent without the container-split that `repo-recall.yml` uses. ## Auth keys Mint one reusable, ephemeral, `tag:agent` auth key per runtime in `terraform/tailscale-devices/`, sync to SSM at `/coilysiren/agents/<runtime>/ts-authkey`. Reusable so a DaemonSet spanning nodes shares one key. Ephemeral so a dead pod deregisters instead of littering the tailnet. Tagged so a tailnet ACL can scope who reaches the agent pods (only Kai's devices). ## Drafted manifest `agent-claude` shown. `agent-codex` and `agent-openai` are identical bar the name and image. Land all three plus the namespace and RBAC in `deploy/agents.yml`. ```yaml # Agent pod fleet - SSH-able Claude runtime on k3s. One DaemonSet per # runtime; this is the claude instance. # # tailscaled runs inside the agent container so `tailscale up --ssh` # puts the agent's own shell on the tailnet: `ssh kai@agent-claude-<node>` # lands where coily and the repos live, same feel as ssh'ing kai-server. # # DaemonSet + nodeSelector agent-host=true: label a node to opt it in. # Only kai-server is labelled today - tailnet pods need kernel-mode # tailscale (/dev/net/tun), which the WSL2 nodes don't expose. See the # node-enablement follow-up issue. # # Apply: sudo k3s kubectl apply -f deploy/agents.yml (confirm first) --- apiVersion: v1 kind: Namespace metadata: name: agents --- apiVersion: v1 kind: ServiceAccount metadata: name: agent-claude namespace: agents --- # In-container tailscaled persists its device identity into a k8s Secret # (TS_KUBE_SECRET) so pod restarts reuse the node identity. Secret name # is per-node so the DaemonSet can span nodes without collision, which is # why the Role can't pin resourceNames - it's scoped to the dedicated # `agents` namespace instead. apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: agent-ts-state namespace: agents rules: - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "update", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: agent-claude-ts-state namespace: agents roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: agent-ts-state subjects: - kind: ServiceAccount name: agent-claude namespace: agents --- apiVersion: apps/v1 kind: DaemonSet metadata: name: agent-claude namespace: agents spec: selector: matchLabels: app: agent-claude template: metadata: labels: app: agent-claude spec: serviceAccountName: agent-claude # Opt a node in: kubectl label node kai-server agent-host=true nodeSelector: agent-host: "true" containers: - name: agent image: ghcr.io/coilysiren/agent-claude:latest imagePullPolicy: IfNotPresent env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName # Per-node tailnet identity + state secret, so the DaemonSet # can span nodes later without name collision. - name: TS_HOSTNAME value: "agent-claude-$(NODE_NAME)" - name: TS_KUBE_SECRET value: "agent-claude-ts-$(NODE_NAME)" - name: TS_AUTH_ONCE value: "true" - name: TS_EXTRA_ARGS value: "--ssh --accept-dns=false --advertise-tags=tag:agent" - name: TS_AUTHKEY valueFrom: secretKeyRef: name: agent-claude-ts-authkey key: authkey securityContext: # Kernel-mode tailscale needs a TUN device. NET_ADMIN + the # mounted /dev/net/tun is narrower than the full privileged: # true that repo-recall.yml needs for the WSL2 kernel. Verify # on kai-server's native 6.8 kernel; widen only if needed. capabilities: add: ["NET_ADMIN"] volumeMounts: - name: tun mountPath: /dev/net/tun - name: workspace mountPath: /home/kai/projects resources: requests: cpu: 200m memory: 512Mi limits: cpu: "4" memory: 4Gi volumes: - name: tun hostPath: path: /dev/net/tun type: CharDevice # Per-node persistent workspace. hostPath, not a PVC: the nodes # are durable physical machines, so work survives a pod restart # on the node. Read-write - the agent edits repos. - name: workspace hostPath: path: /home/kai/projects type: Directory --- # ts-authkey minted by terraform/tailscale-devices/ and synced to SSM. apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: agent-claude-ts-authkey namespace: agents spec: refreshInterval: 1h secretStoreRef: name: aws-parameter-store kind: ClusterSecretStore target: name: agent-claude-ts-authkey creationPolicy: Owner template: type: Opaque data: authkey: "{{ .authkey }}" data: - secretKey: authkey remoteRef: key: /coilysiren/agents/claude/ts-authkey ``` ## Sub-tasks - [ ] Build the agent images (`ghcr.io/coilysiren/agent-<runtime>`): base userland, `coily`, the runtime CLI, `tailscaled`, a `kai` UID-1000 user, the drop-privileges entrypoint. Ship via a `docker.yml` GHA workflow like the other repos. - [ ] Mint three reusable-ephemeral `tag:agent` auth keys in `terraform/tailscale-devices/`, sync to SSM under `/coilysiren/agents/<runtime>/ts-authkey`. - [ ] Write `deploy/agents.yml`: namespace, RBAC, three DaemonSets, three ExternalSecrets. - [ ] Tailnet ACL: `tag:agent` reachable over SSH only from Kai's own devices. - [ ] Label `kai-server` `agent-host=true`, apply, verify `ssh kai@agent-claude-kai-server`. - [ ] Add a deploy-notes entry to `docs/k3s-deploy-notes.md` once it works. ## Open decisions - **Always-on vs on-demand.** Three runtimes is three always-on pods per node, some burning tokens on heartbeat or proactive loops. Resident agents (OpenClaw heartbeat) want always-on. Interactive Claude/Codex sessions Kai SSHes into occasionally do not. The draft defaults to always-on DaemonSets as warm shells. Revisit if idle cost bites. - **Where the image build lives.** A new `coilysiren/agent-images` repo, or a directory in `infrastructure`. Leaning new repo to match the per-repo `docker.yml` pattern. - **Workspace mount scope.** The draft mounts all of `/home/kai/projects` read-write. Could scope per-runtime or use read-only plus an overlay if blast radius matters. ## Out of scope - The interactive-vs-resident split is noted, not designed here. - Enabling the WSL2 nodes: #285.

coilysiren added the

label

2026-06-04 08:16:57 +00:00

coilysiren commented

2026-06-17 08:23:01 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:23:01 +00:00