Node scheduling policy: default-deny via taints, allowlist kai-server #12

New issue

Closed

opened 2026-05-23 20:54:27 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:27 +00:00

Owner

Originally filed by @coilysiren on 2026-05-22T13:24:40Z - https://github.com/coilysiren/infrastructure/issues/277

Requirement - The k3s cluster should be allowlist / default-deny at the node level. Right now only kai-server should take scheduled workloads. The desktop-tower nodes (kai-desktop-tower, kai-desktop-tower-wsl) are flaky WSL2 nodes, and a workload landing on one stalls things - the SigNoz clickhouse-operator (#273) scheduled onto kai-desktop-tower-wsl, ran but never reconciled, and blocked the whole ClickHouse bringup.

Current state - Both WSL nodes are now cordoned as an interim measure. Cordon is imperative and can be lost if a node re-registers, so it is not the durable answer.

Durable ask - Taint both WSL nodes NoSchedule. A tainted node rejects any pod that does not carry a matching toleration, so the allowlist becomes "workloads that explicitly tolerate the taint." kai-server stays untainted and is the default home for everything.

Implementation options - Node taints are not cleanly a static kubectl apply manifest in vanilla k8s (Node objects are controller-managed). Realistic shapes:

k3s agent --node-taint flag set in each WSL node's k3s systemd unit. Most durable - survives node re-registration natively. The infra repo already manages systemd ExecStart args.
A re-runnable apply-script or coily verb that runs kubectl taint, plus a checked-in YAML file recording the intended taints as the source of truth. Closest thing to the "Kubernetes manifest YAML file" framing.
A Node manifest applied via server-side apply. Fragile, not recommended.

Recommend the k3s --node-taint flag for durability, with a checked-in taints YAML as the declared intent.

Caveat - Taints are cluster-wide. The node-exporter DaemonSet currently targets the WSL nodes (already failing there with CreateContainerError). Decide whether it gets a toleration or is allowed to stop on those nodes.

_Originally filed by @coilysiren on 2026-05-22T13:24:40Z - [https://github.com/coilysiren/infrastructure/issues/277](https://github.com/coilysiren/infrastructure/issues/277)_ **Requirement** - The k3s cluster should be allowlist / default-deny at the node level. Right now only `kai-server` should take scheduled workloads. The desktop-tower nodes (`kai-desktop-tower`, `kai-desktop-tower-wsl`) are flaky WSL2 nodes, and a workload landing on one stalls things - the SigNoz clickhouse-operator (#273) scheduled onto `kai-desktop-tower-wsl`, ran but never reconciled, and blocked the whole ClickHouse bringup. **Current state** - Both WSL nodes are now `cordon`ed as an interim measure. Cordon is imperative and can be lost if a node re-registers, so it is not the durable answer. **Durable ask** - Taint both WSL nodes `NoSchedule`. A tainted node rejects any pod that does not carry a matching toleration, so the allowlist becomes "workloads that explicitly tolerate the taint." `kai-server` stays untainted and is the default home for everything. **Implementation options** - Node taints are not cleanly a static `kubectl apply` manifest in vanilla k8s (Node objects are controller-managed). Realistic shapes: - k3s agent `--node-taint` flag set in each WSL node's k3s systemd unit. Most durable - survives node re-registration natively. The infra repo already manages systemd ExecStart args. - A re-runnable apply-script or coily verb that runs `kubectl taint`, plus a checked-in YAML file recording the intended taints as the source of truth. Closest thing to the "Kubernetes manifest YAML file" framing. - A Node manifest applied via server-side apply. Fragile, not recommended. Recommend the k3s `--node-taint` flag for durability, with a checked-in taints YAML as the declared intent. **Caveat** - Taints are cluster-wide. The `node-exporter` DaemonSet currently targets the WSL nodes (already failing there with `CreateContainerError`). Decide whether it gets a toleration or is allowed to stop on those nodes.

coilysiren added the

label

2026-06-04 08:16:57 +00:00

coilysiren commented

2026-06-17 08:23:00 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:23:00 +00:00