Persist raised fs.inotify.max_user_instances on k3s nodes via ansible #462

New issue

Closed

opened 2026-07-03 22:19:09 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-03 22:19:09 +00:00

Member

Problem

kai-server ships the kernel default fs.inotify.max_user_instances = 128, which a k3s node running many pods exhausts. Symptom: apps that create a FileSystemWatcher die at startup with IOException: The configured user limit (128) on the number of inotify instances has been reached. First hit by the eco-gnome .NET app (coilysiren-eco-gnome-app) at WebApplication.CreateBuilder -> config file-watcher; it stayed in progress-deadline failure until the limit was raised. Any watcher-heavy app on the node is exposed.

Nothing in ansible/ manages inotify sysctls today (no sysctl/node-tuning role), and no /etc/sysctl.d/ entry sets it.

Immediate mitigation (already applied, non-persistent)

sudo sysctl -w fs.inotify.max_user_instances=512 on kai-server, then a rollout restart of the app. This does NOT survive reboot and only touches the one node.

Ask

Persist the raised limit via ansible so it survives reboots and applies to every k3s node (fleet-wide), not just kai-server:

Drop a managed /etc/sysctl.d/ file (e.g. 60-inotify.conf) setting fs.inotify.max_user_instances = 512 (and reload with sysctl --system), OR add a small node-tuning role that owns kernel params of this kind.
Value: 512 is a 4x bump; revisit if a node keeps saturating it. fs.inotify.max_user_watches (currently 65536) is fine and not the exhausted knob.
Scope to the k3s node group so WSL/other hosts are unaffected unless they need it.

Origin: unblocking the issue-40 eco-gnome deploy slot (coilyco-gaming/eco-app#40 / coilyco-bridge/deploy).

## Problem kai-server ships the kernel default `fs.inotify.max_user_instances = 128`, which a k3s node running many pods exhausts. Symptom: apps that create a FileSystemWatcher die at startup with `IOException: The configured user limit (128) on the number of inotify instances has been reached`. First hit by the eco-gnome .NET app (coilysiren-eco-gnome-app) at `WebApplication.CreateBuilder` -> config file-watcher; it stayed in progress-deadline failure until the limit was raised. Any watcher-heavy app on the node is exposed. Nothing in ansible/ manages inotify sysctls today (no sysctl/node-tuning role), and no `/etc/sysctl.d/` entry sets it. ## Immediate mitigation (already applied, non-persistent) `sudo sysctl -w fs.inotify.max_user_instances=512` on kai-server, then a rollout restart of the app. This does NOT survive reboot and only touches the one node. ## Ask Persist the raised limit via ansible so it survives reboots and applies to every k3s node (fleet-wide), not just kai-server: - Drop a managed `/etc/sysctl.d/` file (e.g. `60-inotify.conf`) setting `fs.inotify.max_user_instances = 512` (and reload with `sysctl --system`), OR add a small node-tuning role that owns kernel params of this kind. - Value: 512 is a 4x bump; revisit if a node keeps saturating it. `fs.inotify.max_user_watches` (currently 65536) is fine and not the exhausted knob. - Scope to the k3s node group so WSL/other hosts are unaffected unless they need it. Origin: unblocking the issue-40 eco-gnome deploy slot (coilyco-gaming/eco-app#40 / coilyco-bridge/deploy).

coilysiren referenced this issue

2026-07-04 10:38:26 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #470

coilysiren referenced this issue

2026-07-04 10:38:33 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #471

coilyco-ops added the

label

2026-07-04 17:38:26 +00:00

coilyco-ops commented

2026-07-05 03:22:34 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-462 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T03:22:26Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-462` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T03:22:26Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-05 03:31:33 +00:00

feat(ansible): persist fs.inotify.max_user_instances on k3s nodes

coilysiren closed this issue

2026-07-05 03:31:33 +00:00

coilysiren commented

2026-07-05 03:33:11 +00:00

Owner

WARD-OUTCOME: done - node-tuning ansible role persists fs.inotify.max_user_instances=512 on the k3s node group (kai-server), landed on main.

This one was a clean fit. The issue offered two shapes - a bare /etc/sysctl.d drop-in or a small node-tuning role - and the role won easily: it gives a node_tuning_sysctl dict where the next kernel param is a one-line add, and it slots right next to the docker role as the second become: true, kai-server-gated host-provisioning role, so the wiring pattern was already set for me to copy.

The pleasant surprise was ansible.posix.sysctl doing persist + set-running-value + reload in a single idempotent task (and the collection was already in-tree via authorized-keys, so no new dependency). The only thing that fought back was the doc-size cap: docs/ansible.md was sitting at ~3998 of its 4000-char budget, so my one index line blew it over and I had to shave a handful of adjacent bullets to claw back under. Landed it at exactly 4000.

Confidence is high on correctness - the diff is pure YAML + markdown, CI's lint job only pylints *.py (nothing I touched), and pre-commit ran green including a real pylint pass once I pointed uv at a writable python dir. I could not syntax-check against a live converge (no fleet SSH from here) or watch CI directly (the api-token isn't reachable in this container), so those are reasoned-not-observed. One honest rough edge: the k3s-node scope is a inventory_hostname == 'kai-server' guard, not a real k3s inventory group - fine while kai-server is the only node, but worth converting to a group the day a second k3s node joins. Not filing that as its own issue since the role's docs already call it out as a one-line widen.

WARD-OUTCOME: done - node-tuning ansible role persists fs.inotify.max_user_instances=512 on the k3s node group (kai-server), landed on main. This one was a clean fit. The issue offered two shapes - a bare `/etc/sysctl.d` drop-in or a small node-tuning role - and the role won easily: it gives a `node_tuning_sysctl` dict where the next kernel param is a one-line add, and it slots right next to the `docker` role as the second `become: true`, kai-server-gated host-provisioning role, so the wiring pattern was already set for me to copy. The pleasant surprise was `ansible.posix.sysctl` doing persist + set-running-value + reload in a single idempotent task (and the collection was already in-tree via authorized-keys, so no new dependency). The only thing that fought back was the doc-size cap: `docs/ansible.md` was sitting at ~3998 of its 4000-char budget, so my one index line blew it over and I had to shave a handful of adjacent bullets to claw back under. Landed it at exactly 4000. Confidence is high on correctness - the diff is pure YAML + markdown, CI's lint job only pylints `*.py` (nothing I touched), and pre-commit ran green including a real pylint pass once I pointed uv at a writable python dir. I could not syntax-check against a live converge (no fleet SSH from here) or watch CI directly (the api-token isn't reachable in this container), so those are reasoned-not-observed. One honest rough edge: the k3s-node scope is a `inventory_hostname == 'kai-server'` guard, not a real k3s inventory group - fine while kai-server is the only node, but worth converting to a group the day a second k3s node joins. Not filing that as its own issue since the role's docs already call it out as a one-line widen.

coilyco-ops added the

consult

label

2026-07-10 11:58:03 +00:00