kai-server: inotify exhaustion - Failed to allocate directory watch: Too many open files #196

New issue

Closed

opened 2026-06-04 01:37:57 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-04 01:37:57 +00:00

Owner

Symptom

During make claude-remote-control-install on kai-server (2026-06-03), systemctl daemon-reload printed:

Failed to allocate directory watch: Too many open files
Failed to allocate directory watch: Too many open files
Failed to allocate directory watch: Too many open files

The install still completed and the units loaded, so this did not block that work. It is a standalone host-resource problem surfaced by the reload.

Why it matters

Too many open files from systemd's directory-watch allocation means kai-server has exhausted its inotify budget (fs.inotify.max_user_instances, possibly also max_user_watches). When that budget is pegged it degrades anything that relies on file watches host-wide: systemd path units, file watchers, editors, and dev tooling can silently miss events or fail to start watching. On the ops host that runs the homelab this is worth fixing rather than ignoring.

Diagnostics

# current limits
sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches

# how many inotify instances are currently held, and by whom
for p in /proc/*/fd/*; do readlink "$p" 2>/dev/null; done | grep -c inotify
sudo lsof 2>/dev/null | grep -c inotify

The default max_user_instances of 128 is easy to exhaust once several long-lived watchers are running.

Likely fix

Raise the limits durably via a drop-in, then reload:

# /etc/sysctl.d/60-inotify.conf
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 524288

sudo sysctl --system

Pick the final values after the diagnostics show what is actually consuming instances. If a single runaway process is leaking watches, fixing that is the real fix rather than only raising the ceiling.

Out of scope

This does not affect the claude-remote-control daemon recovery shipped the same day. Filed separately so it is tracked on its own.

## Symptom During `make claude-remote-control-install` on kai-server (2026-06-03), `systemctl daemon-reload` printed: ``` Failed to allocate directory watch: Too many open files Failed to allocate directory watch: Too many open files Failed to allocate directory watch: Too many open files ``` The install still completed and the units loaded, so this did not block that work. It is a standalone host-resource problem surfaced by the reload. ## Why it matters `Too many open files` from systemd's directory-watch allocation means kai-server has exhausted its inotify budget (`fs.inotify.max_user_instances`, possibly also `max_user_watches`). When that budget is pegged it degrades anything that relies on file watches host-wide: systemd path units, file watchers, editors, and dev tooling can silently miss events or fail to start watching. On the ops host that runs the homelab this is worth fixing rather than ignoring. ## Diagnostics ```bash # current limits sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches # how many inotify instances are currently held, and by whom for p in /proc/*/fd/*; do readlink "$p" 2>/dev/null; done | grep -c inotify sudo lsof 2>/dev/null | grep -c inotify ``` The default `max_user_instances` of 128 is easy to exhaust once several long-lived watchers are running. ## Likely fix Raise the limits durably via a drop-in, then reload: ```bash # /etc/sysctl.d/60-inotify.conf fs.inotify.max_user_instances = 1024 fs.inotify.max_user_watches = 524288 ``` ```bash sudo sysctl --system ``` Pick the final values after the diagnostics show what is actually consuming instances. If a single runaway process is leaking watches, fixing that is the real fix rather than only raising the ceiling. ## Out of scope This does not affect the claude-remote-control daemon recovery shipped the same day. Filed separately so it is tracked on its own.

coilysiren added the

priority

label

2026-06-17 07:04:44 +00:00

coilysiren commented

2026-06-17 08:22:39 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:22:39 +00:00