Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

ENOSPC from inotify: the watcher limit, not the disk

Three per-user limits, one misleading error. Find the consumer, set a permanent sysctl drop-in, and know when polling is the honest answer.

~5 min readLinux
#Linux#inotify#sysctl#Node.js
A directory tree of watch markers filling a gauge to the top beside a sysctl dial turned by a cyan key

Introduction & Quick Summary

Symptom. The disk is nowhere near full, yet a dev server, an editor or a backup job dies with a space error:

code
Error: ENOSPC: System limit for number of file watchers reached, watch '/home/dev/app/node_modules/.cache'
Failed to watch /etc; upper limit on inotify watches reached! tail: cannot watch '/var/log/app': No space left on device

df -h reports plenty of free space, du finds nothing, and the message mentions a file that obviously exists.

Cause. ENOSPC from inotify is not about bytes. The kernel rations watches, instances and queued events per real user ID, and the process hit one of those rations. The default fs.inotify.max_user_watches is often 8,192, which a modern repository with node_modules, .git objects and an editor's file tree exhausts quickly.

Fix. Raise the specific limit, permanently, through sysctl.d:

bash
cat /proc/sys/fs/inotify/max_user_watches echo 'fs.inotify.max_user_watches=524288' | sudo tee /etc/sysctl.d/60-inotify.conf sudo sysctl --system
bash
# current session only, for testing sudo sysctl -w fs.inotify.max_user_watches=524288

That fixes the ceiling. Whether it is the right fix depends on which of the three limits was hit and which process is holding the watches, and the next sections separate those cases.

Core Mechanics, Signatures & Reproducible Setup

inotify keeps a kernel-side object per watched inode, plus a queue of events. Three sysctls govern it, all per user, not per process:

KnobDefault on typical distributionsWhat it bounds
fs.inotify.max_user_watches8,192 to 65,536total inodes watched across all processes of one user
fs.inotify.max_user_instances128number of inotify file descriptors per user
fs.inotify.max_queued_events16,384queued events before the kernel drops or errors

Reproduce it cheaply without filling a disk. Each watch costs roughly 1 KB of unswappable kernel memory on 64-bit, which is why the limit exists:

bash
mkdir -p /tmp/watches && cd /tmp/watches python3 - <<'PY' import os paths = [] for i in range(20000): p = f"/tmp/watches/f{i}" open(p, "w").close() paths.append(p) fds = [] for p in paths: fd = os.inotify_init() os.inotify_add_watch(fd, p, 0x1) # IN_ACCESS fds.append(fd) print("watches:", len(fds)) PY

On a machine with the stock 8,192 limit this raises OSError: [Errno 28] No space left on device partway through, which is exactly the error applications report. The os.inotify_init() call also shows the instance limit: opening one descriptor per file hits max_user_instances long before the watch limit.

Permanent configuration belongs in /etc/sysctl.d/, and the file name decides precedence:

ini
# /etc/sysctl.d/60-inotify.conf fs.inotify.max_user_watches = 524288 fs.inotify.max_user_instances = 1024
bash
sudo sysctl --system # reload every sysctl.d file in order sysctl fs.inotify.max_user_watches

Files in /etc/sysctl.d/ are read before /run/sysctl.d/ and /usr/lib/sysctl.d/, but within the same directory a later name wins. An older /etc/sysctl.d/10-user-watches.conf with a low value is read before 60-inotify.conf, so the higher file should be the one with the later name. When a value refuses to stick, that ordering is usually why.

Key Comparison Table

Error textLimit hitFix
System limit for number of file watchers reachedmax_user_watchesraise watches, or narrow the watched tree
upper limit on inotify watches reachedmax_user_watchessame
Failed to watch ...: too many open filesmax_user_instances or RLIMIT_NOFILEraise the instance limit, or reuse one descriptor
ENOSPC when the watched tree is smallmax_user_watches consumed by other appsfind the consumer
ENOSPC from write() on a real filethe filesystemdf, df -i, this is not inotify
EMFILE: Too many open files from socketsRLIMIT_NOFILEulimit or LimitNOFILE
Where the limit is setScopeSurvives reboot
sysctl -wkernel, immediatelyno
/etc/sysctl.confkernelyes, legacy location
/etc/sysctl.d/*.confkernelyes, preferred
systemd unit LimitNOFILEone service, file descriptorsyes
sysctl in a containernamespaced or blockeddepends on the runtime
/proc/sys/... writekernel, immediatelyno

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. Finding which process holds the watches

Before raising a limit, see whether one runaway process is responsible. PTY-based tools, editors with large workspaces and test runners each hold their own watches, and the per-user pool is shared:

bash
# per-process inotify watch counts, sorted for pid in $(ls /proc | grep -E '^[0-9]+$'); do n=$(grep -c '^inotify' /proc/$pid/fdinfo/* 2>/dev/null) [ "${n:-0}" -gt 0 ] && echo "$n $pid $(cat /proc/$pid/comm 2>/dev/null)" done | sort -rn | head -10
bash
# totals per user, when the above is too slow sudo sysctl fs.inotify.max_user_watches sudo lsof -p "$(pgrep -f 'code|node|webpack' | head -1)" 2>/dev/null | grep inotify | head

If the top entry is a leftover dev server from three branches ago, the honest fix is killing it, not multiplying the limit.

2. Narrowing what the tool watches

Every bundler and editor has an ignore list, and excluding build output and vendored dependencies reduces watches by an order of magnitude:

js
// vite.config.ts export default defineConfig({ server: { watch: { ignored: ["**/node_modules/**", "**/.git/**", "**/dist/**", "**/.venv/**"] }, }, });
js
// webpack.config.js module.exports = { watchOptions: { ignored: /node_modules/ } };
json
// VS Code settings.json { "files.watcherExclude": { "**/node_modules/**": true, "**/.git/objects/**": true, "**/.next/**": true, "**/.venv/**": true } }

A large Python virtual environment inside the project is a frequent culprit, because a watcher walking the tree sees tens of thousands of files that never change.

3. Polling as a deliberate fallback

Containers on Docker Desktop, network shares and some CI filesystems do not propagate inotify events reliably. Polling costs CPU but is deterministic:

bash
# chokidar-based tools CHOKIDAR_USEPOLLING=true npm run dev
js
// vite export default defineConfig({ server: { watch: { usePolling: true, interval: 300 } } });

Use polling on the boundary where events are lost, and keep inotify everywhere else. Turning polling on globally is a common cause of a laptop fan that never stops.

4. Containers: whose kernel limit is it

fs.inotify.max_user_watches is a per-user kernel limit, so a container shares the host's unless the runtime namespaces the setting. Two practical consequences: raising the limit inside a container is usually impossible without privileges, and raising it on the host affects every container of that user.

docker run --rm -it --sysctl fs.inotify.max_user_watches=524288 alpine \ sh -c 'cat /proc/sys/fs/inotify/max_user_watches' # often: sysctl: setting key ... : Read-only file system or permission denied

On a managed runner, the setting belongs in the runner image or a host provisioning step, not in the job script.

5. ENOSPC that is really about inodes

The same error string also comes from a filesystem with no free inodes, which df -h does not show:

bash
df -h / df -i /

If IUse% is 100, the fix is deleting small files (package caches, old logs, container layers), not touching sysctl. Confirming which one you are looking at takes five seconds and prevents a pointless kernel change.

6. Making the change reproducible

Ad-hoc sysctl -w disappears with the machine. Put the settings where provisioning already runs, and assert them in a startup check:

# ansible - name: inotify limits ansible.posix.sysctl: name: "{{ item.key }}" value: "{{ item.value }}" sysctl_file: /etc/sysctl.d/60-inotify.conf reload: true loop: - { key: fs.inotify.max_user_watches, value: "524288" } - { key: fs.inotify.max_user_instances, value: "1024" }
bash
# fail fast when a host is under-provisioned test "$(cat /proc/sys/fs/inotify/max_user_watches)" -ge 262144 || \ echo "inotify watches too low for this workload" >&2

Common Compilation/Runtime Mistakes & Version Gotchas

  • Treating ENOSPC as a disk problem. Check the message: a watched path and "watchers" mean inotify.
  • Only raising max_user_watches. A per-file descriptor design hits max_user_instances first.
  • Setting the value in /etc/sysctl.conf on a systemd host and assuming it is read last. sysctl.d ordering decides.
  • Raising limits instead of excluding node_modules. Watches grow back with the next install.
  • Expecting --sysctl to work in an unprivileged container. The kernel setting is namespaced or read-only.
  • Using polling everywhere. It trades a hard error for constant CPU.
  • Forgetting that the limit is per user. Two projects and an editor share one pool.
  • Leaving an old dev server running. It keeps its watches until the process exits.

When the flood of events is the actual problem, filter the noise with the regex tester; when the change ships as a configuration file, validate it with the YAML validator and check the permissions on the drop-in with the chmod calculator.

Reference: the kernel documentation on inotify, inotify(7), and sysctl.d(5).

Official references

primary documentation

The Linux documentation that defines the behaviour this guide explains.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.