ENOSPC from inotify: the watcher limit, not the disk
Three per-user limits, one misleading error. Find the consumer, set a permanent sysctl drop-in, and know when polling is the honest answer.

Introduction & Quick Summary
Symptom. The disk is nowhere near full, yet a dev server, an editor or a backup job dies with a space error:
Error: ENOSPC: System limit for number of file watchers reached,
watch '/home/dev/app/node_modules/.cache'Failed to watch /etc; upper limit on inotify watches reached!
tail: cannot watch '/var/log/app': No space left on device
df -h reports plenty of free space, du finds nothing, and the message mentions a file that obviously exists.
Cause. ENOSPC from inotify is not about bytes. The kernel rations watches, instances and queued events per real user ID, and the process hit one of those rations. The default fs.inotify.max_user_watches is often 8,192, which a modern repository with node_modules, .git objects and an editor's file tree exhausts quickly.
Fix. Raise the specific limit, permanently, through sysctl.d:
cat /proc/sys/fs/inotify/max_user_watches
echo 'fs.inotify.max_user_watches=524288' | sudo tee /etc/sysctl.d/60-inotify.conf
sudo sysctl --system# current session only, for testing
sudo sysctl -w fs.inotify.max_user_watches=524288That fixes the ceiling. Whether it is the right fix depends on which of the three limits was hit and which process is holding the watches, and the next sections separate those cases.
Core Mechanics, Signatures & Reproducible Setup
inotify keeps a kernel-side object per watched inode, plus a queue of events. Three sysctls govern it, all per user, not per process:
| Knob | Default on typical distributions | What it bounds |
|---|---|---|
fs.inotify.max_user_watches | 8,192 to 65,536 | total inodes watched across all processes of one user |
fs.inotify.max_user_instances | 128 | number of inotify file descriptors per user |
fs.inotify.max_queued_events | 16,384 | queued events before the kernel drops or errors |
Reproduce it cheaply without filling a disk. Each watch costs roughly 1 KB of unswappable kernel memory on 64-bit, which is why the limit exists:
mkdir -p /tmp/watches && cd /tmp/watches
python3 - <<'PY'
import os
paths = []
for i in range(20000):
p = f"/tmp/watches/f{i}"
open(p, "w").close()
paths.append(p)
fds = []
for p in paths:
fd = os.inotify_init()
os.inotify_add_watch(fd, p, 0x1) # IN_ACCESS
fds.append(fd)
print("watches:", len(fds))
PYOn a machine with the stock 8,192 limit this raises OSError: [Errno 28] No space left on device partway through, which is exactly the error applications report. The os.inotify_init() call also shows the instance limit: opening one descriptor per file hits max_user_instances long before the watch limit.
Permanent configuration belongs in /etc/sysctl.d/, and the file name decides precedence:
# /etc/sysctl.d/60-inotify.conf
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 1024sudo sysctl --system # reload every sysctl.d file in order
sysctl fs.inotify.max_user_watchesFiles in /etc/sysctl.d/ are read before /run/sysctl.d/ and /usr/lib/sysctl.d/, but within the same directory a later name wins. An older /etc/sysctl.d/10-user-watches.conf with a low value is read before 60-inotify.conf, so the higher file should be the one with the later name. When a value refuses to stick, that ordering is usually why.
Key Comparison Table
| Error text | Limit hit | Fix |
|---|---|---|
System limit for number of file watchers reached | max_user_watches | raise watches, or narrow the watched tree |
upper limit on inotify watches reached | max_user_watches | same |
Failed to watch ...: too many open files | max_user_instances or RLIMIT_NOFILE | raise the instance limit, or reuse one descriptor |
ENOSPC when the watched tree is small | max_user_watches consumed by other apps | find the consumer |
ENOSPC from write() on a real file | the filesystem | df, df -i, this is not inotify |
EMFILE: Too many open files from sockets | RLIMIT_NOFILE | ulimit or LimitNOFILE |
| Where the limit is set | Scope | Survives reboot |
|---|---|---|
sysctl -w | kernel, immediately | no |
/etc/sysctl.conf | kernel | yes, legacy location |
/etc/sysctl.d/*.conf | kernel | yes, preferred |
systemd unit LimitNOFILE | one service, file descriptors | yes |
sysctl in a container | namespaced or blocked | depends on the runtime |
/proc/sys/... write | kernel, immediately | no |
Practical Edge Cases, Error Fixes & Deep Dive Code Examples
1. Finding which process holds the watches
Before raising a limit, see whether one runaway process is responsible. PTY-based tools, editors with large workspaces and test runners each hold their own watches, and the per-user pool is shared:
# per-process inotify watch counts, sorted
for pid in $(ls /proc | grep -E '^[0-9]+$'); do
n=$(grep -c '^inotify' /proc/$pid/fdinfo/* 2>/dev/null)
[ "${n:-0}" -gt 0 ] && echo "$n $pid $(cat /proc/$pid/comm 2>/dev/null)"
done | sort -rn | head -10# totals per user, when the above is too slow
sudo sysctl fs.inotify.max_user_watches
sudo lsof -p "$(pgrep -f 'code|node|webpack' | head -1)" 2>/dev/null | grep inotify | headIf the top entry is a leftover dev server from three branches ago, the honest fix is killing it, not multiplying the limit.
2. Narrowing what the tool watches
Every bundler and editor has an ignore list, and excluding build output and vendored dependencies reduces watches by an order of magnitude:
// vite.config.ts
export default defineConfig({
server: {
watch: { ignored: ["**/node_modules/**", "**/.git/**", "**/dist/**", "**/.venv/**"] },
},
});// webpack.config.js
module.exports = { watchOptions: { ignored: /node_modules/ } };// VS Code settings.json
{
"files.watcherExclude": {
"**/node_modules/**": true,
"**/.git/objects/**": true,
"**/.next/**": true,
"**/.venv/**": true
}
}A large Python virtual environment inside the project is a frequent culprit, because a watcher walking the tree sees tens of thousands of files that never change.
3. Polling as a deliberate fallback
Containers on Docker Desktop, network shares and some CI filesystems do not propagate inotify events reliably. Polling costs CPU but is deterministic:
# chokidar-based tools
CHOKIDAR_USEPOLLING=true npm run dev// vite
export default defineConfig({ server: { watch: { usePolling: true, interval: 300 } } });Use polling on the boundary where events are lost, and keep inotify everywhere else. Turning polling on globally is a common cause of a laptop fan that never stops.
4. Containers: whose kernel limit is it
fs.inotify.max_user_watches is a per-user kernel limit, so a container shares the host's unless the runtime namespaces the setting. Two practical consequences: raising the limit inside a container is usually impossible without privileges, and raising it on the host affects every container of that user.
docker run --rm -it --sysctl fs.inotify.max_user_watches=524288 alpine \
sh -c 'cat /proc/sys/fs/inotify/max_user_watches'
# often: sysctl: setting key ... : Read-only file system or permission deniedOn a managed runner, the setting belongs in the runner image or a host provisioning step, not in the job script.
5. ENOSPC that is really about inodes
The same error string also comes from a filesystem with no free inodes, which df -h does not show:
df -h /
df -i /If IUse% is 100, the fix is deleting small files (package caches, old logs, container layers), not touching sysctl. Confirming which one you are looking at takes five seconds and prevents a pointless kernel change.
6. Making the change reproducible
Ad-hoc sysctl -w disappears with the machine. Put the settings where provisioning already runs, and assert them in a startup check:
# ansible
- name: inotify limits
ansible.posix.sysctl:
name: "{{ item.key }}"
value: "{{ item.value }}"
sysctl_file: /etc/sysctl.d/60-inotify.conf
reload: true
loop:
- { key: fs.inotify.max_user_watches, value: "524288" }
- { key: fs.inotify.max_user_instances, value: "1024" }# fail fast when a host is under-provisioned
test "$(cat /proc/sys/fs/inotify/max_user_watches)" -ge 262144 || \
echo "inotify watches too low for this workload" >&2Common Compilation/Runtime Mistakes & Version Gotchas
- Treating
ENOSPCas a disk problem. Check the message: a watched path and "watchers" mean inotify. - Only raising
max_user_watches. A per-file descriptor design hitsmax_user_instancesfirst. - Setting the value in
/etc/sysctl.confon a systemd host and assuming it is read last.sysctl.dordering decides. - Raising limits instead of excluding
node_modules. Watches grow back with the next install. - Expecting
--sysctlto work in an unprivileged container. The kernel setting is namespaced or read-only. - Using polling everywhere. It trades a hard error for constant CPU.
- Forgetting that the limit is per user. Two projects and an editor share one pool.
- Leaving an old dev server running. It keeps its watches until the process exits.
When the flood of events is the actual problem, filter the noise with the regex tester; when the change ships as a configuration file, validate it with the YAML validator and check the permissions on the drop-in with the chmod calculator.
Reference: the kernel documentation on inotify, inotify(7), and sysctl.d(5).
Official references
primary documentationThe Linux documentation that defines the behaviour this guide explains.
Try matching browser tools
All toolsSmall utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Linux guides
All Linux guidesnginx 502 Bad Gateway: upstream connection refused
The log line names the address and errno, which is the diagnosis. Fix the host, the start-up order, the buffers, or the timeout that actually fired.
Elasticsearch: max virtual memory areas vm.max_map_count is too low
The bootstrap check reads a kernel setting that is not namespaced. Set it on the host, persist it with sysctl.d, and tune nodes instead of containers.
/usr/bin/env: bash with a carriage return: no such file or directory
A CRLF shebang names an interpreter that cannot exist. Normalise the file and stop the next one with .gitattributes.
apt: Could not get lock /var/lib/dpkg/lock-frontend
The lock names its holder, so read it instead of deleting the file. Wait, time out, or stop unattended-upgrades for the duration.
More in Linux
All Linux guidesThe guides filed next to this one in Linux, in library order.