Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Linux: Too many open files (EMFILE) and ulimit

EMFILE is a per-process descriptor limit, ENFILE is the kernel's. Read the limit from /proc, raise it in the right layer, and find the leak that made it matter.

~5 min readLinux
#Linux#ulimit#systemd#Troubleshooting
3D diagram of a process box with open file handles filled to the brim and a gauge at maximum, headline TOO MANY FILES

Introduction & Quick Summary

Symptom. A service starts refusing work after some hours, and the log has lines that look unrelated to file handling:

code
2026-11-19T02:14:07Z ERROR accept4() failed (24: Too many open files) java.io.IOException: Too many open files bash: /usr/local/bin/report.sh: Too many open files

Cause. A process hit RLIMIT_NOFILE, the maximum number of file descriptors it may hold. Every socket, pipe, log file, inotify watch, eventfd and database connection is a descriptor, so a service with a leak, a growing thread count, or a heavy reverse proxy can exhaust a 1,024 limit that was inherited from a shell in 1995. The kernel-wide variant, Too many open files in system, means fs.file-max is exhausted and is a different problem.

Fix. Measure the limit and the count for the specific process, raise the limit where it is actually set (shell, systemd unit, container runtime), and separately fix the leak that made the limit matter. Raising the limit without looking at growth moves the failure to a later hour and hides the bug.

Core Mechanics, Signatures & Reproducible Setup

RLIMIT_NOFILE has a soft limit (enforced) and a hard limit (ceiling for the soft one). The soft limit is inherited from the parent, so the shell you launched from, the unit file, and the container runtime each get a chance to cap it:

bash
ulimit -Sn # soft limit in this shell ulimit -Hn # hard limit cat /proc/$(pgrep -f app.jar)/limits | grep "open files" # Max open files 1024 4096 files ls /proc/$(pgrep -f app.jar)/fd | wc -l # descriptors currently open sysctl fs.file-nr # allocated handles max (system-wide)

A reproduction that needs no service:

bash
python3 - <<'PY' import resource, os print("soft", resource.getrlimit(resource.RLIMIT_NOFILE)) fds = [] try: for i in range(100000): fds.append(os.open("/dev/null", os.O_RDONLY)) except OSError as exc: print(f"failed after {len(fds)} descriptors: {exc}") finally: for fd in fds: os.close(fd) PY

Run it under ulimit -n 256 and it fails at about 250; run it under ulimit -n 65535 and it fails at the hard limit or the memory cost of the descriptor table, which is the point: the number that matters is the one the process was launched with.

Key Comparison Table

Where a limit can come from, and where to change it:

LayerInspectChangeApplies to
Shellulimit -Snulimit -n 65535Children launched from that shell only
limits.confcat /etc/security/limits.confadd * soft nofile 65535Login sessions via PAM, not systemd services
systemd unitsystemctl show -p LimitNOFILE appLimitNOFILE=65535 + daemon-reloadThat service
Dockerdocker exec c sh -c 'ulimit -n'--ulimit nofile=65535:65535, compose ulimitsThat container
Kubernetescat /proc/1/limits in the podnot settable per pod; inherits the runtimeEvery container on the node
Kernelsysctl fs.file-nrfs.file-maxAll processes together

The four messages that get confused with each other:

MessageErrnoMeaning
Too many open filesEMFILEThis process reached RLIMIT_NOFILE
Too many open files in systemENFILEThe kernel's fs.file-max is exhausted
No space left on deviceENOSPCOften inotify watches, not disk
Cannot allocate memoryENOMEMProcess or thread limit, not descriptors

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. Read the limit from the process, not the shell

The shell you are debugging from has its own limits, which say nothing about the service:

bash
PID=$(pgrep -f 'gunicorn.*app') grep "open files" /proc/$PID/limits sudo ls -l /proc/$PID/fd | awk '{print $11}' | sed 's/[0-9]\+$//' | sort | uniq -c | sort -rn | head

The second command groups open descriptors by target, which immediately separates "10,000 sockets" (a connection problem) from "10,000 log files" (a leak) from "/dev/null repeated" (a pool).

2. Raise it for a systemd service

limits.conf does not apply to units started by systemd, which is the most common reason a "fixed" limit has no effect:

ini
# /etc/systemd/system/api.service.d/limits.conf [Service] LimitNOFILE=65535
bash
sudo systemctl daemon-reload sudo systemctl restart api systemctl show -p LimitNOFILE api # LimitNOFILE=65535

LimitNOFILE=infinity works on current systemd and maps to the kernel's maximum, but an explicit number is easier to reason about and to alert on.

3. Docker and Compose

A container inherits the runtime's default, and a ulimit set inside the entrypoint does not survive because the entrypoint is not the process that matters:

services: proxy: image: nginx:1.27 ulimits: nofile: soft: 65535 hard: 65535
docker run --ulimit nofile=65535:65535 nginx:1.27 docker inspect proxy --format '{{.HostConfig.Ulimits}}'

Setting only the soft limit leaves the hard limit at the daemon's value, and a process that wants to raise its own soft limit later (ulimit -n inside a shell script) then fails silently at the hard ceiling.

4. Find the leak before raising the number

A count that grows with traffic and never falls is a leak. Two minutes of observation tells you which kind:

bash
PID=$(pgrep -f 'node .*server.js') for i in $(seq 1 12); do printf '%s %s\n' "$(date +%H:%M:%S)" "$(ls /proc/$PID/fd | wc -l)" sleep 5 done

Then look at socket states, because CLOSE_WAIT is a leak on your side (the peer closed, you did not close) while TIME_WAIT is normal churn:

bash
ss -tanp | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn | head

In application code the usual culprits are an unclosed stream on an error path, an HTTP client created per request, and a try without finally around a file handle. In Java, Files.lines() and Files.list() return streams that hold a descriptor until closed; in Python, open() outside a with block; in Node, fs.createReadStream without destroy() on the error path.

5. The inotify trap

File watchers report ENOSPC No space left on device when the watch limit is reached, with gigabytes free on the volume. Build tools, editors and log shippers all consume watches:

bash
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances cat /proc/sys/fs/inotify/max_user_watches
bash
# persistent, not just for the current boot echo "fs.inotify.max_user_watches=524288" | sudo tee /etc/sysctl.d/99-inotify.conf echo "fs.inotify.max_user_instances=1024" | sudo tee -a /etc/sysctl.d/99-inotify.conf sudo sysctl --system

Raising this fixes the message; a watcher that creates an instance per request fixes the count, not the limit.

6. Kernel-wide exhaustion

ENFILE means every process together has consumed fs.file-max. Raising a per-process limit here makes things worse by letting one process take a larger share:

bash
sysctl fs.file-max fs.file-nr # fs.file-max = 9223372036854775807 # fs.file-nr = 48576 0 9223372036854775807

On a modern kernel fs.file-max is effectively unlimited and the real ceiling is memory for the file table, so ENFILE points at a node with hundreds of thousands of sockets — a load balancer with no connection cap, or a leak across many workers. Cap connections at the proxy, not at the kernel.

Common Compilation/Runtime Mistakes & Version Gotchas

  • ulimit -n in a script, then launching the service from another shell. The limit belongs to the process tree; a service started by systemd never sees it.
  • Editing limits.conf for a systemd service. PAM applies it to login sessions only. Use a unit drop-in.
  • Raising only the soft limit. The hard limit is the ceiling; a soft value above it is clamped at exec time.
  • Raising the limit as the whole fix. A leak with a 65,535 limit becomes an incident a day later, with a larger blast radius.
  • systemctl daemon-reload without restart. The unit file is re-read, the running process keeps its old limits.
  • Counting descriptors with lsof on a container. lsof sees the host namespace unless you enter the container's; ls /proc/<pid>/fd from the host is the honest count.
  • Kubernetes assumptions. There is no per-pod nofile field; pods inherit the container runtime's value, so check /proc/1/limits in the running pod instead of trusting a chart value.
  • Confusing EMFILE with connection pool exhaustion. A pool that refuses checkouts while ls /proc/<pid>/fd | wc -l is far below the limit is not an EMFILE problem; look at the pool and the database's own connection limit.

Related tools. When the limit lives in a unit's EnvironmentFile or a Compose env block, paste the file into the .env parser to see which LimitNOFILE or ulimits value is actually in effect, and diff the working and failing unit files with the text diff tool — a drop-in that appears earlier alphabetically overrides the one you edited.

Official references

primary documentation

The Linux documentation that defines the behaviour this guide explains.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.