Linux: Too many open files (EMFILE) and ulimit
EMFILE is a per-process descriptor limit, ENFILE is the kernel's. Read the limit from /proc, raise it in the right layer, and find the leak that made it matter.

Introduction & Quick Summary
Symptom. A service starts refusing work after some hours, and the log has lines that look unrelated to file handling:
2026-11-19T02:14:07Z ERROR accept4() failed (24: Too many open files)
java.io.IOException: Too many open files
bash: /usr/local/bin/report.sh: Too many open filesCause. A process hit RLIMIT_NOFILE, the maximum number of file descriptors it may hold. Every socket, pipe, log file, inotify watch, eventfd and database connection is a descriptor, so a service with a leak, a growing thread count, or a heavy reverse proxy can exhaust a 1,024 limit that was inherited from a shell in 1995. The kernel-wide variant, Too many open files in system, means fs.file-max is exhausted and is a different problem.
Fix. Measure the limit and the count for the specific process, raise the limit where it is actually set (shell, systemd unit, container runtime), and separately fix the leak that made the limit matter. Raising the limit without looking at growth moves the failure to a later hour and hides the bug.
Core Mechanics, Signatures & Reproducible Setup
RLIMIT_NOFILE has a soft limit (enforced) and a hard limit (ceiling for the soft one). The soft limit is inherited from the parent, so the shell you launched from, the unit file, and the container runtime each get a chance to cap it:
ulimit -Sn # soft limit in this shell
ulimit -Hn # hard limit
cat /proc/$(pgrep -f app.jar)/limits | grep "open files"
# Max open files 1024 4096 files
ls /proc/$(pgrep -f app.jar)/fd | wc -l # descriptors currently open
sysctl fs.file-nr # allocated handles max (system-wide)A reproduction that needs no service:
python3 - <<'PY'
import resource, os
print("soft", resource.getrlimit(resource.RLIMIT_NOFILE))
fds = []
try:
for i in range(100000):
fds.append(os.open("/dev/null", os.O_RDONLY))
except OSError as exc:
print(f"failed after {len(fds)} descriptors: {exc}")
finally:
for fd in fds:
os.close(fd)
PYRun it under ulimit -n 256 and it fails at about 250; run it under ulimit -n 65535 and it fails at the hard limit or the memory cost of the descriptor table, which is the point: the number that matters is the one the process was launched with.
Key Comparison Table
Where a limit can come from, and where to change it:
| Layer | Inspect | Change | Applies to |
|---|---|---|---|
| Shell | ulimit -Sn | ulimit -n 65535 | Children launched from that shell only |
limits.conf | cat /etc/security/limits.conf | add * soft nofile 65535 | Login sessions via PAM, not systemd services |
| systemd unit | systemctl show -p LimitNOFILE app | LimitNOFILE=65535 + daemon-reload | That service |
| Docker | docker exec c sh -c 'ulimit -n' | --ulimit nofile=65535:65535, compose ulimits | That container |
| Kubernetes | cat /proc/1/limits in the pod | not settable per pod; inherits the runtime | Every container on the node |
| Kernel | sysctl fs.file-nr | fs.file-max | All processes together |
The four messages that get confused with each other:
| Message | Errno | Meaning |
|---|---|---|
Too many open files | EMFILE | This process reached RLIMIT_NOFILE |
Too many open files in system | ENFILE | The kernel's fs.file-max is exhausted |
No space left on device | ENOSPC | Often inotify watches, not disk |
Cannot allocate memory | ENOMEM | Process or thread limit, not descriptors |
Practical Edge Cases, Error Fixes & Deep Dive Code Examples
1. Read the limit from the process, not the shell
The shell you are debugging from has its own limits, which say nothing about the service:
PID=$(pgrep -f 'gunicorn.*app')
grep "open files" /proc/$PID/limits
sudo ls -l /proc/$PID/fd | awk '{print $11}' | sed 's/[0-9]\+$//' | sort | uniq -c | sort -rn | headThe second command groups open descriptors by target, which immediately separates "10,000 sockets" (a connection problem) from "10,000 log files" (a leak) from "/dev/null repeated" (a pool).
2. Raise it for a systemd service
limits.conf does not apply to units started by systemd, which is the most common reason a "fixed" limit has no effect:
# /etc/systemd/system/api.service.d/limits.conf
[Service]
LimitNOFILE=65535sudo systemctl daemon-reload
sudo systemctl restart api
systemctl show -p LimitNOFILE api # LimitNOFILE=65535LimitNOFILE=infinity works on current systemd and maps to the kernel's maximum, but an explicit number is easier to reason about and to alert on.
3. Docker and Compose
A container inherits the runtime's default, and a ulimit set inside the entrypoint does not survive because the entrypoint is not the process that matters:
services:
proxy:
image: nginx:1.27
ulimits:
nofile:
soft: 65535
hard: 65535docker run --ulimit nofile=65535:65535 nginx:1.27
docker inspect proxy --format '{{.HostConfig.Ulimits}}'Setting only the soft limit leaves the hard limit at the daemon's value, and a process that wants to raise its own soft limit later (ulimit -n inside a shell script) then fails silently at the hard ceiling.
4. Find the leak before raising the number
A count that grows with traffic and never falls is a leak. Two minutes of observation tells you which kind:
PID=$(pgrep -f 'node .*server.js')
for i in $(seq 1 12); do
printf '%s %s\n' "$(date +%H:%M:%S)" "$(ls /proc/$PID/fd | wc -l)"
sleep 5
doneThen look at socket states, because CLOSE_WAIT is a leak on your side (the peer closed, you did not close) while TIME_WAIT is normal churn:
ss -tanp | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn | headIn application code the usual culprits are an unclosed stream on an error path, an HTTP client created per request, and a try without finally around a file handle. In Java, Files.lines() and Files.list() return streams that hold a descriptor until closed; in Python, open() outside a with block; in Node, fs.createReadStream without destroy() on the error path.
5. The inotify trap
File watchers report ENOSPC No space left on device when the watch limit is reached, with gigabytes free on the volume. Build tools, editors and log shippers all consume watches:
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
cat /proc/sys/fs/inotify/max_user_watches# persistent, not just for the current boot
echo "fs.inotify.max_user_watches=524288" | sudo tee /etc/sysctl.d/99-inotify.conf
echo "fs.inotify.max_user_instances=1024" | sudo tee -a /etc/sysctl.d/99-inotify.conf
sudo sysctl --systemRaising this fixes the message; a watcher that creates an instance per request fixes the count, not the limit.
6. Kernel-wide exhaustion
ENFILE means every process together has consumed fs.file-max. Raising a per-process limit here makes things worse by letting one process take a larger share:
sysctl fs.file-max fs.file-nr
# fs.file-max = 9223372036854775807
# fs.file-nr = 48576 0 9223372036854775807On a modern kernel fs.file-max is effectively unlimited and the real ceiling is memory for the file table, so ENFILE points at a node with hundreds of thousands of sockets — a load balancer with no connection cap, or a leak across many workers. Cap connections at the proxy, not at the kernel.
Common Compilation/Runtime Mistakes & Version Gotchas
ulimit -nin a script, then launching the service from another shell. The limit belongs to the process tree; a service started by systemd never sees it.- Editing
limits.conffor a systemd service. PAM applies it to login sessions only. Use a unit drop-in. - Raising only the soft limit. The hard limit is the ceiling; a soft value above it is clamped at exec time.
- Raising the limit as the whole fix. A leak with a 65,535 limit becomes an incident a day later, with a larger blast radius.
systemctl daemon-reloadwithoutrestart. The unit file is re-read, the running process keeps its old limits.- Counting descriptors with
lsofon a container.lsofsees the host namespace unless you enter the container's;ls /proc/<pid>/fdfrom the host is the honest count. - Kubernetes assumptions. There is no per-pod
nofilefield; pods inherit the container runtime's value, so check/proc/1/limitsin the running pod instead of trusting a chart value. - Confusing
EMFILEwith connection pool exhaustion. A pool that refuses checkouts whilels /proc/<pid>/fd | wc -lis far below the limit is not anEMFILEproblem; look at the pool and the database's own connection limit.
Related tools. When the limit lives in a unit's EnvironmentFile or a Compose env block, paste the file into the .env parser to see which LimitNOFILE or ulimits value is actually in effect, and diff the working and failing unit files with the text diff tool — a drop-in that appears earlier alphabetically overrides the one you edited.
Official references
primary documentationThe Linux documentation that defines the behaviour this guide explains.
Try matching browser tools
All toolsSmall utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Linux guides
All Linux guidessystemd status=203/EXEC and 217/USER: Exec and Permission Failures
203 is a failed execve, 217 is a user systemd could not resolve. Both happen before your program runs, and the check that finds each takes one command.
systemd Timers vs Cron: Missing PATH, Overlapping Runs, Time Drift
Cron runs a bare shell with no profile and no locking, and systemd reads the system time zone. The environment, single-instance and schedule fixes for both.
Bash Indexed Arrays, Associative Maps, and Parameter Expansion
Quoting decides whether an array stays a list, and the expansion operators replace most external text tools. Includes set -u, mapfile, namerefs, and the bash 3.2 differences that break scripts on macOS.
Parsing JSON and YAML in Shell Pipelines with jq and yq
jq is a filter language, not a pretty-printer. The operators that do the work, --arg versus string interpolation, in-place YAML edits, and how to tell which yq is installed before writing a command.
More in Linux
All Linux guidesThe guides filed next to this one in Linux, in library order.