Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Debugging Linux Processes and Kernel Errors with lsof, strace, and dmesg

Deleted files still holding disk space, syscalls that explain a hang, and the kernel's record of an OOM kill. What each tool answers, what it costs, and the permissions that block it.

~5 min readLinux
#Linux#Debugging#Processes
3D illustration of a process box linked by file descriptors to open files, syscall traces and a kernel log panel

Quick Summary

lsof answers "who has this?", strace answers "what is this process doing?", and dmesg answers "what did the kernel say?" They are the three tools that turn a vague production symptom — a full disk that df says is empty, a service that hangs on startup, a process that vanished — into a fact.

lsof enumerates open file descriptors: regular files, directories, sockets, pipes, devices, and memory maps. Its two highest-value uses are finding the process holding a deleted file that is still occupying disk space (lsof +L1), and mapping a listening port to a process (lsof -i :5432) when ss is not enough. It needs root to see other users' descriptors, and -n -P to stay fast by not resolving names and ports.

strace observes system calls through ptrace. Every file that could not be opened, every socket that timed out, every failed connect appears with an error name, which is why it resolves problems no amount of application logging can. The cost is real: a traced process runs several times slower, and a per-call context switch makes a busy server crawl. -f follows children, -e trace= narrows the noise, -y prints the path behind each descriptor, and -c prints a syscall summary.

dmesg reads the kernel's ring buffer, where OOM kills, hardware errors, dropped links, and driver failures land. The lines are the only record of a process killed by the kernel, and -T turns the monotonic timestamps into wall-clock time. This guide covers the option sets that matter, reading each tool's output correctly, and the permission and configuration walls that block them.

Basic Syntax & Core Signatures

bash
# lsof: what is using what lsof -i :8080 # who owns a port lsof -i tcp -sTCP:LISTEN # listening sockets only lsof -i @10.0.0.5:5432 # a specific remote endpoint lsof -p 4242 # everything that PID has open lsof -u deploy # by user lsof -c nginx # by command name, truncated to 9 chars lsof +D /srv/app # recursive directory (slow on big trees) lsof /var/log/app.log # who is writing this file lsof -nP -i :80 # no DNS/service resolution: fast lsof +L1 # files with link count < 1: deleted but held open lsof -t -i :3000 # terse: PIDs only, for scripts lsof -Fpc -i :3000 # machine-readable fields (p=pid, c=command)
bash
# strace: what a process is doing strace -p 4242 # attach (needs ptrace permission) strace -f -p 4242 -e trace=network # follow threads/children, network calls only strace -f -e trace=file -o /tmp/trace.log ./start.sh strace -tt -T -s 256 -e trace=openat,connect ./app strace -c -f -p 4242 # summary: count, error, time per syscall strace -yy -e trace=connect ./app # resolve socket fds to addresses and paths strace -k -e trace=openat ./app # kernel stack trace (if compiled in)

Common -e trace= groups: %file (all path-taking calls), %network, %desc (descriptor operations), %process, %signal, %ipc, %memory, plus individual names such as openat, connect, futex, epoll_wait. Errors print symbolically: ENOENT, EACCES, ECONNREFUSED, ETIMEDOUT, EAGAIN, EMFILE.

bash
# dmesg: what the kernel recorded dmesg -T # human-readable timestamps dmesg -w # follow, like tail -f dmesg -l err,warn # by level: emerg alert crit err warn notice info debug dmesg -f kern,daemon # by facility dmesg -H # pager with relative times dmesg | grep -i -E 'oom|killed process' journalctl -k --since '1 hour ago' -p err # the same buffer via journald

An OOM kill looks like this, and the numbers are the evidence:

text
kernel: node invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0, task_malloc-test,pid=3312,uid=1001 kernel: Out of memory: Killed process 3312 (node) total-vm:2048400kB, anon-rss:1902332kB, file-rss:0kB, shmem-rss:0kB

Key Comparison Table

QuestionToolCommand
Which process holds port 8080?lsoflsof -nP -i :8080
Which process deleted this log but holds it open?lsoflsof +L1 | grep deleted
What file did this program try to open?stracestrace -f -e trace=openat ./app
Why does this request time out?stracestrace -f -e trace=network -T -p PID
Where is the time going in syscalls?stracestrace -c -f -p PID
Was my process killed by the kernel?dmesc / journalctldmesg -T | grep -i 'killed process'
Did the disk or NIC report errors?dmesgdmesg -l err
Is the host under memory pressure?dmesg + vmstatOOM lines, si/so columns
ToolNeeds root?OverheadSurvives process exit
lsofFor other users' fdsLow, but slow on huge trees without -nNo
straceFor other users' processes (or CAP_SYS_PTRACE)High: 2-100x slowdownNo, the data dies with the process
dmesgWith kernel.dmesg_restrict=1, yesNoneYes, until the ring buffer wraps

Practical Edge Cases & Deep Dives

1. The deleted file that fills a disk

bash
$ df -h / /dev/sda1 40G 39G 0G 100% / $ du -sh /var/log 1.2G /var/log $ lsof +L1 | awk '$7 > 1000000 {print $1, $2, $7, $9}' nginx 1187 32212254720 /var/log/nginx/access.log (deleted)

An unlinked file's blocks are freed only when the last descriptor closes, so a log rotated with mv plus rm (instead of logrotate's copytruncate or a SIGHUP reload) keeps the whole file alive in the process. The fix is to restart or reload the holder, or to truncate through the descriptor:

bash
: > /proc/1187/fd/9 # truncates the deleted file the process still holds kill -HUP 1187 # or make the process reopen its log

lsof +L1 is the reliable discovery step; du and df will never agree until the descriptor is closed.

2. Port ownership and the "address already in use" bind

bash
$ ss -tlnp 'sport = :8080' State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=3312,fd=21)) $ lsof -nP -i :8080 node 3312 app 21u IPv6 0x... 0t0 TCP *:8080 (LISTEN)

ss reports the socket and the process through /proc/net; lsof reports the descriptor and its owner and works when the socket is inherited by a child or held through a unix socket. A bind error with a port that ss shows as free usually means a socket in TIME_WAIT (not the same as in use), or a process in another network namespace — check lsns and ss inside the container rather than on the host.

3. Reading strace: ENOENT, EACCES, and the config file that is not there

bash
$ strace -f -e trace=openat,stat,connect -s 128 ./app 2>&1 | grep -v ENOENT | tail -20 openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/app/config.yaml", O_RDONLY) = 3 connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.5")}, 16) = -1 ETIMEDOUT (Connection timed out)

ENOENT on a library or config path is usually benign probing; EACCES is a permission problem; ETIMEDOUT from connect is a network or firewall problem, not an application one; ECONNREFUSED means the host answered and nothing listened. The -y flag adds the path for each descriptor, which turns read(3, ...) into read(3</etc/app/config.yml>, ...) and removes most of the guesswork.

bash
strace -yy -f -e trace=desc,file -p 4242 -o /tmp/t.log grep -E '= -1 E' /tmp/t.log | awk '{print $NF}' | sort | uniq -c | sort -rn

That second command counts failures by error, which usually names the problem in one line.

4. Tracing a service, a container, and a process that will not be traced

bash
# A service: find the main PID, then attach with the service's user systemctl show -p MainPID app.service sudo strace -f -p "$(systemctl show -p MainPID --value app.service)" -e trace=file # A container: trace the host-side PID of the container process docker inspect -f '{{.State.Pid}}' app sudo strace -f -p "$(docker inspect -f '{{.State.Pid}}' app)" -e trace=network # If attach fails: # ptrace: Operation not permitted -> Yama: /proc/sys/kernel/yama/ptrace_scope=1 # (root may still attach; a non-root user cannot attach to a non-child)

ptrace_scope=1 (the default on Ubuntu) restricts ptrace to descendants, so a user cannot attach to an unrelated process; root can, unless a container capability drop removed SYS_PTRACE. Tracing a setuid binary is refused by design. Overhead is proportional to syscall rate: a process doing 200k read calls per second becomes unusable under strace -f, which is why perf trace or bpftrace is the right tool above roughly a few thousand calls per second.

5. dmesg permission, ring buffer, and the timestamps

bash
$ dmesg dmesg: read kernel buffer failed: Operation not permitted $ sysctl kernel.dmesg_restrict # 1 on many hardened images $ sudo dmesg -T | tail -50 $ sudo dmesg -c > /tmp/dmesg.txt # read AND clear the buffer (destructive) $ journalctl -k -b -1 -p err # previous boot's kernel errors, no clear

-T converts the monotonic clock to wall-clock time but can be wrong after suspend or clock changes; the untranslated value is the kernel's monotonic uptime, and -d (delta) or journalctl -k -o short-precise is more reliable for correlating with application logs. Never use dmesg -c on a production host — the buffer is the only record of what the kernel saw, and journalctl persistence is the correct replacement.

6. Turning the three into one investigation

bash
# 1. Kernel: was something killed, or did hardware fail? sudo dmesg -T | grep -iE 'oom|killed process|I/O error|link is (up|down)|hardware error' # 2. Filesystem: is a deleted file holding space? sudo lsof +L1 | awk 'NR==1 || $7 > 1048576' # 3. The process itself: what is it stuck on? sudo strace -f -tt -T -e trace=network,desc -p "$(pgrep -f /opt/app/bin/server)" -o /tmp/app.trace & sleep 5; kill %1 awk '{print $3}' /tmp/app.trace | sort | uniq -c | sort -rn | head

The third snippet names the syscall the process spent its time in — futex (lock contention), epoll_wait (idle, waiting on events), read on a socket (slow client), fsync (slow disk). That single distribution often decides whether the answer is code, configuration, or hardware.

Common Runtime & Configuration Mistakes

SymptomCauseFix
lsof shows nothing for a port that is in useRunning as a non-root user, or another namespacesudo lsof, check the container's namespace
lsof hangsReverse DNS/service lookups on a large fd set-n -P
Disk full but du shows littleDeleted file held openlsof +L1, truncate via /proc/PID/fd/N
strace: ptrace(PTRACE_ATTACH...): Operation not permittedptrace_scope, capabilities, or setuid targetAttach as root, or trace at launch
Service unusably slow while tracedSyscall rate × ptrace overheadNarrow -e trace=, short window, perf trace
Output truncated stringsDefault -s 32-s 256 (and check the actual length limit)
Child processes not tracedMissing -f-f or -ff with -o prefix
dmesg empty or refusedkernel.dmesg_restrict=1sudo dmesg, or journalctl -k
Timestamps look wrong-T after clock changeUse monotonic values, correlate with uptime
Log buffer lost after an incidentdmesg -c cleared it, or ring wrappedConfigure persistent journald
OOM kills attributed to the wrong processOnly the victim is loggedCheck oom_score_adj and the oom-kill: line

These tools are diagnostic instruments, not monitoring: each is meant to be pointed at one process or one symptom for a short time. The durable practice is to read the kernel's record first (it needs no permissions beyond journalctl), establish which process owns the resource, and only then trace the process — because the answer is very often already in a line the kernel wrote before the application had a chance to log anything.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.