Debugging Linux Processes and Kernel Errors with lsof, strace, and dmesg
Deleted files still holding disk space, syscalls that explain a hang, and the kernel's record of an OOM kill. What each tool answers, what it costs, and the permissions that block it.

Quick Summary
lsof answers "who has this?", strace answers "what is this process doing?", and dmesg answers "what did the kernel say?" They are the three tools that turn a vague production symptom — a full disk that df says is empty, a service that hangs on startup, a process that vanished — into a fact.
lsof enumerates open file descriptors: regular files, directories, sockets, pipes, devices, and memory maps. Its two highest-value uses are finding the process holding a deleted file that is still occupying disk space (lsof +L1), and mapping a listening port to a process (lsof -i :5432) when ss is not enough. It needs root to see other users' descriptors, and -n -P to stay fast by not resolving names and ports.
strace observes system calls through ptrace. Every file that could not be opened, every socket that timed out, every failed connect appears with an error name, which is why it resolves problems no amount of application logging can. The cost is real: a traced process runs several times slower, and a per-call context switch makes a busy server crawl. -f follows children, -e trace= narrows the noise, -y prints the path behind each descriptor, and -c prints a syscall summary.
dmesg reads the kernel's ring buffer, where OOM kills, hardware errors, dropped links, and driver failures land. The lines are the only record of a process killed by the kernel, and -T turns the monotonic timestamps into wall-clock time. This guide covers the option sets that matter, reading each tool's output correctly, and the permission and configuration walls that block them.
Basic Syntax & Core Signatures
# lsof: what is using what
lsof -i :8080 # who owns a port
lsof -i tcp -sTCP:LISTEN # listening sockets only
lsof -i @10.0.0.5:5432 # a specific remote endpoint
lsof -p 4242 # everything that PID has open
lsof -u deploy # by user
lsof -c nginx # by command name, truncated to 9 chars
lsof +D /srv/app # recursive directory (slow on big trees)
lsof /var/log/app.log # who is writing this file
lsof -nP -i :80 # no DNS/service resolution: fast
lsof +L1 # files with link count < 1: deleted but held open
lsof -t -i :3000 # terse: PIDs only, for scripts
lsof -Fpc -i :3000 # machine-readable fields (p=pid, c=command)# strace: what a process is doing
strace -p 4242 # attach (needs ptrace permission)
strace -f -p 4242 -e trace=network # follow threads/children, network calls only
strace -f -e trace=file -o /tmp/trace.log ./start.sh
strace -tt -T -s 256 -e trace=openat,connect ./app
strace -c -f -p 4242 # summary: count, error, time per syscall
strace -yy -e trace=connect ./app # resolve socket fds to addresses and paths
strace -k -e trace=openat ./app # kernel stack trace (if compiled in)Common -e trace= groups: %file (all path-taking calls), %network, %desc (descriptor operations), %process, %signal, %ipc, %memory, plus individual names such as openat, connect, futex, epoll_wait. Errors print symbolically: ENOENT, EACCES, ECONNREFUSED, ETIMEDOUT, EAGAIN, EMFILE.
# dmesg: what the kernel recorded
dmesg -T # human-readable timestamps
dmesg -w # follow, like tail -f
dmesg -l err,warn # by level: emerg alert crit err warn notice info debug
dmesg -f kern,daemon # by facility
dmesg -H # pager with relative times
dmesg | grep -i -E 'oom|killed process'
journalctl -k --since '1 hour ago' -p err # the same buffer via journaldAn OOM kill looks like this, and the numbers are the evidence:
kernel: node invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,
task_malloc-test,pid=3312,uid=1001
kernel: Out of memory: Killed process 3312 (node)
total-vm:2048400kB, anon-rss:1902332kB, file-rss:0kB, shmem-rss:0kBKey Comparison Table
| Question | Tool | Command |
|---|---|---|
| Which process holds port 8080? | lsof | lsof -nP -i :8080 |
| Which process deleted this log but holds it open? | lsof | lsof +L1 | grep deleted |
| What file did this program try to open? | strace | strace -f -e trace=openat ./app |
| Why does this request time out? | strace | strace -f -e trace=network -T -p PID |
| Where is the time going in syscalls? | strace | strace -c -f -p PID |
| Was my process killed by the kernel? | dmesc / journalctl | dmesg -T | grep -i 'killed process' |
| Did the disk or NIC report errors? | dmesg | dmesg -l err |
| Is the host under memory pressure? | dmesg + vmstat | OOM lines, si/so columns |
| Tool | Needs root? | Overhead | Survives process exit |
|---|---|---|---|
lsof | For other users' fds | Low, but slow on huge trees without -n | No |
strace | For other users' processes (or CAP_SYS_PTRACE) | High: 2-100x slowdown | No, the data dies with the process |
dmesg | With kernel.dmesg_restrict=1, yes | None | Yes, until the ring buffer wraps |
Practical Edge Cases & Deep Dives
1. The deleted file that fills a disk
$ df -h /
/dev/sda1 40G 39G 0G 100% /
$ du -sh /var/log
1.2G /var/log
$ lsof +L1 | awk '$7 > 1000000 {print $1, $2, $7, $9}'
nginx 1187 32212254720 /var/log/nginx/access.log (deleted)An unlinked file's blocks are freed only when the last descriptor closes, so a log rotated with mv plus rm (instead of logrotate's copytruncate or a SIGHUP reload) keeps the whole file alive in the process. The fix is to restart or reload the holder, or to truncate through the descriptor:
: > /proc/1187/fd/9 # truncates the deleted file the process still holds
kill -HUP 1187 # or make the process reopen its loglsof +L1 is the reliable discovery step; du and df will never agree until the descriptor is closed.
2. Port ownership and the "address already in use" bind
$ ss -tlnp 'sport = :8080'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=3312,fd=21))
$ lsof -nP -i :8080
node 3312 app 21u IPv6 0x... 0t0 TCP *:8080 (LISTEN)ss reports the socket and the process through /proc/net; lsof reports the descriptor and its owner and works when the socket is inherited by a child or held through a unix socket. A bind error with a port that ss shows as free usually means a socket in TIME_WAIT (not the same as in use), or a process in another network namespace — check lsns and ss inside the container rather than on the host.
3. Reading strace: ENOENT, EACCES, and the config file that is not there
$ strace -f -e trace=openat,stat,connect -s 128 ./app 2>&1 | grep -v ENOENT | tail -20
openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/app/config.yaml", O_RDONLY) = 3
connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.5")}, 16) = -1 ETIMEDOUT (Connection timed out)ENOENT on a library or config path is usually benign probing; EACCES is a permission problem; ETIMEDOUT from connect is a network or firewall problem, not an application one; ECONNREFUSED means the host answered and nothing listened. The -y flag adds the path for each descriptor, which turns read(3, ...) into read(3</etc/app/config.yml>, ...) and removes most of the guesswork.
strace -yy -f -e trace=desc,file -p 4242 -o /tmp/t.log
grep -E '= -1 E' /tmp/t.log | awk '{print $NF}' | sort | uniq -c | sort -rnThat second command counts failures by error, which usually names the problem in one line.
4. Tracing a service, a container, and a process that will not be traced
# A service: find the main PID, then attach with the service's user
systemctl show -p MainPID app.service
sudo strace -f -p "$(systemctl show -p MainPID --value app.service)" -e trace=file
# A container: trace the host-side PID of the container process
docker inspect -f '{{.State.Pid}}' app
sudo strace -f -p "$(docker inspect -f '{{.State.Pid}}' app)" -e trace=network
# If attach fails:
# ptrace: Operation not permitted -> Yama: /proc/sys/kernel/yama/ptrace_scope=1
# (root may still attach; a non-root user cannot attach to a non-child)ptrace_scope=1 (the default on Ubuntu) restricts ptrace to descendants, so a user cannot attach to an unrelated process; root can, unless a container capability drop removed SYS_PTRACE. Tracing a setuid binary is refused by design. Overhead is proportional to syscall rate: a process doing 200k read calls per second becomes unusable under strace -f, which is why perf trace or bpftrace is the right tool above roughly a few thousand calls per second.
5. dmesg permission, ring buffer, and the timestamps
$ dmesg
dmesg: read kernel buffer failed: Operation not permitted
$ sysctl kernel.dmesg_restrict # 1 on many hardened images
$ sudo dmesg -T | tail -50
$ sudo dmesg -c > /tmp/dmesg.txt # read AND clear the buffer (destructive)
$ journalctl -k -b -1 -p err # previous boot's kernel errors, no clear-T converts the monotonic clock to wall-clock time but can be wrong after suspend or clock changes; the untranslated value is the kernel's monotonic uptime, and -d (delta) or journalctl -k -o short-precise is more reliable for correlating with application logs. Never use dmesg -c on a production host — the buffer is the only record of what the kernel saw, and journalctl persistence is the correct replacement.
6. Turning the three into one investigation
# 1. Kernel: was something killed, or did hardware fail?
sudo dmesg -T | grep -iE 'oom|killed process|I/O error|link is (up|down)|hardware error'
# 2. Filesystem: is a deleted file holding space?
sudo lsof +L1 | awk 'NR==1 || $7 > 1048576'
# 3. The process itself: what is it stuck on?
sudo strace -f -tt -T -e trace=network,desc -p "$(pgrep -f /opt/app/bin/server)" -o /tmp/app.trace &
sleep 5; kill %1
awk '{print $3}' /tmp/app.trace | sort | uniq -c | sort -rn | headThe third snippet names the syscall the process spent its time in — futex (lock contention), epoll_wait (idle, waiting on events), read on a socket (slow client), fsync (slow disk). That single distribution often decides whether the answer is code, configuration, or hardware.
Common Runtime & Configuration Mistakes
| Symptom | Cause | Fix |
|---|---|---|
lsof shows nothing for a port that is in use | Running as a non-root user, or another namespace | sudo lsof, check the container's namespace |
lsof hangs | Reverse DNS/service lookups on a large fd set | -n -P |
Disk full but du shows little | Deleted file held open | lsof +L1, truncate via /proc/PID/fd/N |
strace: ptrace(PTRACE_ATTACH...): Operation not permitted | ptrace_scope, capabilities, or setuid target | Attach as root, or trace at launch |
| Service unusably slow while traced | Syscall rate × ptrace overhead | Narrow -e trace=, short window, perf trace |
| Output truncated strings | Default -s 32 | -s 256 (and check the actual length limit) |
| Child processes not traced | Missing -f | -f or -ff with -o prefix |
dmesg empty or refused | kernel.dmesg_restrict=1 | sudo dmesg, or journalctl -k |
| Timestamps look wrong | -T after clock change | Use monotonic values, correlate with uptime |
| Log buffer lost after an incident | dmesg -c cleared it, or ring wrapped | Configure persistent journald |
| OOM kills attributed to the wrong process | Only the victim is logged | Check oom_score_adj and the oom-kill: line |
These tools are diagnostic instruments, not monitoring: each is meant to be pointed at one process or one symptom for a short time. The durable practice is to read the kernel's record first (it needs no permissions beyond journalctl), establish which process owns the resource, and only then trace the process — because the answer is very often already in a line the kernel wrote before the application had a chance to log anything.
Try matching browser tools
All toolsSmall utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Linux guides
All Linux guidesBash Indexed Arrays, Associative Maps, and Parameter Expansion
Quoting decides whether an array stays a list, and the expansion operators replace most external text tools. Includes set -u, mapfile, namerefs, and the bash 3.2 differences that break scripts on macOS.
Parsing JSON and YAML in Shell Pipelines with jq and yq
jq is a filter language, not a pretty-printer. The operators that do the work, --arg versus string interpolation, in-place YAML edits, and how to tell which yq is installed before writing a command.
Linux Memory, Swap, the OOM Killer, vmstat, and sysctl Tuning
MemAvailable versus used, the vmstat columns that prove thrashing, how the OOM killer picks its victim, and the sysctl keys that belong in /etc/sysctl.d instead of a one-off command.
Linux Firewalls: ufw, iptables, and nftables Rules and Persistence
First-match ordering, stateful filtering, and persistence: the three ideas behind every Linux firewall. Plus why Docker-published ports ignore ufw and how DOCKER-USER closes them.
More in Linux
All Linux guidesThe guides filed next to this one in Linux, in library order.