Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Linux Process Management: ps, lsof, Signals (SIGTERM vs SIGKILL), and nice

Process states that explain why kill -9 does nothing, the signal table, port ownership with ss and lsof, and how niceness actually affects the scheduler.

~5 min readLinux
#Linux#Processes#SysAdmin
3D illustration of a Linux process tree prioritized by nice scheduler weights and controlled by SIGTERM and SIGKILL signals

Quick Summary

A Linux process is identified by a PID, owned by a user, and controlled by signals. ps shows what is running, lsof shows what it has open, and kill sends it a signal. The three commands answer the three questions that come up in every incident: what is eating the CPU, what is holding port 8080, and how do I stop it without corrupting its data.

Signals are not all the same kind of thing. SIGTERM (15) is a request: the process can catch it, flush buffers, close connections, and exit. SIGKILL (9) is not a request: the kernel terminates the process immediately, the handler never runs, and any in-flight write is lost. SIGHUP (1) conventionally means "reload your configuration" for daemons. SIGINT (2) is what Ctrl-C sends.

nice is the scheduling counterpart. Every process has a niceness from -20 (most CPU) to 19 (least), default 0, and only root can lower it. Backups, indexing, and transcoding belong at positive niceness so an interactive session keeps its responsiveness; renice changes the value of a process that is already running.

Basic Syntax & Signature

bash
# ps: snapshots ps aux # BSD syntax: user, pid, %cpu, %mem, command ps -ef # POSIX syntax: uid, pid, ppid, cmd ps -eo pid,ppid,stat,pcpu,pmem,etime,cmd --sort=-pcpu | head -15 ps auxf # process tree ps -p 1234 -o pid,ppid,stat,wchan:30,cmd pgrep -af node # PIDs and command lines matching a name pgrep -u www-data -f php-fpm # lsof: open files and sockets lsof -i :8080 # who is listening on 8080 lsof -p 1234 # everything that PID has open lsof +D /var/log/app # processes with files open under a directory lsof -u deploy -a -i # sockets owned by a user lsof /var/lib/dpkg/lock # what is holding a lock file # Signals kill -l # list signal names kill 1234 # SIGTERM kill -15 1234 # same, explicit kill -HUP 1234 # reload kill -9 1234 # SIGKILL, last resort kill -TERM -- -1234 # whole process group pkill -f 'node server.js' # by command line pkill -u deploy -9 # every process of a user # Priorities nice -n 10 tar czf backup.tgz /srv/data renice 15 -p 4321 ionice -c3 -p 4321 # idle I/O class, on top of nice

SIGKILL cannot be caught, blocked, or ignored, which is why a process in uninterruptible sleep (D state) does not die even with kill -9 — it is waiting in the kernel for I/O, and it will exit when the I/O completes or fails.

Key Comparison Table

SignalNumberCatchableDefault actionTypical use
SIGHUP1YesTerminateReload configuration
SIGINT2YesTerminateCtrl-C
SIGQUIT3YesCore dumpCtrl-\ , debugging
SIGKILL9NoTerminateForce stop
SIGSEGV11NoCore dumpMemory fault
SIGTERM15YesTerminateNormal stop
SIGSTOP19NoStopPause a process
SIGCONT18YesContinueResume a paused process
SIGCHLD17YesIgnoreChild state change
SIGPIPE13YesTerminateWrite to a closed pipe

Practical Edge Cases & Deep Dives

1. Reading ps state codes

bash
ps -eo pid,stat,etime,cmd | grep -E '^ *[0-9]+ +(D|Z)'
CodeMeaningConsequence
RRunning or runnableUsing or waiting for CPU
SInterruptible sleepWaiting for an event; normal
DUninterruptible sleepWaiting on I/O; SIGKILL does not take effect
ZZombieExited, parent has not reaped it
TStoppedSIGSTOP or a debugger
IIdle kernel threadNot a user process

A Z process consumes no memory beyond its exit record and cannot be killed; only its parent can reap it, by calling wait(). Killing the parent hands the child to PID 1, which reaps it. A growing count of zombies means a process is forking without waiting, which is a bug in that process, not in ps.

2. Finding what holds a port

bash
# Three ways, in order of preference ss -ltnp 'sport = :8080' lsof -nP -iTCP:8080 -sTCP:LISTEN fuser -n tcp 8080 # Then inspect before killing ps -o pid,ppid,user,cmd -p $(ss -ltnp 'sport = :8080' | grep -oP 'pid=\K[0-9]+' | head -1)

ss reads the kernel's socket table directly and is the fast path; lsof resolves more context but is slower on a busy host. The -nP flags matter on lsof: without them, DNS lookups and service-name resolution turn a millisecond answer into a multi-second one.

3. Killing a process tree, not just a PID

bash
# nginx-style master/worker: TERM the master and let it drain workers kill -TERM "$(cat /run/api.pid)" # A supervisor that ignores TERM: signal the whole group kill -TERM -- -1234 sleep 5 kill -KILL -- -1234 # Everything matching a pattern, safely previewed first pgrep -af 'celery worker' pkill -TERM -f 'celery worker'

kill -- -PGID requires the negative sign and a group id, not a single PID's id. Getting it wrong signals your own shell's group. The preview step matters: pkill -f matches the pattern anywhere in the command line, including the pkill invocation itself on some shells and any editor that has the string open.

4. A shutdown handler that actually drains

python
# Python: SIGTERM triggers a graceful stop, SIGKILL cannot be handled. import signal, sys, time def handler(signum, frame): print("draining connections", flush=True) time.sleep(2) # close pools, finish in-flight requests sys.exit(0) signal.signal(signal.SIGTERM, handler) signal.signal(signal.SIGINT, handler)
bash
kill -TERM "$PID" for i in $(seq 1 30); do kill -0 "$PID" 2>/dev/null || break; sleep 1; done kill -0 "$PID" 2>/dev/null && kill -KILL "$PID"

kill -0 tests whether a process exists without sending a signal, which is the correct polling primitive. Sending SIGKILL first skips the handler entirely: databases left with unclean shutdowns, temp files left behind, and connection pools closed mid-transaction.

5. nice, renice, and what they do not do

bash
# Start a CPU-heavy job below interactive work nice -n 19 tar -cf /backup/data.tar /srv/data # Lower the priority of a running process (positive only, as a normal user) renice 10 -p 4321 renice 10 -g 1000 # by process group renice 10 -u backup # all of a user's processes # Add an I/O class on top; CPU niceness does not throttle disk ionice -c 2 -n 7 -p 4321

Niceness is a hint to the scheduler, not a limit. On a machine with idle cores, a niced process still uses every CPU it can get, and renice does nothing for a process blocked on disk until ionice is applied. Cgroups (systemd's CPUQuota=, IOWeight=) are the mechanism that enforces a ceiling.

6. When the OOM killer is the cause

bash
dmesg -T | grep -i -E 'killed process|out of memory' journalctl -k --since "1 hour ago" | grep -i oom ps -eo pid,pmem,rss,cmd --sort=-rss | head -10 cat /proc/meminfo | grep -E 'MemTotal|MemAvailable|SwapTotal'

An OOMKilled service that stops without an error in its own log was killed by the kernel, not by systemd. The evidence is in the kernel ring buffer, and the useful follow-up is RSS per process rather than %MEM, which is a share of total RAM and misleading on machines with a large page cache.

7. Mapping a PID back to its unit, container, and cwd

bash
# Which systemd unit owns this process cat /proc/1234/cgroup systemctl status 1234 # Where it is and what it was launched as tr '\0' ' ' < /proc/1234/cmdline; echo readlink /proc/1234/cwd readlink /proc/1234/exe # Its open descriptors, with sizes for the growing ones lsof -p 1234 | sort -k7 -n | tail -10 # Its threads, for a process that is stuck at 100% CPU ps -L -o pid,tid,pcpu,stat,comm -p 1234 | sort -k3 -rn | head

/proc/<pid>/cgroup is the fastest way to tell whether a process belongs to a systemd unit, a container runtime, or your own shell — which then decides who is supposed to restart it. ps -L breaks a process into threads, and the thread with the highest CPU is what a profiler or gdb -p should attach to.

Common Compilation/Runtime Mistakes

SymptomCauseFix
kill -9 does not stop the processD state waiting on I/OFix the device or reboot
Service restarts with no log entrySIGKILL from OOM or systemdCheck dmesg / journalctl -k
Zombie processes accumulateParent never calls wait()Fix the parent; killing the zombie does nothing
Port still bound after stopChild process survivedSignal the process group
pkill -f kills the wrong thingPattern matched the callerPreview with pgrep -af
%CPU looks impossible (300%)Per-core percentageDivide by core count
lsof hangs for secondsDNS and service-name lookupslsof -nP
Nice value has no effectI/O bound, or idle coresUse ionice or cgroup limits
renice deniedLowering below the current value as non-rootRun as root, or raise instead
Process tree half-killedOnly the parent was signalledkill -- -PGID

The working order for any "it will not stop" problem is: identify the PID and its state, send SIGTERM, wait with kill -0, then escalate to SIGKILL on the group. Escalating first is the one habit that turns a routine restart into an outage, because whatever the process was writing at that moment never reaches disk.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.