Linux Memory, Swap, the OOM Killer, vmstat, and sysctl Tuning
MemAvailable versus used, the vmstat columns that prove thrashing, how the OOM killer picks its victim, and the sysctl keys that belong in /etc/sysctl.d instead of a one-off command.

Quick Summary
Linux memory problems are diagnosed from three places: /proc/meminfo and free for the totals, vmstat for the rate of change, and the kernel log for what was killed when the totals ran out. free -h prints available, the kernel's own estimate of what a new workload could use after reclaiming cache, and that number is the one that matters — used includes page cache, which is doing useful work and will be dropped under pressure.
Swap is the overflow area for anonymous memory. vm.swappiness (0-200, default 60 since 5.8) decides how eagerly the kernel trades file-backed cache for anonymous pages, and the si/so columns in vmstat decide whether you have a swap problem at all: steady nonzero values mean continuous thrashing, while a burst at boot is normal. On SSDs and in containers, small zram or zswap devices often beat a disk swapfile outright.
When memory cannot be reclaimed, the OOM killer picks a victim by oom_score (roughly the process's share of RAM) modified by oom_score_adj (-1000 to 1000). The kernel logs the choice with the numbers attached, which is the only durable proof of why a process disappeared. sysctl is where the policy lives — overcommit mode, min_free_kbytes, panic behaviour — and it belongs in /etc/sysctl.d/*.conf, not in an ad-hoc sysctl -w. This guide covers reading the numbers, sizing swap, controlling the victim selection, and the sysctl keys that change how the kernel behaves at the edge.
Basic Syntax & Core Signatures
# Current state
free -h ; free -m -w # -w wide: separates buffers and cache columns
cat /proc/meminfo | head -25
vmstat 1 5 # 5 samples, one second apart
vmstat -s | head -20
ps -eo pid,rss,pmem,comm --sort=-rss | head -10
# Detail from the kernel
grep -E '^(MemTotal|MemFree|MemAvailable|Cached|Buffers|SReclaimable|SwapTotal|SwapFree|Dirty|Committed_AS|CommitLimit):' /proc/meminfo
cat /proc/pressure/memory # PSI: some/full avg10 avg60 avg300
# Swap devices
swapon --show
zramctl # zram devices, if used
cat /proc/swaps# Create and enable a swapfile (ext4/xfs; not supported on btrfs)
sudo fallocate -l 4G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=4096
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw,pri=10 0 0' | sudo tee -a /etc/fstab
# policy
sudo sysctl -w vm.swappiness=10
sudo sysctl -w vm.vfs_cache_pressure=50# OOM victim selection
cat /proc/<pid>/oom_score
cat /proc/<pid>/oom_score_adj
echo -500 | sudo tee /proc/<pid>/oom_score_adj # -1000: never kill
choom -n -500 -p <pid> # same, with a check
systemctl show -p OOMScoreAdjust app.service
# cgroup v2 memory limits (systemd units and containers)
systemctl show -p MemoryMax -p MemoryHigh app.service
cat /sys/fs/cgroup/system.slice/app.service/memory.max
cat /sys/fs/cgroup/system.slice/app.service/memory.events # oom, oom_kill counters
cat /sys/fs/cgroup/system.slice/app.service/memory.stat | head# sysctl: the persistent form
sudo sysctl -a | grep '^vm\.'
sudo sysctl -w vm.overcommit_memory=1 # temporary, lost at reboot
echo 'vm.min_free_kbytes = 131072' | sudo tee /etc/sysctl.d/90-memory.conf
sudo sysctl --system # apply all /etc/sysctl.d filesKey Comparison Table
| Field | Meaning | Watch for |
|---|---|---|
MemTotal | Usable RAM after firmware reservations | Lower than installed RAM is normal |
MemFree | Completely unused | Small values are normal and healthy |
MemAvailable | Free + reclaimable cache (kernel estimate) | The number that predicts OOM |
Cached | Page cache | Reclaimable; not a leak |
Dirty / Writeback | Pages waiting for or in I/O | Growth means storage is the bottleneck |
Committed_AS | Address space promised by overcommit | Above CommitLimit with mode 2 fails allocations |
SwapFree | Swap remaining | Falling to zero precedes OOM in thrash cases |
oom_score | 0-1000, memory-share based | Not settable, only readable |
| vmstat column | Meaning | Red flag |
|---|---|---|
r | Runnable processes | Persistently > CPU count |
b | Blocked in uninterruptible I/O | High with high wa |
swpd | Swap used (KB) | Static is fine; growing is not |
si / so | Swap in/out per second | Sustained nonzero = thrashing |
wa | I/O wait percent | High on a database host is expected in bursts |
st | Time stolen by the hypervisor | Nonzero in a VM means a noisy neighbour |
| sysctl key | Default | Effect |
|---|---|---|
vm.swappiness | 60 | Willingness to swap anon pages (0-200) |
vm.vfs_cache_pressure | 100 | Tendency to reclaim dentry/inode cache |
vm.overcommit_memory | 0 | 0 heuristic, 1 always, 2 never beyond limit |
vm.overcommit_ratio | 50 | Backs CommitLimit in mode 2 |
vm.min_free_kbytes | ~3-6 MB per GB | Reserved for atomic allocations |
vm.panic_on_oom | 0 | 1 reboots instead of killing a process |
vm.oom_kill_allocating_task | 0 | 1 kills the allocating process, not the largest |
vm.dirty_ratio / dirty_background_ratio | 20 / 10 | Cap on dirty page cache |
Practical Edge Cases & Deep Dives
1. free says 90% used, and nothing is wrong
$ free -h
total used free shared buff/cache available
Mem: 31Gi 28Gi 1.2Gi 512Mi 2.1Gi 2.4Giused counts the page cache that the kernel will drop the moment a workload needs it. available (2.4 GiB here) is the honest figure, computed from free memory plus the reclaimable portion of cache. The real warning is available approaching zero and si/so nonzero in vmstat; the fake warning is a high used on a healthy file server. /proc/pressure/memory is the modern version of the same signal: some avg10 above a few percent means tasks are stalling on memory reclaim.
2. Sizing swap: what swappiness actually does
# Small emergency swap for a server whose working set fits in RAM
sudo sysctl -w vm.swappiness=10
# Container host or laptop: keep some swap interest
sudo sysctl -w vm.swappiness=60swappiness is a bias, not a switch: 0 still swaps to avoid OOM, and 100 makes the kernel reclaim anonymous pages as readily as page cache. For a latency-sensitive service, small swappiness plus a modest swapfile gives the kernel room to evict cold pages before the OOM killer runs; with no swap at all, the only options are reclaim or kill. On systems where swap sits on a slow disk, a zram device is usually better because the "disk" is compressed RAM:
sudo modprobe zram
echo 2G | sudo tee /sys/block/zram0/disksize
sudo mkswap /dev/zram0 && sudo swapon -p 100 /dev/zram03. Reading an OOM kill correctly
kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=app.service,
mems_allowed=0,global_oom,task_memcg=/system.slice/app.service,task=java,pid=2214,uid=998
kernel: Out of memory: Killed process 2214 (java) total-vm:8112342kB, anon-rss:3933424kB,
file-rss:0kB, shmem-rss:0kB, UID:998 pgtables:8100kB oom_score_adj:200constraint=CONSTRAINT_NONE plus global_oom means the whole machine ran out; a line with constraint=CONSTRAINT_MEMCG means one cgroup hit its own memory.max, which is a limit problem rather than a capacity problem. anon-rss is the memory that actually mattered, oom_score_adj explains why this process rather than another, and task_memcg names the group. A service that dies every few hours with no application log entry is almost always this, and journalctl -k | grep -i oom finds it after the fact.
4. Protecting a critical process, and killing the right one
# Systemd: keep the agent alive, let the batch job die
systemctl edit agent.service
# [Service]
# OOMScoreAdjust=-800
# The batch job: make it the preferred victim
systemctl edit batch.service
# [Service]
# OOMScoreAdjust=500
# MemoryMax=2Goom_score_adj is the supported lever; oom_score itself is read-only. A value of -1000 excludes a process from the OOM killer entirely, which is dangerous for anything sizeable because the kernel then picks something else instead. vm.oom_kill_allocating_task=1 changes the rule to "kill whoever asked", which is predictable for a single runaway job and wrong for a multi-tenant host. In cgroup v2, memory.high provides throttling before the hard limit: the group is pushed into reclaim and slowed down rather than killed.
5. vmstat versus free: rate beats snapshot
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 10240 20132 3120 714288 0 0 3 41 812 1204 9 3 87 1 0
0 4 0 10244 1024 402112 812 1024 980 3102 1102 2201 4 12 61 23 0The second line is the diagnosis: si/so nonzero with wa at 23% means the machine is swapping and waiting on the device, so the fix is memory or a smaller working set, not more CPU. Snapshots from free cannot show this; a one-minute vmstat 1 60 can, and it costs nothing. si/so from vmstat count pages, so multiply by 4 KB for bytes per second.
6. sysctl values that only apply at boot
# Apply now and persist
sudo sysctl -w vm.min_free_kbytes=131072
echo 'vm.min_free_kbytes = 131072' | sudo tee /etc/sysctl.d/90-memory.conf
sudo sysctl --system
# Check the effective value, not the file
sysctl vm.min_free_kbytes
cat /proc/sys/vm/min_free_kbytesFiles in /etc/sysctl.d/ are read in lexical order, so a later file overrides an earlier one; a distribution file such as 99-sysctl.conf in /etc/ can quietly win over /etc/sysctl.d/90-memory.conf depending on the loader, which is why sysctl --system output (it prints every applied file) is worth reading once. Some keys are boot-only: vm.overcommit_memory=2 changes allocation behaviour immediately, but vm.nr_hugepages needs contiguous memory that a fragmented system may no longer have, and transparent huge page settings under /sys/kernel/mm/transparent_hugepage/ are not sysctl at all.
Common Runtime & Configuration Mistakes
| Symptom | Cause | Fix |
|---|---|---|
"Out of memory" with free showing free RAM | Overcommit disabled and CommitLimit reached, or a cgroup limit | Read the OOM line's constraint, check memory.max |
| Process killed with no error in its log | OOM killer | journalctl -k | grep -i oom |
available near zero, si/so high | Working set exceeds RAM | Add memory, reduce cache, add swap |
| Disk full during swapfile creation | fallocate on btrfs/XFS without support | dd instead |
| Swapfile re-enabled after reboot fails | Missing fstab entry, or wrong permissions | chmod 600, fstab line with sw,pri= |
swapon: /swapfile: insecure permissions 0644 | Mode too open | chmod 600 /swapfile |
swapon: read swap header failed | File not prepared with mkswap | mkswap before swapon |
| OOM keeps killing the same large process | Highest oom_score by RSS | OOMScoreAdjust, MemoryMax, fix the leak |
-1000 protected process and host still dies | Nothing left to kill | Remove the protection, add MemoryHigh throttling |
| sysctl change gone after reboot | sysctl -w only | /etc/sysctl.d/*.conf + sysctl --system |
vm.swappiness=0 still swaps | 0 is a bias, not a prohibition | Accept it, or remove swap entirely |
| Container OOM-killed below its limit | Limit counts RSS+cache in cgroup v2 | memory.high, measure memory.current |
Memory used grows in Cached and never falls | Normal page cache | Check MemAvailable and PSI, not used |
Memory troubleshooting is a sequence, not a single command: confirm pressure with available and PSI, confirm the trend with vmstat's si/so and wa, confirm the victim and the constraint from the kernel's OOM line, and only then change a limit — vm.swappiness, an OOMScoreAdjust, or a MemoryMax — with the value written to /etc/sysctl.d or the unit file so it survives the next boot.
Try matching browser tools
All toolsSmall utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Linux guides
All Linux guidesBash Indexed Arrays, Associative Maps, and Parameter Expansion
Quoting decides whether an array stays a list, and the expansion operators replace most external text tools. Includes set -u, mapfile, namerefs, and the bash 3.2 differences that break scripts on macOS.
Parsing JSON and YAML in Shell Pipelines with jq and yq
jq is a filter language, not a pretty-printer. The operators that do the work, --arg versus string interpolation, in-place YAML edits, and how to tell which yq is installed before writing a command.
Linux Firewalls: ufw, iptables, and nftables Rules and Persistence
First-match ordering, stateful filtering, and persistence: the three ideas behind every Linux firewall. Plus why Docker-published ports ignore ufw and how DOCKER-USER closes them.
Linux Users, Groups, sudoers, and PAM Access Control
The four account files, the sudoers grammar with aliases and drop-ins, and the PAM stack where logins actually succeed or fail. Includes the flags that lock an account without disabling SSH keys.
More in Linux
All Linux guidesThe guides filed next to this one in Linux, in library order.