Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Linux Memory, Swap, the OOM Killer, vmstat, and sysctl Tuning

MemAvailable versus used, the vmstat columns that prove thrashing, how the OOM killer picks its victim, and the sysctl keys that belong in /etc/sysctl.d instead of a one-off command.

~6 min readLinux
#Linux#Performance#Kernel#SysAdmin
3D illustration of a RAM bar overflowing into a swap cylinder with a memory pressure gauge and OOM marker

Quick Summary

Linux memory problems are diagnosed from three places: /proc/meminfo and free for the totals, vmstat for the rate of change, and the kernel log for what was killed when the totals ran out. free -h prints available, the kernel's own estimate of what a new workload could use after reclaiming cache, and that number is the one that matters — used includes page cache, which is doing useful work and will be dropped under pressure.

Swap is the overflow area for anonymous memory. vm.swappiness (0-200, default 60 since 5.8) decides how eagerly the kernel trades file-backed cache for anonymous pages, and the si/so columns in vmstat decide whether you have a swap problem at all: steady nonzero values mean continuous thrashing, while a burst at boot is normal. On SSDs and in containers, small zram or zswap devices often beat a disk swapfile outright.

When memory cannot be reclaimed, the OOM killer picks a victim by oom_score (roughly the process's share of RAM) modified by oom_score_adj (-1000 to 1000). The kernel logs the choice with the numbers attached, which is the only durable proof of why a process disappeared. sysctl is where the policy lives — overcommit mode, min_free_kbytes, panic behaviour — and it belongs in /etc/sysctl.d/*.conf, not in an ad-hoc sysctl -w. This guide covers reading the numbers, sizing swap, controlling the victim selection, and the sysctl keys that change how the kernel behaves at the edge.

Basic Syntax & Core Signatures

bash
# Current state free -h ; free -m -w # -w wide: separates buffers and cache columns cat /proc/meminfo | head -25 vmstat 1 5 # 5 samples, one second apart vmstat -s | head -20 ps -eo pid,rss,pmem,comm --sort=-rss | head -10 # Detail from the kernel grep -E '^(MemTotal|MemFree|MemAvailable|Cached|Buffers|SReclaimable|SwapTotal|SwapFree|Dirty|Committed_AS|CommitLimit):' /proc/meminfo cat /proc/pressure/memory # PSI: some/full avg10 avg60 avg300 # Swap devices swapon --show zramctl # zram devices, if used cat /proc/swaps
bash
# Create and enable a swapfile (ext4/xfs; not supported on btrfs) sudo fallocate -l 4G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=4096 sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile echo '/swapfile none swap sw,pri=10 0 0' | sudo tee -a /etc/fstab # policy sudo sysctl -w vm.swappiness=10 sudo sysctl -w vm.vfs_cache_pressure=50
bash
# OOM victim selection cat /proc/<pid>/oom_score cat /proc/<pid>/oom_score_adj echo -500 | sudo tee /proc/<pid>/oom_score_adj # -1000: never kill choom -n -500 -p <pid> # same, with a check systemctl show -p OOMScoreAdjust app.service # cgroup v2 memory limits (systemd units and containers) systemctl show -p MemoryMax -p MemoryHigh app.service cat /sys/fs/cgroup/system.slice/app.service/memory.max cat /sys/fs/cgroup/system.slice/app.service/memory.events # oom, oom_kill counters cat /sys/fs/cgroup/system.slice/app.service/memory.stat | head
bash
# sysctl: the persistent form sudo sysctl -a | grep '^vm\.' sudo sysctl -w vm.overcommit_memory=1 # temporary, lost at reboot echo 'vm.min_free_kbytes = 131072' | sudo tee /etc/sysctl.d/90-memory.conf sudo sysctl --system # apply all /etc/sysctl.d files

Key Comparison Table

FieldMeaningWatch for
MemTotalUsable RAM after firmware reservationsLower than installed RAM is normal
MemFreeCompletely unusedSmall values are normal and healthy
MemAvailableFree + reclaimable cache (kernel estimate)The number that predicts OOM
CachedPage cacheReclaimable; not a leak
Dirty / WritebackPages waiting for or in I/OGrowth means storage is the bottleneck
Committed_ASAddress space promised by overcommitAbove CommitLimit with mode 2 fails allocations
SwapFreeSwap remainingFalling to zero precedes OOM in thrash cases
oom_score0-1000, memory-share basedNot settable, only readable
vmstat columnMeaningRed flag
rRunnable processesPersistently > CPU count
bBlocked in uninterruptible I/OHigh with high wa
swpdSwap used (KB)Static is fine; growing is not
si / soSwap in/out per secondSustained nonzero = thrashing
waI/O wait percentHigh on a database host is expected in bursts
stTime stolen by the hypervisorNonzero in a VM means a noisy neighbour
sysctl keyDefaultEffect
vm.swappiness60Willingness to swap anon pages (0-200)
vm.vfs_cache_pressure100Tendency to reclaim dentry/inode cache
vm.overcommit_memory00 heuristic, 1 always, 2 never beyond limit
vm.overcommit_ratio50Backs CommitLimit in mode 2
vm.min_free_kbytes~3-6 MB per GBReserved for atomic allocations
vm.panic_on_oom01 reboots instead of killing a process
vm.oom_kill_allocating_task01 kills the allocating process, not the largest
vm.dirty_ratio / dirty_background_ratio20 / 10Cap on dirty page cache

Practical Edge Cases & Deep Dives

1. free says 90% used, and nothing is wrong

bash
$ free -h total used free shared buff/cache available Mem: 31Gi 28Gi 1.2Gi 512Mi 2.1Gi 2.4Gi

used counts the page cache that the kernel will drop the moment a workload needs it. available (2.4 GiB here) is the honest figure, computed from free memory plus the reclaimable portion of cache. The real warning is available approaching zero and si/so nonzero in vmstat; the fake warning is a high used on a healthy file server. /proc/pressure/memory is the modern version of the same signal: some avg10 above a few percent means tasks are stalling on memory reclaim.

2. Sizing swap: what swappiness actually does

bash
# Small emergency swap for a server whose working set fits in RAM sudo sysctl -w vm.swappiness=10 # Container host or laptop: keep some swap interest sudo sysctl -w vm.swappiness=60

swappiness is a bias, not a switch: 0 still swaps to avoid OOM, and 100 makes the kernel reclaim anonymous pages as readily as page cache. For a latency-sensitive service, small swappiness plus a modest swapfile gives the kernel room to evict cold pages before the OOM killer runs; with no swap at all, the only options are reclaim or kill. On systems where swap sits on a slow disk, a zram device is usually better because the "disk" is compressed RAM:

bash
sudo modprobe zram echo 2G | sudo tee /sys/block/zram0/disksize sudo mkswap /dev/zram0 && sudo swapon -p 100 /dev/zram0

3. Reading an OOM kill correctly

text
kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=app.service, mems_allowed=0,global_oom,task_memcg=/system.slice/app.service,task=java,pid=2214,uid=998 kernel: Out of memory: Killed process 2214 (java) total-vm:8112342kB, anon-rss:3933424kB, file-rss:0kB, shmem-rss:0kB, UID:998 pgtables:8100kB oom_score_adj:200

constraint=CONSTRAINT_NONE plus global_oom means the whole machine ran out; a line with constraint=CONSTRAINT_MEMCG means one cgroup hit its own memory.max, which is a limit problem rather than a capacity problem. anon-rss is the memory that actually mattered, oom_score_adj explains why this process rather than another, and task_memcg names the group. A service that dies every few hours with no application log entry is almost always this, and journalctl -k | grep -i oom finds it after the fact.

4. Protecting a critical process, and killing the right one

bash
# Systemd: keep the agent alive, let the batch job die systemctl edit agent.service # [Service] # OOMScoreAdjust=-800 # The batch job: make it the preferred victim systemctl edit batch.service # [Service] # OOMScoreAdjust=500 # MemoryMax=2G

oom_score_adj is the supported lever; oom_score itself is read-only. A value of -1000 excludes a process from the OOM killer entirely, which is dangerous for anything sizeable because the kernel then picks something else instead. vm.oom_kill_allocating_task=1 changes the rule to "kill whoever asked", which is predictable for a single runaway job and wrong for a multi-tenant host. In cgroup v2, memory.high provides throttling before the hard limit: the group is pushed into reclaim and slowed down rather than killed.

5. vmstat versus free: rate beats snapshot

bash
$ vmstat 1 5 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 10240 20132 3120 714288 0 0 3 41 812 1204 9 3 87 1 0 0 4 0 10244 1024 402112 812 1024 980 3102 1102 2201 4 12 61 23 0

The second line is the diagnosis: si/so nonzero with wa at 23% means the machine is swapping and waiting on the device, so the fix is memory or a smaller working set, not more CPU. Snapshots from free cannot show this; a one-minute vmstat 1 60 can, and it costs nothing. si/so from vmstat count pages, so multiply by 4 KB for bytes per second.

6. sysctl values that only apply at boot

bash
# Apply now and persist sudo sysctl -w vm.min_free_kbytes=131072 echo 'vm.min_free_kbytes = 131072' | sudo tee /etc/sysctl.d/90-memory.conf sudo sysctl --system # Check the effective value, not the file sysctl vm.min_free_kbytes cat /proc/sys/vm/min_free_kbytes

Files in /etc/sysctl.d/ are read in lexical order, so a later file overrides an earlier one; a distribution file such as 99-sysctl.conf in /etc/ can quietly win over /etc/sysctl.d/90-memory.conf depending on the loader, which is why sysctl --system output (it prints every applied file) is worth reading once. Some keys are boot-only: vm.overcommit_memory=2 changes allocation behaviour immediately, but vm.nr_hugepages needs contiguous memory that a fragmented system may no longer have, and transparent huge page settings under /sys/kernel/mm/transparent_hugepage/ are not sysctl at all.

Common Runtime & Configuration Mistakes

SymptomCauseFix
"Out of memory" with free showing free RAMOvercommit disabled and CommitLimit reached, or a cgroup limitRead the OOM line's constraint, check memory.max
Process killed with no error in its logOOM killerjournalctl -k | grep -i oom
available near zero, si/so highWorking set exceeds RAMAdd memory, reduce cache, add swap
Disk full during swapfile creationfallocate on btrfs/XFS without supportdd instead
Swapfile re-enabled after reboot failsMissing fstab entry, or wrong permissionschmod 600, fstab line with sw,pri=
swapon: /swapfile: insecure permissions 0644Mode too openchmod 600 /swapfile
swapon: read swap header failedFile not prepared with mkswapmkswap before swapon
OOM keeps killing the same large processHighest oom_score by RSSOOMScoreAdjust, MemoryMax, fix the leak
-1000 protected process and host still diesNothing left to killRemove the protection, add MemoryHigh throttling
sysctl change gone after rebootsysctl -w only/etc/sysctl.d/*.conf + sysctl --system
vm.swappiness=0 still swaps0 is a bias, not a prohibitionAccept it, or remove swap entirely
Container OOM-killed below its limitLimit counts RSS+cache in cgroup v2memory.high, measure memory.current
Memory used grows in Cached and never fallsNormal page cacheCheck MemAvailable and PSI, not used

Memory troubleshooting is a sequence, not a single command: confirm pressure with available and PSI, confirm the trend with vmstat's si/so and wa, confirm the victim and the constraint from the kernel's OOM line, and only then change a limit — vm.swappiness, an OOMScoreAdjust, or a MemoryMax — with the value written to /etc/sysctl.d or the unit file so it survives the next boot.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.