Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Requests vs Limits: CPU Throttling, QoS Classes, and OOMKilled

CPU over its limit is throttled and memory over its limit is killed. How that asymmetry, the requests-to-limits ratio, and the QoS class decide latency, eviction order, and which pods go Pending.

~6 min readKubernetes
#Kubernetes#Resources#Performance
3D illustration of a container with a CPU quota valve and memory limit wall beside QoS class badges

Quick Summary

requests is what the scheduler reserves for a container; limits is what the kernel enforces at runtime. They are different mechanisms with different consequences: the scheduler sums requests to decide whether the pod fits on a node, and the cgroup enforces limits afterwards. A container with a CPU limit is throttled by the CFS (Completely Fair Scheduler) quota when it exceeds it — it is never killed. A container with a memory limit that exceeds it is killed with OOMKilled (exit code 137). That asymmetry is the whole reason CPU limits are contested and memory limits are not.

The ratio between requests and limits also defines the pod's QoS class. Guaranteed requires every container to have CPU and memory requests equal to limits; BestEffort requires none of them; anything in between is Burstable. QoS decides the eviction order under node pressure — BestEffort pods go first, then Burstable pods exceeding their requests, and Guaranteed last — which is why a critical service should be Guaranteed or at least have accurate requests.

Getting these numbers right is empirical. Requests too high cause Pending pods on a node with plenty of unused capacity (because "used" is measured against requests); memory requests too low let a node overcommit until the OOM killer picks victims. This guide covers the fields, the enforcement mechanics, QoS classes and eviction, LimitRange defaults and quotas, and the measurement loop — kubectl top, CFS throttling metrics, OOM events — that turns guesses into values.

Basic Syntax & Core Signatures

yaml
containers: - name: api image: ghcr.io/acme/api:1.4.2 resources: requests: cpu: 250m # 0.25 CPU; 1000m = 1 core memory: 512Mi ephemeral-storage: 1Gi limits: cpu: "1" # no throttling below 1 core memory: 768Mi # OOMKilled above this
yaml
# LimitRange: defaults and bounds applied to pods in a namespace apiVersion: v1 kind: LimitRange metadata: { name: defaults, namespace: prod } spec: limits: - type: Container default: { cpu: 500m, memory: 512Mi } defaultRequest: { cpu: 100m, memory: 256Mi } max: { cpu: "2", memory: 2Gi } min: { cpu: 50m, memory: 64Mi }
yaml
# ResourceQuota: aggregate ceiling for a namespace apiVersion: v1 kind: ResourceQuota metadata: { name: prod, namespace: prod } spec: hard: requests.cpu: "20" requests.memory: 40Gi limits.cpu: "40" limits.memory: 80Gi pods: "100"
bash
kubectl describe node | sed -n '/Allocated resources/,/Events/p' kubectl top pod -n prod --sort-by=memory kubectl top node kubectl get pod api-1 -o jsonpath='{.status.qosClass}{"\n"}' kubectl get pod api-1 -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' kubectl get events -n prod --field-selector reason=OOMKilling kubectl get resourcequota,limitrange -n prod

Units: cpu: 100m is 0.1 core, cpu: 1 is one core; memory accepts Mi/Gi (binary) and M/G (decimal) — 500M is 500,000,000 bytes, 500Mi is 524,288,000, and mixing them in requests and limits is a common source of surprise.

Key Comparison Table

ResourceOver requestOver limit
CPUNo enforcement; may use idle capacityThrottled by CFS quota (latency rises)
MemoryNo enforcement until node pressureOOMKilled, exit 137
Ephemeral storageNo enforcementPod evicted
Used by schedulerYes (sum of requests)No
Used by kubelet evictionThresholds on node pressureYes
QoS classConditionEviction orderTypical use
GuaranteedEvery container: cpu and memory requests = limitsLastLatency-critical services
BurstableAt least one request or limit set, not equalSecondMost workloads
BestEffortNo requests or limits at allFirstBatch, tests
SituationWhat actually happens
CPU limit 200m, app needs 400mThrottled to 200m; requests slow, no crash
Memory limit 512Mi, app needs 700MiOOMKilled, container restarts
No requests, no limitsScheduled anywhere, first to be evicted
Requests > node allocatablePending, FailedScheduling: Insufficient cpu
Requests far below usageNode oversubscribed; eviction under pressure

Practical Edge Cases & Deep Dives

1. CPU throttling: the latency problem with no error

bash
$ kubectl top pod -n prod | grep api api-6d9f 180m 640Mi # Throttling ratio, from the kubelet/cAdvisor metrics: container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total # 0.42 -> 42% of 100ms periods hit the quota

A CPU limit of 200m means 20 ms of CPU per 100 ms period. A request that needs 30 ms of CPU is paused for the rest of the period, so p99 latency jumps even though average CPU (kubectl top) looks comfortable at 180m. Throttling leaves no event, no OOM, no restart — just slow responses, which is why it is the hardest limit problem to find. The remedies are to raise or remove the CPU limit (keeping the request for scheduling), or to accept it only where latency does not matter.

Removing the CPU limit is safe when the request is accurate: the scheduler still reserves capacity, and the pod may use idle CPU on the node. The risk is a runaway process starving neighbours, which matters on shared nodes and not at all on dedicated ones.

2. OOMKilled and the JVM that ignores its limit

bash
$ kubectl get pod api-1 -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' OOMKilled $ kubectl get pod api-1 -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}' 137

A JVM in a container historically read the host's memory and sized its heap from it, so a 512Mi limit with a host of 32 GiB produced a heap larger than the limit and an OOM kill within minutes. Modern JVMs are container-aware (UseContainerSupport, default since 10) and use -XX:MaxRAMPercentage (25% by default) of the cgroup limit — but a -Xmx flag in the image's JAVA_OPTS overrides it. The memory limit must also cover JVM non-heap: metaspace, code cache, thread stacks, direct buffers, and the GC itself, which in practice is 20-40% above the heap.

yaml
env: - name: JAVA_TOOL_OPTIONS value: "-XX:MaxRAMPercentage=65 -XX:InitialRAMPercentage=65" resources: requests: { memory: 1Gi } limits: { memory: 1Gi }

3. Requests and the Pending pod on an idle-looking node

bash
$ kubectl describe node worker-2 | sed -n '/Allocated resources/,/Events/p' Resource Requests Limits cpu 3850m (96%) 6000m (150%) memory 14Gi (88%) 20Gi (125%) $ kubectl top node worker-2 worker-2 cpu: 900m (22%) memory: 6Gi (38%)

The scheduler compares requests to allocatable, not usage to capacity. A node at 96% requested with 22% used cannot accept another 250m pod. The fix is not to raise the node size but to correct the requests: sample actual usage, set requests near the p50-p80 of real consumption, and keep a limit for the ceiling. Overestimating requests "for safety" is the most common cause of wasteful clusters.

4. QoS class as an availability decision

yaml
# Guaranteed: a critical service that must be evicted last resources: requests: { cpu: "1", memory: 2Gi } limits: { cpu: "1", memory: 2Gi }

Every container in the pod must satisfy equality for both cpu and memory, otherwise the pod is Burstable. A single sidecar without limits demotes the whole pod, so a service mesh injector that adds an unlimited proxy silently changes the pod's eviction class.

Under memory pressure, the kubelet evicts BestEffort pods first, then Burstable pods using more than their requests, and Guaranteed last; within a class, the ranking uses usage relative to requests plus priority. A Guaranteed pod can still be OOM killed by the kernel if it exceeds its own memory limit, so Guaranteed is protection against eviction, not against its own limit.

5. Quotas, LimitRanges, and the pod that is rejected, not unscheduled

bash
$ kubectl apply -f api.yaml Error from server (Forbidden): error when creating "api.yaml": pods "api-1" is forbidden: exceeded quota: prod, requested: requests.cpu=500m, used: requests.cpu=20, limited: requests.cpu=20

A quota rejection happens at admission, before scheduling, and the message names the exceeded resource. A LimitRange turns a missing resources block into defaults; without one, a container with no requests is BestEffort and counts as zero against requests.* quotas, which is how a namespace can run at "0% quota" and still be full of pods. In a namespace with a quota on requests.cpu, every pod needs explicit requests or a LimitRange default, otherwise admission fails with must specify limits.cpu.

6. Measuring before setting numbers

bash
# Observed usage over time (metrics-server history is short; use Prometheus for real trends) kubectl top pod -n prod --containers | sort -k3 -h | tail -20 # Peak memory from the container's own history kubectl get pod api-1 -o jsonpath-range='{range .status.containerStatuses[*]}{.name} {.lastState.terminated.reason}{"\n"}{end}' # Right-sizing from Prometheus # max_over_time(container_memory_working_set_bytes{namespace="prod"}[7d]) by (pod) # quantile_over_time(0.95, rate(container_cpu_usage_seconds_total[5m])[7d:]) by (pod)

Set memory requests at the observed peak working set plus a margin, memory limits at the peak plus headroom for bursts, CPU requests at the p95 usage, and CPU limits either absent or well above p99. container_memory_working_set_bytes is the figure the OOM killer effectively acts on, not container_memory_usage_bytes.

Common Runtime & Configuration Mistakes

SymptomCauseFix
OOMKilled, exit 137Memory limit below working setRaise the limit; fix heap/MaxRAMPercentage
Slow responses, no errors, low CPU usageCPU throttlingRaise or remove the CPU limit
Pending: Insufficient cpuRequests exceed allocatableLower requests, add nodes, check quotas
Node "full" at 20% usageRequests overestimatedRight-size from measurements
Pod rejected on createNamespace quota exhaustedRaise the quota or request less
must specify limits.cpuQuota requires limits, none setSet limits or add a LimitRange
Container gets no default resourcesNo LimitRangeAdd defaultRequest/default
Critical pod evicted firstBestEffort or low requestsSet requests=limits (Guaranteed), priority
JVM heap ignores the limit-Xmx overrides container awarenessRemove -Xmx, use MaxRAMPercentage
Sidecar demotes the pod to BurstableSidecar without resourcesSet resources on every container
Memory grows until the node OOMsNo memory limit anywhereSet limits, monitor working set
Ephemeral storage evictionNo ephemeral-storage request/limitSet both; clean logs and temp files

Requests and limits are two different promises: requests tell the scheduler what to reserve and decide QoS, limits tell the kernel what to enforce and decide whether the symptom is throttling or death. Measure the working set, set memory requests at the real peak with a limit slightly above it, keep CPU requests accurate and CPU limits loose or absent, and check every container in the pod — because one unset sidecar changes the class of the whole workload.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.