Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Exit Code 137 OOMKilled vs Exit Code 134 Heap Limits

137 is SIGKILL from the cgroup or a liveness probe; 134 is an abort, usually V8's own heap ceiling. Telling them apart and sizing the two limits together.

~5 min readKubernetes
#Kubernetes#Memory#Node.js#Troubleshooting
3D illustration of a pod cube cut in half with an overflowing red memory bar and an abort badge

Introduction & Quick Summary

Symptom. A pod restarts every few minutes. kubectl get pod shows OOMKilled or Error, and kubectl describe pod prints Last State: Terminated Reason: OOMKilled Exit Code: 137 — or the same pod shows Exit Code: 134 with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory in the previous container's log. Memory graphs look fine right up to the restart, and the app logged nothing before dying.

Cause. Both are memory failures, but at different layers. 137 is 128 + 9 (SIGKILL): the kernel's cgroup OOM killer chose the process because the container's working set exceeded its memory.limit_in_bytes, or the kubelet killed it because a liveness probe failed — both send SIGKILL, and only the former sets Reason: OOMKilled. 134 is 128 + 6 (SIGABRT): the process aborted itself, in Node typically because V8's old-space limit was reached while the container still had room, or a native library called abort().

Fix. Make the runtime's internal ceiling sit below the container's ceiling. For Node, set NODE_OPTIONS=--max-old-space-size to roughly 70–80% of the memory limit and leave the rest for buffers, threads, and the native stack. Set requests == limits so the pod is Guaranteed and is not evicted for a neighbour's pressure, add a startupProbe so a slow boot is not killed by liveness, and watch container_memory_working_set_bytes rather than the application's own heap metric — the gap between them is the leak you cannot see from inside.

Core Mechanics, Signatures & Reproducible Setup

Reproduce the 134 case in a pod with a limit far above the heap, which is exactly how it happens in production:

yaml
apiVersion: v1 kind: Pod metadata: name: node-134 spec: containers: - name: app image: node:22-alpine command: ["node", "-e", "const a=[];for(;;)a.push(Buffer.alloc(1024*1024).toString('hex'))"] resources: requests: { memory: "256Mi", cpu: "100m" } limits: { memory: "2Gi", cpu: "1" }

The container limit is 2 GiB and V8's default old space is derived from what Node believes is available, so the process aborts with 134 when V8 gives up, long before the cgroup would kill it. Set the ceiling explicitly:

yaml
env: - name: NODE_OPTIONS value: "--max-old-space-size=1200" # ~60% of 2Gi, headroom for buffers resources: requests: { memory: "2Gi", cpu: "500m" } limits: { memory: "2Gi", cpu: "1" }

Read the two states apart with the evidence Kubernetes keeps:

bash
kubectl get pod app-7d9 -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' kubectl describe pod app-7d9 | sed -n '/Last State/,/Ready/p' kubectl logs app-7d9 --previous --tail=100 kubectl get events --field-selector involvedObject.name=app-7d9 --sort-by=.lastTimestamp

lastState.terminated.reason is OOMKilled for a cgroup kill, Error for a non-zero exit, and Completed for 0. The Events section distinguishes a probe kill: Killing container ... Container app failed liveness probe, will be restarted and Liveness probe failed: ... appear next to Killing with no OOMKilled reason.

Key Comparison Table

Exit codeSignalWho kills itWhere the evidence is
1379 SIGKILLcgroup OOM killerReason: OOMKilled, node dmesg
137 (no OOMKilled)9 SIGKILLkubelet, failed liveness/startup probeEvents: failed liveness probe
1346 SIGABRTThe process (V8 fatal, glibc abort)Previous log: FATAL ERROR
14315 SIGTERMkubelet, graceful stop, node drainterminationGracePeriodSeconds
13911 SIGSEGVNative crash, bad pointerStack trace in log or core
1—Application errorApplication log
InvestigationCommandTells you
Reason and codekubectl get pod -o jsonpath='…lastState.terminated'OOMKilled vs Error
Probe killskubectl get events --sort-by=.lastTimestampLiveness, startup, readiness
Last wordskubectl logs --previousV8 fatal error text
Live memorykubectl top pod, cAdvisorWorking set vs limit
Node-level OOMdmesg -T | grep -i oom on the nodeWhich container, which size
Heap inside the app/debug/pprof, process.memoryUsage()Heap vs RSS gap

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. OOMKilled with a memory graph that never reached the limit

container_memory_working_set_bytes is sampled, and the last sample before a kill can be one scrape interval old; a burst between samples is invisible. Worse, the metric the app exports (process.memoryUsage().heapUsed) excludes Buffers, ArrayBuffer, native allocations, worker threads, and the V8 code space. Log the difference:

js
setInterval(() => { const m = process.memoryUsage(); log.info({ rss: m.rss, heap: m.heapUsed, external: m.external, arrayBuffers: m.arrayBuffers }); }, 30_000);

rss growing while heap is flat means off-heap growth: leaked Buffers, unclosed streams, or a native addon. Raising --max-old-space-size will not fix it.

2. Exit 137 from a liveness probe, not memory

A probe with initialDelaySeconds: 5 on a service that takes 40 s to warm up kills it forever, and each restart begins the warm-up again. Use a startup probe with a generous budget and let liveness only cover the steady state:

yaml
startupProbe: httpGet: { path: /healthz, port: 3000 } failureThreshold: 30 periodSeconds: 2 livenessProbe: httpGet: { path: /healthz, port: 3000 } periodSeconds: 10 failureThreshold: 3

3. Heap set above the container limit

--max-old-space-size=2048 in a 512Mi container is the mirror image of 134: V8 never aborts, the cgroup kills at 512Mi, and you get 137 with no diagnostic from the app. Keep the ratio explicit and assert it at boot:

js
const limitMb = Number(process.env.MEMORY_LIMIT_MB ?? 0); const heapMb = Number(process.execArgv.find((a) => a.startsWith('--max-old-space-size='))?.split('=')[1] ?? 0); if (limitMb && heapMb > limitMb * 0.85) log.warn({ limitMb, heapMb }, 'heap too close to cgroup limit');

4. Worker threads and cluster forks multiply the ceiling

--max-old-space-size applies per isolate. Four workers in a 1 GiB container each get the flag and compete for the same cgroup: total heap can reach four times the number you set. Divide deliberately:

js
import os from 'node:os'; const workers = Number(process.env.WORKERS ?? 2); const perWorkerMb = Math.floor(totalMb / workers * 0.7);

5. JVM and .NET have the same split

-XX:MaxRAMPercentage=75 is the JVM equivalent of --max-old-space-size, and without it older JDKs read the host's memory rather than the cgroup limit — the same 137/134 confusion appears as java.lang.OutOfMemoryError: Java heap space (abort) beside OOMKilled (kernel). Both runtimes need the container limit discovered at startup, which cgroup v2 exposes in memory.max.

6. Capturing a heap snapshot before the kill

Once the OOM killer acts, there is nothing to inspect. Run with a headroom-triggered snapshot, or attach an ephemeral container with kubectl debug and take a profile while it is alive:

bash
kubectl debug -it app-7d9 --image=node:22-alpine --target=app -- sh node -e "process.kill(1, 'SIGUSR2')" # if your app writes a snapshot on SIGUSR2

--previous logs plus one heap snapshot taken at 80% of the limit identify more leaks than any dashboard.

Common Compilation/Runtime Mistakes & Version Gotchas

  • limits without requests. The pod becomes Burstable; the scheduler packs it onto a busy node, and the kernel reclaims its memory under pressure before the cgroup limit is hit.
  • Relying on Reason: OOMKilled only. A container killed by the kernel while another container on the node was over its limit can be reported without that reason; correlate with node events.
  • cgroup v2 accounting differs. Page cache counts toward the memory limit differently, and a process reading large files can be killed while its anonymous memory is small. --memory-swap interacts as well.
  • Node 20+ reads the cgroup limit correctly, but only for the main thread's heap. Buffers, SharedArrayBuffer, and native memory are outside it; the 70–80% ratio exists for exactly that reason.
  • terminationGracePeriodSeconds: 30 with slow drains. The pod receives SIGTERM (143) and is SIGKILLed (137) at the deadline, which shows as 137 without OOMKilled — raise the grace period or shut down faster.
  • restartPolicy: Always hides the pattern. Count restarts (kubectl get pod -o jsonpath='{.status.containerStatuses[0].restartCount}') and alert on the rate; a crash loop is a trend, not an event.
  • resources.limits.memory below ~64Mi. Node cannot even start; the failure looks like a missing binary or an instant OOM with no log.

Related tools. Decode the previous container's structured log lines with the JSON formatter, convert the restart timestamps into a comparable timeline with the timestamp converter, and encode the memory-limit values you pass through Helm or envsubst with the base64 encoder when they arrive from a secret.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.