Kubernetes Pod Evicted: ephemeral-storage Pressure Explained
Evicted pods are killed by the kubelet, not restarted. Requests, log caps and emptyDir size limits decide who dies first on a full node.

Introduction & Quick Summary
Symptom. Pods disappear from a node and kubectl get pods shows STATUS: Failed with REASON: Evicted, while the Deployment keeps creating replacements that die the same way. kubectl describe pod says The node was low on resource: ephemeral-storage. Threshold quantity: 10433324Ki, available: 9912Ki. Nothing in the container logs explains it, because the container was never restarted — the kubelet deleted the whole pod.
Cause. The kubelet continuously compares node filesystem capacity against eviction thresholds (nodefs.available by default 10% of capacity, plus inode thresholds). When a node crosses a threshold softly or hard, it evicts pods ranked by QoS class and by how far they exceed their ephemeral-storage requests. The usual consumers are container logs, writable layers, emptyDir volumes and unused image layers on a node whose disk was sized for the application alone.
Fix. Give every container an ephemeral-storage request and a sane limit, cap log growth with kubelet's containerLogMaxSize/containerLogMaxFiles, keep scratch data out of the container filesystem, and clean up images on small nodes. Evicted pods are terminal: they must be deleted (kubectl delete pod --field-selector=status.phase=Failed) — the Deployment's replacement is the real new pod.
Core Mechanics, Signatures & Reproducible Setup
There are two filesystems the kubelet accounts for, and the message names which one is short:
nodefs— the root filesystem: container writable layers, logs under/var/log/pods,emptyDirvolumes.imagefs— the filesystem holding image layers, when it is a separate mount.
Thresholds come from kubelet flags and can be inspected per node:
kubectl get node worker-1 -o jsonpath='{.status.capacity.ephemeral-storage}{"\n"}{.status.allocatable.ephemeral-storage}{"\n"}'
kubectl describe node worker-1 | sed -n '/Conditions/,/Addresses/p'
# DiskPressure True kubelet has disk pressureA pod that requests and limits ephemeral storage looks like this, and the request is what protects it from being evicted first:
apiVersion: v1
kind: Pod
metadata:
name: exporter
spec:
containers:
- name: exporter
image: myorg/exporter:1.4.0
resources:
requests:
cpu: 100m
memory: 128Mi
ephemeral-storage: 512Mi # scheduling + eviction ranking
limits:
memory: 256Mi
ephemeral-storage: 2Gi # container is killed at this usage
volumeMounts:
- name: scratch
mountPath: /var/scratch
volumes:
- name: scratch
emptyDir:
sizeLimit: 1Gi # refuses writes beyond 1GiNote the two distinct effects: limits.ephemeral-storage makes the container be killed for exceeding its own limit, while a node-level threshold makes the pod be evicted. The messages differ, and so do the fixes.
Key Comparison Table
The failure reasons that look alike:
| Reason | Where the limit lives | Message | Who kills it |
|---|---|---|---|
Evicted (ephemeral-storage) | Node threshold | The node was low on resource: ephemeral-storage | kubelet |
Evicted (memory) | Node threshold | The node was low on resource: memory | kubelet |
OOMKilled | Container limit | Last State: Terminated, Reason: OOMKilled, Exit Code: 137 | cgroup |
Evicted (DiskPressure) | Node condition | The node had condition: [DiskPressure] | kubelet |
FailedScheduling | Requests vs allocatable | 0/3 nodes are available: 3 Insufficient ephemeral-storage | scheduler |
ContainerCreating forever | Volume attach | FailedMount events | kubelet |
Eviction order follows QoS, so the request you set changes who dies first:
| QoS class | Condition | Evicted | Notes |
|---|---|---|---|
BestEffort | No requests or limits on any container | First | Always the first to go |
Burstable | Some requests set, not equal to limits | Second, by usage above request | Common default for apps |
Guaranteed | Requests equal limits for cpu and memory | Last | Still evictable on hard pressure |
Practical Edge Cases, Error Fixes & Deep Dive Code Examples
1. Logs that fill the disk
A chatty container writing to stdout is the most common cause on small nodes. The kubelet rotates logs only up to the configured caps, and a pod logging 50 MB per hour will cross a 20 GB node in two weeks:
# kubelet config: /var/lib/kubelet/config.yaml
containerLogMaxSize: 50Mi
containerLogMaxFiles: 3kubectl logs deploy/api --tail=100 --timestamps | wc -l
kubectl exec deploy/api -- du -sh /var/log/pods 2>/dev/null || trueApplication-level fixes beat kubelet tuning: raise the log level in production, sample request logs, and send structured logs to a collector instead of stdout on the node's disk.
2. emptyDir used as a scratch disk
emptyDir lives on the node's disk by default, and its usage counts against the pod's ephemeral-storage request. A batch job that writes temp files without cleanup evicts itself and, on a shared node, its neighbours:
volumes:
- name: tmp
emptyDir:
medium: Memory # counts against memory, not ephemeral storage
sizeLimit: 256MiA memory-backed emptyDir is the right answer for small scratch data and the wrong one for a 5 GB dataset: it then counts against the pod's memory limit and produces OOMKilled instead of Evicted.
3. Image churn on a node that cannot hold the images
Frequent deploys of large images fill imagefs. The kubelet garbage-collects images at 85% by default (imageGCHighThresholdPercent), but on a 20 GB node with 4 GB images, collection and re-pull can loop:
kubectl describe node worker-1 | grep -i -A3 "Images"
docker system df # on the node, if you have accessPinning a smaller base image, using multi-stage builds and a local registry mirror reduces both pull time and disk churn.
4. Requests that let the scheduler over-pack
Without ephemeral-storage requests the scheduler knows nothing about disk usage and packs pods onto the fullest node. Adding a request that reflects reality — logs plus scratch — is what makes the packing decision correct:
kubectl get pods -A -o json | jq -r '
.items[] | [.metadata.namespace, .metadata.name,
(.spec.containers[0].resources.requests["ephemeral-storage"] // "none")] | @tsv' | headThe jq output is a quick audit: none on a log-heavy workload is a future eviction.
5. Soft and hard eviction thresholds
Hard thresholds evict immediately; soft thresholds only after eviction-soft-grace-period. A cluster tuned for latency-critical workloads often sets memory.available soft at 1Gi with a 90-second grace period, which gives the application time to release caches. The flags live on the kubelet, not in the pod spec:
# kubelet config
evictionHard:
memory.available: "500Mi"
nodefs.available: "10%"
imagefs.available: "15%"
evictionSoft:
memory.available: "1Gi"
evictionSoftGracePeriod:
memory.available: "1m30s"Thresher thresholds than the node can honour (evictionHard above capacity) are ignored, which is why a node can look misconfigured and behave normally.
6. Cleaning up evicted pods and confirming the real state
Evicted pods are Succeeded-like leftovers: the ReplicaSet already created a replacement, and the failed object stays until something deletes it. Scripts that count kubectl get pods see double during an incident:
kubectl get pods -A --field-selector=status.phase=Failed \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.reason
kubectl delete pod -A --field-selector=status.phase=FailedNever delete pods by name during an eviction storm without checking REASON: a Failed pod caused by a crash loop is a different investigation.
Common Compilation/Runtime Mistakes & Version Gotchas
- Treating
Evictedas a crash loop. The container never restarted, sokubectl logs --previousis empty andrestartCountis 0. The evidence is in the pod'sstatus.messageand node conditions. - Assuming
ephemeral-storagelimits are enforced like memory. Container-level limits are enforced by the runtime with its own accounting; the exact byte count that trips is not identical toduon the writable layer, and a limit far below real usage kills healthy containers. - Setting
medium: Memoryfor large scratch data. It moves the failure from disk eviction toOOMKilled, which is harder to attribute because the message points at memory limits. - Forgetting inodes.
nodefs.inodesFreecan cross its threshold while bytes are plentiful; millions of tiny log or cache files evict pods on a disk thatdf -hcalls empty. emptyDir.sizeLimitas a security boundary. It limits what the pod writes through that volume, not what a process writes to the container's own filesystem; set a container limit too.- Assuming a PVC protects you. A
PersistentVolumeClaimmoves data to a separate volume, but logs and the writable layer still live onnodefs. - Older clusters without local storage isolation.
LocalStorageCapacityIsolation(GA since 1.25) is what makes ephemeral-storage requests meaningful; clusters older than that ignore them entirely, so an upgrade changes eviction behaviour without a spec change.
Related tools. When comparing the pod spec that survived an incident with the one that was deployed, paste both YAML files into the text diff tool to find the missing sizeLimit or request, and if the container writes JSON logs that you need to inspect while the pod is still around, pretty-print a line with the JSON formatter instead of scrolling a minified stream.
Try matching browser tools
All toolsSmall utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Kubernetes guides
All Kubernetes guidesKubernetes Pending Pod: CPU Requests, Unbound PVC, Taints
The scheduler event names every node and its reason. Requests versus allocatable, untolerated taints, zonal volume binding, and impossible anti-affinity.
Kubernetes Exit Code 137 OOMKilled vs Exit Code 134 Heap Limits
137 is SIGKILL from the cgroup or a liveness probe; 134 is an abort, usually V8's own heap ceiling. Telling them apart and sizing the two limits together.
CrashLoopBackOff: Reading Exit Codes and Debugging Restart Loops
The backoff is only the waiting state. The evidence is in Last State, the exit code, and the previous container's last log line, and each points to a different fix.
ImagePullBackOff and ErrImagePull: Reading the Real Pull Error
ImagePullBackOff hides five different failures. The event text names which one: wrong reference, missing credentials, wrong architecture, unreachable registry, or a rate limit.
More in Kubernetes
All Kubernetes guidesThe guides filed next to this one in Kubernetes, in library order.