Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Pod Evicted: ephemeral-storage Pressure Explained

Evicted pods are killed by the kubelet, not restarted. Requests, log caps and emptyDir size limits decide who dies first on a full node.

~5 min readKubernetes
#Kubernetes#Storage#Eviction#Troubleshooting
3D illustration of a pod being pushed off a node grid by a rising red usage bar

Introduction & Quick Summary

Symptom. Pods disappear from a node and kubectl get pods shows STATUS: Failed with REASON: Evicted, while the Deployment keeps creating replacements that die the same way. kubectl describe pod says The node was low on resource: ephemeral-storage. Threshold quantity: 10433324Ki, available: 9912Ki. Nothing in the container logs explains it, because the container was never restarted — the kubelet deleted the whole pod.

Cause. The kubelet continuously compares node filesystem capacity against eviction thresholds (nodefs.available by default 10% of capacity, plus inode thresholds). When a node crosses a threshold softly or hard, it evicts pods ranked by QoS class and by how far they exceed their ephemeral-storage requests. The usual consumers are container logs, writable layers, emptyDir volumes and unused image layers on a node whose disk was sized for the application alone.

Fix. Give every container an ephemeral-storage request and a sane limit, cap log growth with kubelet's containerLogMaxSize/containerLogMaxFiles, keep scratch data out of the container filesystem, and clean up images on small nodes. Evicted pods are terminal: they must be deleted (kubectl delete pod --field-selector=status.phase=Failed) — the Deployment's replacement is the real new pod.

Core Mechanics, Signatures & Reproducible Setup

There are two filesystems the kubelet accounts for, and the message names which one is short:

  • nodefs — the root filesystem: container writable layers, logs under /var/log/pods, emptyDir volumes.
  • imagefs — the filesystem holding image layers, when it is a separate mount.

Thresholds come from kubelet flags and can be inspected per node:

bash
kubectl get node worker-1 -o jsonpath='{.status.capacity.ephemeral-storage}{"\n"}{.status.allocatable.ephemeral-storage}{"\n"}' kubectl describe node worker-1 | sed -n '/Conditions/,/Addresses/p' # DiskPressure True kubelet has disk pressure

A pod that requests and limits ephemeral storage looks like this, and the request is what protects it from being evicted first:

yaml
apiVersion: v1 kind: Pod metadata: name: exporter spec: containers: - name: exporter image: myorg/exporter:1.4.0 resources: requests: cpu: 100m memory: 128Mi ephemeral-storage: 512Mi # scheduling + eviction ranking limits: memory: 256Mi ephemeral-storage: 2Gi # container is killed at this usage volumeMounts: - name: scratch mountPath: /var/scratch volumes: - name: scratch emptyDir: sizeLimit: 1Gi # refuses writes beyond 1Gi

Note the two distinct effects: limits.ephemeral-storage makes the container be killed for exceeding its own limit, while a node-level threshold makes the pod be evicted. The messages differ, and so do the fixes.

Key Comparison Table

The failure reasons that look alike:

ReasonWhere the limit livesMessageWho kills it
Evicted (ephemeral-storage)Node thresholdThe node was low on resource: ephemeral-storagekubelet
Evicted (memory)Node thresholdThe node was low on resource: memorykubelet
OOMKilledContainer limitLast State: Terminated, Reason: OOMKilled, Exit Code: 137cgroup
Evicted (DiskPressure)Node conditionThe node had condition: [DiskPressure]kubelet
FailedSchedulingRequests vs allocatable0/3 nodes are available: 3 Insufficient ephemeral-storagescheduler
ContainerCreating foreverVolume attachFailedMount eventskubelet

Eviction order follows QoS, so the request you set changes who dies first:

QoS classConditionEvictedNotes
BestEffortNo requests or limits on any containerFirstAlways the first to go
BurstableSome requests set, not equal to limitsSecond, by usage above requestCommon default for apps
GuaranteedRequests equal limits for cpu and memoryLastStill evictable on hard pressure

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. Logs that fill the disk

A chatty container writing to stdout is the most common cause on small nodes. The kubelet rotates logs only up to the configured caps, and a pod logging 50 MB per hour will cross a 20 GB node in two weeks:

bash
# kubelet config: /var/lib/kubelet/config.yaml containerLogMaxSize: 50Mi containerLogMaxFiles: 3
bash
kubectl logs deploy/api --tail=100 --timestamps | wc -l kubectl exec deploy/api -- du -sh /var/log/pods 2>/dev/null || true

Application-level fixes beat kubelet tuning: raise the log level in production, sample request logs, and send structured logs to a collector instead of stdout on the node's disk.

2. emptyDir used as a scratch disk

emptyDir lives on the node's disk by default, and its usage counts against the pod's ephemeral-storage request. A batch job that writes temp files without cleanup evicts itself and, on a shared node, its neighbours:

yaml
volumes: - name: tmp emptyDir: medium: Memory # counts against memory, not ephemeral storage sizeLimit: 256Mi

A memory-backed emptyDir is the right answer for small scratch data and the wrong one for a 5 GB dataset: it then counts against the pod's memory limit and produces OOMKilled instead of Evicted.

3. Image churn on a node that cannot hold the images

Frequent deploys of large images fill imagefs. The kubelet garbage-collects images at 85% by default (imageGCHighThresholdPercent), but on a 20 GB node with 4 GB images, collection and re-pull can loop:

bash
kubectl describe node worker-1 | grep -i -A3 "Images" docker system df # on the node, if you have access

Pinning a smaller base image, using multi-stage builds and a local registry mirror reduces both pull time and disk churn.

4. Requests that let the scheduler over-pack

Without ephemeral-storage requests the scheduler knows nothing about disk usage and packs pods onto the fullest node. Adding a request that reflects reality — logs plus scratch — is what makes the packing decision correct:

bash
kubectl get pods -A -o json | jq -r ' .items[] | [.metadata.namespace, .metadata.name, (.spec.containers[0].resources.requests["ephemeral-storage"] // "none")] | @tsv' | head

The jq output is a quick audit: none on a log-heavy workload is a future eviction.

5. Soft and hard eviction thresholds

Hard thresholds evict immediately; soft thresholds only after eviction-soft-grace-period. A cluster tuned for latency-critical workloads often sets memory.available soft at 1Gi with a 90-second grace period, which gives the application time to release caches. The flags live on the kubelet, not in the pod spec:

yaml
# kubelet config evictionHard: memory.available: "500Mi" nodefs.available: "10%" imagefs.available: "15%" evictionSoft: memory.available: "1Gi" evictionSoftGracePeriod: memory.available: "1m30s"

Thresher thresholds than the node can honour (evictionHard above capacity) are ignored, which is why a node can look misconfigured and behave normally.

6. Cleaning up evicted pods and confirming the real state

Evicted pods are Succeeded-like leftovers: the ReplicaSet already created a replacement, and the failed object stays until something deletes it. Scripts that count kubectl get pods see double during an incident:

bash
kubectl get pods -A --field-selector=status.phase=Failed \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.reason kubectl delete pod -A --field-selector=status.phase=Failed

Never delete pods by name during an eviction storm without checking REASON: a Failed pod caused by a crash loop is a different investigation.

Common Compilation/Runtime Mistakes & Version Gotchas

  • Treating Evicted as a crash loop. The container never restarted, so kubectl logs --previous is empty and restartCount is 0. The evidence is in the pod's status.message and node conditions.
  • Assuming ephemeral-storage limits are enforced like memory. Container-level limits are enforced by the runtime with its own accounting; the exact byte count that trips is not identical to du on the writable layer, and a limit far below real usage kills healthy containers.
  • Setting medium: Memory for large scratch data. It moves the failure from disk eviction to OOMKilled, which is harder to attribute because the message points at memory limits.
  • Forgetting inodes. nodefs.inodesFree can cross its threshold while bytes are plentiful; millions of tiny log or cache files evict pods on a disk that df -h calls empty.
  • emptyDir.sizeLimit as a security boundary. It limits what the pod writes through that volume, not what a process writes to the container's own filesystem; set a container limit too.
  • Assuming a PVC protects you. A PersistentVolumeClaim moves data to a separate volume, but logs and the writable layer still live on nodefs.
  • Older clusters without local storage isolation. LocalStorageCapacityIsolation (GA since 1.25) is what makes ephemeral-storage requests meaningful; clusters older than that ignore them entirely, so an upgrade changes eviction behaviour without a spec change.

Related tools. When comparing the pod spec that survived an incident with the one that was deployed, paste both YAML files into the text diff tool to find the missing sizeLimit or request, and if the container writes JSON logs that you need to inspect while the pod is still around, pretty-print a line with the JSON formatter instead of scrolling a minified stream.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.