Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Pending Pod: CPU Requests, Unbound PVC, Taints

The scheduler event names every node and its reason. Requests versus allocatable, untolerated taints, zonal volume binding, and impossible anti-affinity.

~4 min readKubernetes
#Kubernetes#Scheduling#Storage#Troubleshooting
3D illustration of a pod cube floating unscheduled above a row of node cubes, one with a taint seal and one with an unattached disk volume

Introduction & Quick Summary

Symptom. A pod stays Pending and the rollout never finishes. kubectl describe pod ends with Events: Warning FailedScheduling 0/6 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint {dedicated: gpu}, 1 node(s) had volume node affinity conflict. Or the message is shorter: 0/6 nodes are available: pod has unbound immediate PersistentVolumeClaims.

Cause. Pending is a scheduler verdict, not an application failure, and the event string lists every node with its own reason. The common four are: the sum of requests (not usage) of all pods on a node leaves no room for this one; a taint on the eligible nodes has no matching toleration; a requiredDuringSchedulingIgnoredDuringExecution affinity rule matches no node; and the PersistentVolumeClaim is not bound, either because no PersistentVolume exists or because its zone does not match any schedulable node.

Fix. Read the event string node by node before changing anything. Then either lower the request to something real, add the toleration, remove the hard affinity, or fix the claim — and remember that with volumeBindingMode: WaitForFirstConsumer the claim binds only after scheduling, so a nodeSelector narrow enough to exclude the volume's zone deadlocks.

Core Mechanics, Signatures & Reproducible Setup

The scheduler filters nodes in two phases: Filter (does this node satisfy requests, taints, affinity, volumes?) and Score. A pod that passes no node's Filter stays Pending with a per-node reason string.

bash
kubectl describe pod web-7d9f8b6c5-x2k4p | sed -n '/Events:/,$p' kubectl get events --field-selector involvedObject.name=web-7d9f8b6c5-x2k4p \ --sort-by=.lastTimestamp
yaml
# a deliberately over-requesting pod: reproduces Insufficient cpu on a small cluster apiVersion: v1 kind: Pod metadata: name: requests-too-high spec: containers: - name: app image: nginx:1.27-alpine resources: requests: { cpu: "2", memory: "4Gi" } limits: { cpu: "2", memory: "4Gi" }

With 3 × 2-vCPU nodes and any other workload present, this pod reports 3 Insufficient cpu and never schedules. Change requests.cpu to 200m and it runs immediately — which proves the request, not the limit, was the blocker.

Key Comparison Table

Event fragmentMeansFirst check
Insufficient cpunode's allocatable minus sum of requestskubectl describe node allocated resources
Insufficient memorysame, for memorylargest resident requests, not usage
unbound immediate PersistentVolumeClaimsclaim not boundkubectl get pvc -o wide
volume node affinity conflictPV zone excludes all nodesPV nodeAffinity vs node labels
untolerated taint {key: value}no tolerationkubectl get nodes -o custom-columns
node(s) didn't match Pod's node affinity/selectorlabels do not matchkubectl get nodes --show-labels
node(s) didn't match pod anti-affinity rulesreplica co-location blockedcount nodes vs replicas
Preemption is not helpfulnothing to evictpriority class
Volume binding modeBinds whenDeadlock risk
Immediateat PVC creationclaim bound in a zone with no node
WaitForFirstConsumerafter the pod is schedulednodeSelector excludes the volume's zone
pre-bound PV (volumeName)immediatelyPV already in the wrong zone

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. Reading the node capacity that matters

requests are reserved whether or not the process uses them, and only allocatable counts — the kubelet's reserved resources are not available to pods.

bash
kubectl describe node ip-10-0-1-24 | sed -n '/Allocated resources/,/Events/p'
code
Allocated resources: Resource Requests Limits cpu 1850m (92%) 4 (200%) memory 3.1Gi (81%) 6Gi (156%)

92% of CPU requests means a pod asking for 200m cannot fit even though the node reports low real usage. Two fixes: lower the request to a measured value, or add a node. Deleting other pods to free requests is only correct if those pods are genuinely unnecessary.

2. Taints and the node that was supposed to be excluded

A GPU node pool tainted dedicated=gpu:NoSchedule will not accept ordinary pods, and the scheduler reports 2 node(s) had untolerated taint. Adding a toleration lets the pod land there, which is usually the wrong outcome — it consumes an expensive node. Prefer nodeSelector on a CPU pool, and add tolerations only for daemons that must run everywhere.

yaml
tolerations: - key: dedicated operator: Equal value: gpu effect: NoSchedule

NoSchedule blocks new pods; NoExecute also evicts running ones, which is why adding a NoExecute taint causes a restart wave rather than a scheduling failure.

3. PVC that never binds

Three distinct causes share one symptom, and the PVC's events name which:

bash
kubectl describe pvc data-postgres-0 | sed -n '/Events:/,$p'
  • no persistent volumes available for this claim and no storage class is set — storageClassName missing or misspelled.
  • waiting for first consumer to be created before binding — normal with WaitForFirstConsumer, not an error; check whether the pod was scheduled at all.
  • provisioned volume ... zone us-east-1a together with nodes only in 1b and 1c — the class says Immediate and provisioned in the wrong zone.
yaml
# the usual correct shape on a zonal cloud apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3 provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true

With that class the volume is created in the zone the pod actually scheduled into, so the conflict cannot arise.

4. Anti-affinity with fewer nodes than replicas

requiredDuringSchedulingIgnoredDuringExecution on topologyKey: kubernetes.io/hostname with replicas: 5 and 3 nodes leaves two pods permanently Pending — by design. Switch to preferredDuringSchedulingIgnoredDuringExecution, or use topologySpreadConstraints with whenUnsatisfiable: ScheduleAnyway.

yaml
topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: { app: web }

5. ResourceQuota and LimitRange rejections

A LimitRange can rewrite a missing request to a default that no node can satisfy, and a ResourceQuota blocks creation with exceeded quota: compute-resources, requested: requests.cpu=2, used: requests.cpu=7, limited: requests.cpu=8. The pod object exists and is Pending with no scheduling event, which is easy to misread.

bash
kubectl describe quota -n prod kubectl get events -n prod --field-selector reason=FailedCreate

6. Preemption and priority classes

When nothing fits, the scheduler can evict lower-priority pods if the incoming pod's priorityClassName is higher. Preemption is not helpful for scheduling means even evicting everything on the best node does not make room — a signal that the request itself is impossible, not that the cluster is full.

yaml
apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: { name: critical } value: 1000000 globalDefault: false

Preemption is disruptive to whatever it evicts; it is a tool for a genuinely critical workload, not for a web deployment that should simply have a realistic request.

Common Compilation/Runtime Mistakes & Version Gotchas

  • Reading kubectl top to explain Insufficient cpu. The scheduler uses requests; usage is irrelevant to it.
  • requests copied from a VM's size. A 2 CPU request on a 2-vCPU node leaves room for nothing else.
  • NoExecute taints added during maintenance. They evict running pods, and tolerations must already be deployed.
  • Immediate binding on a zonal storage class. The volume can be provisioned in a zone with no schedulable node.
  • nodeSelector plus WaitForFirstConsumer pointing at different zones. Permanent Pending with two separate reasons.
  • Anti-affinity required with more replicas than nodes. Mathematically impossible; use preferred or spread constraints.
  • A LimitRange default larger than any node's allocatable. Every pod without explicit requests becomes unschedulable.

Related tools. Normalise the pod's full JSON status before diffing it against a working replica with the JSON formatter, diff two manifests to spot the affinity or request that changed with the text diff, and convert the event timestamps to local time with the timestamp converter.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.