Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Taints, Tolerations, and Node Affinity: How Placement Really Decides

Taints repel, tolerations permit, affinity prefers. The three taint effects, the NoExecute grace window, and why a label change never moves a running pod.

~4 min readKubernetes
#Kubernetes#Scheduling#Taints#Affinity
3D illustration of a node cube ringed by a warning barrier with a key-shaped toleration chip unlocking entry

Quick Summary

Taints and tolerations decide which pods a node refuses; affinity decides which nodes a pod prefers. They are two independent filters applied in order: the scheduler first finds nodes that fit the pod's resource requests, then excludes nodes whose taints the pod does not tolerate, then ranks what remains by affinity, topology spread, and other scores.

A taint is on the node (key=value:Effect) and a toleration is on the pod. NoSchedule blocks new scheduling, PreferNoSchedule makes it a last resort, and NoExecute also evicts running pods that do not tolerate it — with an optional tolerationSeconds grace window. A toleration is permission, not a placement instruction: a pod that tolerates a taint may still land anywhere.

Node affinity has two forms with the same syntax: requiredDuringSchedulingIgnoredDuringExecution is a hard filter, preferredDuringSchedulingIgnoredDuringExecution is a weighted score. The "IgnoredDuringExecution" suffix is the important half — once a pod is placed, changing the node's labels never evicts it, which is why nodeSelector and affinity are placement rules and taints are the only thing that can remove a running pod.

Basic Syntax & Core Signatures

yaml
apiVersion: v1 kind: Pod metadata: name: gpu-trainer spec: tolerations: - key: dedicated operator: Equal # Equal | Exists value: gpu effect: NoSchedule - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 120 # tolerate for 2 min, then evict affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: accelerator operator: In # In | NotIn | Exists | DoesNotExist | Gt | Lt values: ["nvidia-a100"] preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: topology.kubernetes.io/zone operator: In values: ["ap-northeast-2a"] containers: - name: trainer image: registry.example.com/ml/trainer:3.1 resources: limits: { nvidia.com/gpu: 1 }
bash
# Node side kubectl taint nodes node-7 dedicated=gpu:NoSchedule kubectl taint nodes node-7 dedicated=gpu:NoSchedule- # remove (trailing dash) kubectl label nodes node-7 accelerator=nvidia-a100 kubectl describe node node-7 | sed -n '/Taints/,/Unschedulable/p'
bash
# Why did this pod not schedule? kubectl describe pod gpu-trainer | tail -6 # 0/8 nodes are available: 4 node(s) had untolerated taint {dedicated: gpu}, # 2 Insufficient nvidia.com/gpu, 2 node(s) didn't match Pod's node affinity/selector. kubectl get pods -o wide --field-selector spec.nodeName=node-7

Key Comparison Table

EffectBlocks new podsEvicts running pods
NoScheduleYesNo
PreferNoScheduleSoft preference againstNo
NoExecuteYesYes, immediately or after tolerationSeconds
MechanismSet onPurposeCan evict later?
Taint / tolerationNode / podRepel pods from a nodeNoExecute only
nodeSelectorPodExact label matchNo
Required node affinityPodExpression filterNo
Preferred node affinityPodWeighted rankingNo
nodeNamePodBypasses the schedulerNo
Taint added automaticallyEffectNotes
node.kubernetes.io/not-readyNoExecute300s default toleration on pods
node.kubernetes.io/unreachableNoExecuteSame, tunable per pod
node.kubernetes.io/disk-pressureNoScheduleEviction by kubelet under pressure
node.kubernetes.io/memory-pressureNoSchedule
node.kubernetes.io/unschedulableNoScheduleSet by kubectl cordon
node.kubernetes.io/network-unavailableNoScheduleuntil the CNI reports ready

Practical Edge Cases & Deep Dives

1. Exists with no key tolerates everything

yaml
tolerations: - operator: Exists # matches every taint, any key, any effect

This is how infrastructure DaemonSets run everywhere, and how a pod can end up on a cordoned node during maintenance. The narrower form is almost always correct:

yaml
tolerations: - key: dedicated operator: Exists # that key, any value, any effect

When a pod still lands on a node you were draining, look for an operator: Exists toleration before suspecting the scheduler.

2. NoExecute and tolerationSeconds

A node becomes unreachable. The node controller adds node.kubernetes.io/unreachable:NoExecute, and every pod that does not tolerate it is evicted after the default toleration of 300 seconds. StatefulSets and quorum workloads often need longer:

yaml
tolerations: - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 600 # wait 10 minutes before giving up

tolerationSeconds only matters with NoExecute; on a NoSchedule taint it is accepted and ignored. Shortening it makes failover faster and split-brain more likely — a 30-second window on a database will promote a replica while the old primary is still alive.

3. Affinity is not a placement guarantee after scheduling

IgnoredDuringExecution means a label change never moves the pod. Remove accelerator=nvidia-a100 from a node whose GPU failed and the pods running there stay put, keep failing, and no event reports a scheduling problem. Eviction and rescheduling is the operator's job (kubectl drain, or a controller that deletes the pod). The same asymmetry applies to topology: a pod scheduled for zone balance today is not rebalanced when a new node appears in another zone.

4. Combining taints with affinity for dedicated node pools

The robust pattern for a GPU or licensed-software pool is both: a NoSchedule taint keeps general workloads out, and required affinity keeps GPU pods on those nodes.

bash
kubectl taint nodes -l pool=gpu dedicated=gpu:NoSchedule kubectl label nodes -l pool=gpu accelerator=nvidia-a100

With a taint alone, any pod with a broad Exists toleration can occupy the pool. With affinity alone, general pods with no affinity can still land there. Between the two there is a subtle failure: if the pool is full and the affinity is required, GPU pods go Pending rather than spilling onto ordinary nodes — usually what you want for licensing, rarely what you want for throughput. A preferred-affinity variant with a lower-priority fallback is the alternative.

5. Debugging the schedule decision

text
0/8 nodes are available: 4 node(s) had untolerated taint {dedicated: gpu}, 2 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector.

Read it right to left: the largest group is usually the taint exclusion, and Insufficient is a resource problem, not a placement one. kubectl get nodes --show-labels plus kubectl describe node <node> | grep Taints reconstructs everything the scheduler saw. For a pod that scheduled but misbehaves, kubectl get pod -o wide gives the node, and the kubelet's own view is in journalctl -u kubelet.

Common Runtime & Configuration Mistakes

SymptomCauseFix
Pod Pending, untolerated taintMissing tolerationAdd the specific key/effect
Tolerated but not scheduledAffinity or resources still exclude the nodeCheck the other exclusion reasons
Pod on a cordoned nodeoperator: Exists tolerationNarrow the toleration
Pod evicted after ~5 minutesNoExecute from node-not-readyRaise tolerationSeconds
Quorum cluster splitstolerationSeconds too shortLengthen the window
GPU pods on CPU nodesAffinity missing or preferredRequired affinity plus taint
General pods on GPU nodesTaint missing, only labelsAdd the NoSchedule taint
Node label change did not move podsIgnoredDuringExecutionDrain and reschedule deliberately
DaemonSet missing on new nodesTaint added to the node poolTolerate it in the template
Insufficient cpu after adding affinityPool too smallScale the pool, or relax affinity

Taints answer "who may come here", affinity answers "where does this go", and neither is retroactive. Diagnose a placement problem by reading the scheduler's exclusion list, and diagnose a running-pod problem by reading taints and eviction events.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.