Kubernetes Taints, Tolerations, and Node Affinity: How Placement Really Decides
Taints repel, tolerations permit, affinity prefers. The three taint effects, the NoExecute grace window, and why a label change never moves a running pod.

Quick Summary
Taints and tolerations decide which pods a node refuses; affinity decides which nodes a pod prefers. They are two independent filters applied in order: the scheduler first finds nodes that fit the pod's resource requests, then excludes nodes whose taints the pod does not tolerate, then ranks what remains by affinity, topology spread, and other scores.
A taint is on the node (key=value:Effect) and a toleration is on the pod. NoSchedule blocks new scheduling, PreferNoSchedule makes it a last resort, and NoExecute also evicts running pods that do not tolerate it — with an optional tolerationSeconds grace window. A toleration is permission, not a placement instruction: a pod that tolerates a taint may still land anywhere.
Node affinity has two forms with the same syntax: requiredDuringSchedulingIgnoredDuringExecution is a hard filter, preferredDuringSchedulingIgnoredDuringExecution is a weighted score. The "IgnoredDuringExecution" suffix is the important half — once a pod is placed, changing the node's labels never evicts it, which is why nodeSelector and affinity are placement rules and taints are the only thing that can remove a running pod.
Basic Syntax & Core Signatures
apiVersion: v1
kind: Pod
metadata:
name: gpu-trainer
spec:
tolerations:
- key: dedicated
operator: Equal # Equal | Exists
value: gpu
effect: NoSchedule
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 120 # tolerate for 2 min, then evict
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: accelerator
operator: In # In | NotIn | Exists | DoesNotExist | Gt | Lt
values: ["nvidia-a100"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values: ["ap-northeast-2a"]
containers:
- name: trainer
image: registry.example.com/ml/trainer:3.1
resources:
limits: { nvidia.com/gpu: 1 }# Node side
kubectl taint nodes node-7 dedicated=gpu:NoSchedule
kubectl taint nodes node-7 dedicated=gpu:NoSchedule- # remove (trailing dash)
kubectl label nodes node-7 accelerator=nvidia-a100
kubectl describe node node-7 | sed -n '/Taints/,/Unschedulable/p'# Why did this pod not schedule?
kubectl describe pod gpu-trainer | tail -6
# 0/8 nodes are available: 4 node(s) had untolerated taint {dedicated: gpu},
# 2 Insufficient nvidia.com/gpu, 2 node(s) didn't match Pod's node affinity/selector.
kubectl get pods -o wide --field-selector spec.nodeName=node-7Key Comparison Table
| Effect | Blocks new pods | Evicts running pods |
|---|---|---|
NoSchedule | Yes | No |
PreferNoSchedule | Soft preference against | No |
NoExecute | Yes | Yes, immediately or after tolerationSeconds |
| Mechanism | Set on | Purpose | Can evict later? |
|---|---|---|---|
| Taint / toleration | Node / pod | Repel pods from a node | NoExecute only |
nodeSelector | Pod | Exact label match | No |
| Required node affinity | Pod | Expression filter | No |
| Preferred node affinity | Pod | Weighted ranking | No |
nodeName | Pod | Bypasses the scheduler | No |
| Taint added automatically | Effect | Notes |
|---|---|---|
node.kubernetes.io/not-ready | NoExecute | 300s default toleration on pods |
node.kubernetes.io/unreachable | NoExecute | Same, tunable per pod |
node.kubernetes.io/disk-pressure | NoSchedule | Eviction by kubelet under pressure |
node.kubernetes.io/memory-pressure | NoSchedule | |
node.kubernetes.io/unschedulable | NoSchedule | Set by kubectl cordon |
node.kubernetes.io/network-unavailable | NoSchedule | until the CNI reports ready |
Practical Edge Cases & Deep Dives
1. Exists with no key tolerates everything
tolerations:
- operator: Exists # matches every taint, any key, any effectThis is how infrastructure DaemonSets run everywhere, and how a pod can end up on a cordoned node during maintenance. The narrower form is almost always correct:
tolerations:
- key: dedicated
operator: Exists # that key, any value, any effectWhen a pod still lands on a node you were draining, look for an operator: Exists toleration before suspecting the scheduler.
2. NoExecute and tolerationSeconds
A node becomes unreachable. The node controller adds node.kubernetes.io/unreachable:NoExecute, and every pod that does not tolerate it is evicted after the default toleration of 300 seconds. StatefulSets and quorum workloads often need longer:
tolerations:
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 600 # wait 10 minutes before giving uptolerationSeconds only matters with NoExecute; on a NoSchedule taint it is accepted and ignored. Shortening it makes failover faster and split-brain more likely — a 30-second window on a database will promote a replica while the old primary is still alive.
3. Affinity is not a placement guarantee after scheduling
IgnoredDuringExecution means a label change never moves the pod. Remove accelerator=nvidia-a100 from a node whose GPU failed and the pods running there stay put, keep failing, and no event reports a scheduling problem. Eviction and rescheduling is the operator's job (kubectl drain, or a controller that deletes the pod). The same asymmetry applies to topology: a pod scheduled for zone balance today is not rebalanced when a new node appears in another zone.
4. Combining taints with affinity for dedicated node pools
The robust pattern for a GPU or licensed-software pool is both: a NoSchedule taint keeps general workloads out, and required affinity keeps GPU pods on those nodes.
kubectl taint nodes -l pool=gpu dedicated=gpu:NoSchedule
kubectl label nodes -l pool=gpu accelerator=nvidia-a100With a taint alone, any pod with a broad Exists toleration can occupy the pool. With affinity alone, general pods with no affinity can still land there. Between the two there is a subtle failure: if the pool is full and the affinity is required, GPU pods go Pending rather than spilling onto ordinary nodes — usually what you want for licensing, rarely what you want for throughput. A preferred-affinity variant with a lower-priority fallback is the alternative.
5. Debugging the schedule decision
0/8 nodes are available:
4 node(s) had untolerated taint {dedicated: gpu},
2 Insufficient cpu,
2 node(s) didn't match Pod's node affinity/selector.Read it right to left: the largest group is usually the taint exclusion, and Insufficient is a resource problem, not a placement one. kubectl get nodes --show-labels plus kubectl describe node <node> | grep Taints reconstructs everything the scheduler saw. For a pod that scheduled but misbehaves, kubectl get pod -o wide gives the node, and the kubelet's own view is in journalctl -u kubelet.
Common Runtime & Configuration Mistakes
| Symptom | Cause | Fix |
|---|---|---|
Pod Pending, untolerated taint | Missing toleration | Add the specific key/effect |
| Tolerated but not scheduled | Affinity or resources still exclude the node | Check the other exclusion reasons |
| Pod on a cordoned node | operator: Exists toleration | Narrow the toleration |
| Pod evicted after ~5 minutes | NoExecute from node-not-ready | Raise tolerationSeconds |
| Quorum cluster splits | tolerationSeconds too short | Lengthen the window |
| GPU pods on CPU nodes | Affinity missing or preferred | Required affinity plus taint |
| General pods on GPU nodes | Taint missing, only labels | Add the NoSchedule taint |
| Node label change did not move pods | IgnoredDuringExecution | Drain and reschedule deliberately |
| DaemonSet missing on new nodes | Taint added to the node pool | Tolerate it in the template |
Insufficient cpu after adding affinity | Pool too small | Scale the pool, or relax affinity |
Taints answer "who may come here", affinity answers "where does this go", and neither is retroactive. Diagnose a placement problem by reading the scheduler's exclusion list, and diagnose a running-pod problem by reading taints and eviction events.
Try matching browser tools
All toolsSmall utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Kubernetes guides
All Kubernetes guideskubectl Output: jsonpath, custom-columns, and the Debugging Commands Worth Knowing
jsonpath for one value in a script, custom-columns for tables people read, and kubectl debug when the container has no shell. Includes the three quoting layers that break jsonpath.
Kubernetes Namespaces, ResourceQuotas, and LimitRanges
A namespace is only a boundary when quota and limit range make it one. Aggregate caps, per-pod defaults, the admission errors they produce, and namespaces stuck Terminating.
Kubernetes StatefulSets: Ordinal Identities, Headless Services, and PVC Lifecycle
Why each StatefulSet pod keeps a name, a DNS record, and its own PVC; how OrderedReady gates startup on readiness; and how partition-based updates canary a single quorum member.
Kubernetes securityContext, Capabilities, and Pod Security Standards
The five fields that remove most container escapes, which of them only exist at container level, and how the restricted profile enforces the same settings at admission.
More in Kubernetes
All Kubernetes guidesThe guides filed next to this one in Kubernetes, in library order.