Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes PodDisruptionBudgets and Graceful Node Drains

A PDB paces voluntary disruption only. The percentage rounding that silently blocks every drain, the eviction API versus kubectl delete, and unhealthy pods that lock a node.

~5 min readKubernetes
#Kubernetes#PDB#Availability#Operations
3D illustration of a shield gauge limiting pod evictions beside a node being drained

Quick Summary

A PodDisruptionBudget limits how many pods of a workload may be down at once because of voluntary disruptions: node drains, cluster upgrades, autoscaler scale-in, and anything else that calls the Eviction API. It says nothing about crashes, OOM kills, or a node dying — those are involuntary and the PDB has no vote.

minAvailable and maxUnavailable are two spellings of the same budget, and exactly one is allowed. minAvailable: 2 with replicas: 3 means one pod may be evicted at a time. Percentages round up for minAvailable and down for maxUnavailable, so maxUnavailable: 25% on three replicas is 0 — a budget that blocks every drain while looking reasonable.

The operational habit that matters: a PDB is only useful if it can be satisfied. A PDB requiring minAvailable: 1 on a single-replica Deployment makes kubectl drain hang forever, and a PDB that selects DaemonSet pods can block every node in the cluster from draining. Two replicas is the practical minimum for a workload that must survive a drain.

Basic Syntax & Core Signatures

yaml
apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: api-pdb namespace: shop spec: minAvailable: 2 # or maxUnavailable: 1 — not both unhealthyPodEvictionPolicy: IfHealthyBudget # or AlwaysAllow selector: matchLabels: { app: api }
yaml
# Percentage form: maxUnavailable rounds down spec: maxUnavailable: 25% # 3 replicas -> 0 allowed; 4 replicas -> 1 allowed selector: matchExpressions: - { key: app, operator: In, values: [api] }
bash
kubectl -n shop get pdb # NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE # api-pdb 2 N/A 1 6d kubectl -n shop describe pdb api-pdb kubectl drain node-7 --ignore-daemonsets --delete-emptydir-data --dry-run=server kubectl get pdb -A -o custom-columns=\ NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed

Key Comparison Table

FieldMeaningRounding
minAvailablePods that must stay availablePercentage rounds up
maxUnavailablePods allowed to be downPercentage rounds down
status.disruptionsAllowedEvictions currently permittedComputed live
status.currentHealthyHealthy pods matching the selectorLive
unhealthyPodEvictionPolicyWhether unhealthy pods may be evicted regardless of budgetIfHealthyBudget (default)
Voluntary disruptionPDB appliesInvoluntaryPDB applies
kubectl drainYesNode crashNo
Cluster upgrade / node rotationYesOOMKill, probe failureNo
Cluster Autoscaler scale-inYeskubectl delete podNo
Eviction API callYesSpot instance reclaimNo
SymptomMeaning
ALLOWED DISRUPTIONS 0 and drain hangsBudget cannot be met right now
Cannot evict pod as it would violate the pod's disruption budgetExpected refusal, not an error to retry
PDB with 0 healthy podsSomething else is broken; the budget follows
Drain never finishesPDB selects DaemonSet pods, or replicas below minAvailable

Practical Edge Cases & Deep Dives

1. Two replicas versus one: the arithmetic of a safe drain

text
replicas: 3, minAvailable: 2 -> 1 eviction allowed, always 2 serving replicas: 1, minAvailable: 1 -> 0 evictions; drain blocks until something else changes

A single-replica workload with minAvailable: 1 is not protected, it is frozen. The two honest options are minAvailable: 0 (accept downtime during drains, keep the budget only as documentation) or two replicas. For a workload that genuinely cannot run twice, maxUnavailable: 1 with replicas: 1 allows the eviction and explicitly says "this will be down for a moment" — which is what an operator needs to know before a cluster upgrade.

2. kubectl delete pod bypasses the budget

The PDB is enforced by the Eviction API (POST /api/v1/namespaces/{ns}/pods/{name}/eviction), not by deletion. kubectl delete pod removes the pod directly, and the Deployment recreates it; the budget never applies. That is why a script that "drains" by deleting pods ignores PDBs entirely, and why kubectl drain is the correct tool — it calls eviction and retries with backoff when the budget refuses.

bash
kubectl drain node-7 --ignore-daemonsets --delete-emptydir-data --timeout=10m # error: cannot delete Pods violating disruption budgets; use --disable-eviction to override

--disable-eviction is the escape hatch that turns a drain into deletions. It exists for a real emergency and should never appear in a runbook that runs unattended.

3. Unhealthy pods and unhealthyPodEvictionPolicy

Before this field, a workload with a CrashLooping pod could not be drained at all: the budget counted the broken pod as unavailable and refused every eviction. IfHealthyBudget (the default) allows evicting unhealthy pods only when the healthy count satisfies the budget; AlwaysAllow lets unhealthy pods be evicted regardless, which is what most operators want during a node rotation.

bash
kubectl -n shop get pdb api-pdb -o jsonpath=\ '{.status.currentHealthy}/{.status.desiredHealthy} allowed={.status.disruptionsAllowed}{"\n"}' # 1/2 allowed=0

4. PDBs during an upgrade: the "one node at a time" budget is per workload

A cluster upgrade drains nodes sequentially. If two workloads share the same three nodes and both need minAvailable: 2 out of 3, draining one node is fine; draining the next before the first workload's evicted pod is rescheduled elsewhere is not, and the upgrade stalls with the node cordoned and the drain retrying. Two conditions make that happen: insufficient capacity elsewhere for the evicted pod (it stays Pending), or required anti-affinity that forbids it from landing on the cordoned node's replacement.

bash
kubectl get pods -A --field-selector=status.phase=Pending kubectl describe pod <pending> # Insufficient cpu / node affinity

The fix is capacity headroom, not a bigger budget.

5. PDBs and DaemonSets, emptyDir, and local-path volumes

A PDB whose selector matches DaemonSet pods has desiredHealthy equal to the node count and can block drains cluster-wide, because DaemonSet pods are never rescheduled elsewhere — the eviction is refused and the drain waits for a pod that will not move. Keep infrastructure agents out of PDBs.

Pods with emptyDir or local-path volumes lose their data when evicted, so drain requires --delete-emptydir-data to acknowledge that. A PDB does not protect the data; it only paces the loss.

6. A drain that respects budgets, step by step

bash
kubectl cordon node-7 # stop new pods first kubectl get pdb -n shop # ALLOWED DISRUPTIONS per workload kubectl drain node-7 --ignore-daemonsets --delete-emptydir-data --timeout=15m kubectl uncordon node-7 # only after the node is healthy again

Cordoning first prevents the scheduler from filling the node while evictions are in flight. The --timeout matters: without it a drain retries forever on a budget that will never free up, and a rollout script that waits on it looks hung rather than failing loudly. --ignore-daemonsets is required whenever an agent runs on the node, and --delete-emptydir-data is the explicit acknowledgement that pods with temporary volumes will lose them.

Common Runtime & Configuration Mistakes

SymptomCauseFix
ALLOWED DISRUPTIONS 0 permanentlyminAvailable equals or exceeds healthy podsRelax the budget or add replicas
maxUnavailable: 25% blocks drainsPercentage rounds down to 0Use an absolute number
Drain hangs on a CrashLooping podUnhealthy pod counted against the budgetunhealthyPodEvictionPolicy: AlwaysAllow
PDB blocks cluster upgradeCapacity for evicted pods missingAdd headroom
PDB has no effectDeletions used instead of evictionUse kubectl drain
DaemonSet pods in a PDBSelector too broadNarrow the selector
Evictions succeed past minAvailableminAvailable plus rapid recreation racesAccept: the budget is a pace, not a lock
Two PDBs for one workloadOverlapping selectorsOne budget per workload
status emptyPDB applied before the workload existedRe-create or wait for the controller
minAvailable and maxUnavailable both setValidation rejects the objectKeep exactly one

A PodDisruptionBudget paces voluntary disruption; it does not make a workload highly available. The work of availability is in replicas, readiness, and spare capacity, and the budget is how you tell the cluster to respect that work while it does maintenance.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.