Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes DaemonSets: Node Selectors, Taints, and Rolling Update Strategies

One pod per eligible node, decided by labels and taints rather than replicas. How maxUnavailable and maxSurge actually behave, and why an agent can be missing on a brand-new node.

~5 min readKubernetes
#Kubernetes#DaemonSet#Scheduling#Operations
3D illustration of a stack of server nodes each running one identical DaemonSet agent cube

Quick Summary

A DaemonSet runs exactly one pod per eligible node. You do not set replicas; the controller watches the node list and schedules a pod wherever the pod template's node constraints match. Log shippers, CNI plugins, kube-proxy, node exporters, and storage drivers are the canonical cases — anything that must be present on a machine rather than behind a Service.

Eligibility is decided by ordinary scheduling rules: nodeSelector matches labels exactly, nodeAffinity allows required and preferred forms, and taints are the other half. A node with node-role.kubernetes.io/control-plane:NoSchedule receives no DaemonSet pod unless the template tolerates that taint. The controller adds a small set of tolerations automatically (not-ready, unreachable, disk-pressure, memory-pressure, pid-pressure, unschedulable), which is why a daemon pod stays on a cordoned node.

Updates are node-scoped rollouts. RollingUpdate with maxUnavailable (default 1) or maxSurge replaces pods a few nodes at a time, in no guaranteed order. OnDelete hands the timing to you — the pattern for CNI and device plugins, where a bad upgrade takes the node's networking with it.

Basic Syntax & Core Signatures

yaml
apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: monitoring spec: selector: matchLabels: { app: node-exporter } updateStrategy: type: RollingUpdate # RollingUpdate | OnDelete rollingUpdate: maxUnavailable: 10% # default 1; integer or percentage # maxSurge: 5% # alternative, cannot be combined minReadySeconds: 10 revisionHistoryLimit: 5 template: metadata: labels: { app: node-exporter } spec: priorityClassName: system-node-critical tolerations: - operator: Exists # every taint: control-plane and GPU nodes too nodeSelector: kubernetes.io/os: linux affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/worker operator: Exists containers: - name: exporter image: quay.io/prometheus/node-exporter:v1.8.2 args: ["--path.rootfs=/host"] ports: [{ name: metrics, containerPort: 9100, hostPort: 9100 }] resources: requests: { cpu: 50m, memory: 64Mi } limits: { memory: 128Mi } securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65534 volumeMounts: - { name: root, mountPath: /host, readOnly: true } volumes: - name: root hostPath: { path: / }
bash
kubectl -n monitoring get daemonset node-exporter # NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE # node-exporter 5 5 5 5 5 kubernetes.io/os=linux 12d kubectl -n monitoring get pods -l app=node-exporter -o wide kubectl -n monitoring rollout status daemonset/node-exporter kubectl -n monitoring rollout undo daemonset/node-exporter # Why is there no pod on node-3? kubectl describe node node-3 | sed -n '/Taints/,/^$/p' kubectl -n monitoring get events --field-selector \ involvedObject.name=node-exporter --sort-by=.lastTimestamp

Key Comparison Table

FieldDaemonSetJobDeployment
Unit of placementNodeClusterCluster
replicasNot set (derived from nodes)parallelism/completionsreplicas
Typical podAgent, driver, shipperBatch taskStateless service
Node taint defaultTolerates key node taintsRespects taintsRespects taints
Update knobMeaningWatch out
maxUnavailablePods unavailable at onceDefault 1; percentage rounds down to at least 1
maxSurgeExtra pods beyond node countNeeds IP/hostPort room; mutually exclusive
OnDeleteNo automatic replacementYou delete pods; order is yours
revisionHistoryLimitControllerRevisions retainedNeeded for rollout undo
SymptomMeaning
DESIRED below node countNode constraints or taints exclude nodes
Pod Pending with 0/N nodes are availableTaints or node affinity
hostPort collision, pod stuck ContainerCreatingAnother pod owns that port on the node

Practical Edge Cases & Deep Dives

1. Taints decide placement more often than labels do

A DaemonSet pod is scheduled like any other pod, so a NoSchedule taint on the node wins over a matching nodeSelector. Managed node pools routinely ship with taints (dedicated=gpu:NoSchedule, node-role.kubernetes.io/control-plane:NoSchedule), and the event is explicit:

text
Warning FailedScheduling 0/6 nodes are available: 3 node(s) had untolerated taint {dedicated: gpu}, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }

Two valid answers: tolerate that specific taint with key: dedicated, operator: Equal, value: gpu, effect: NoSchedule, or leave it untolerated so the plugin does not land on nodes it cannot use. operator: Exists with an empty key tolerates everything — convenient for infrastructure agents, dangerous for anything expecting a specific runtime, because it also overrides a manual maintenance:NoSchedule taint during an incident.

The controller adds tolerations to the pod spec at creation (not-ready, unreachable, disk-pressure, memory-pressure, pid-pressure, unschedulable), which is why draining a node shows the daemon pod as ignored rather than evicted.

2. DESIRED counts nodes, not pods — and hostPort fights the scheduler

DESIRED is computed from nodes that satisfy the template's constraints. If DESIRED: 4 on a six-node cluster, two nodes are excluded before any pod exists — read kubectl describe node for taints and labels rather than debugging the pods.

The subtle trap is hostPort. The scheduler only sees a requested hostPort when it is declared; with 9100 already taken, the pod schedules and then fails at sandbox creation with port 9100 is already allocated. Prefer a Service with internalTrafficPolicy: Local over hostPort when you need node IP addressing; if you keep it, make the port unique per cluster and keep maxSurge: 0 so two pods never compete for the same node port during a rollout.

3. maxSurge versus maxUnavailable under real constraints

maxUnavailable: 1 is the default and the safest choice for agents that modify the node (CNI, kube-proxy, device plugins): the old pod is deleted, then the new one starts. The cost is that a broken image rolls through the cluster one node at a time, and each node passes through a degraded window.

maxSurge starts the new pod before removing the old one, which is zero-downtime — but two pods on one node means two hostPort bindings, double memory, and CURRENT = DESIRED + surge. It is viable only for pods without node-exclusive resources. If the surge pod cannot schedule, the controller retries and the rollout never completes (error: timed out waiting for the condition).

4. Missing agents on new nodes: labels, taints, and tolerations drift

A node joins, and no pod appears. In order: does the node carry the label the DaemonSet selects, does it hold a taint the template does not tolerate, is the node Ready? A NotReady node still gets a daemon pod scheduled, so "no pod at all" points at labels or taints while "pod exists but 0/1" points at the container.

bash
kubectl get node node-4 -o jsonpath='{.metadata.labels}{"\n"}{.spec.taints}{"\n"}' kubectl -n monitoring get pods -o wide --field-selector spec.nodeName=node-4

Autoscaled clusters add this bug routinely: the node template the autoscaler uses is missing a label the DaemonSet requires, so the cluster grows capacity with no logging agent. Assert the label in the node group template, not in a post-provisioning script.

5. DaemonSet pods, eviction, and PDBs

DaemonSet pods ignore the unschedulable taint, so kubectl drain deletes them directly, and a PodDisruptionBudget that selects them can block a drain permanently because their desired count equals the node count. Do not put infrastructure agents in a PDB meant for application availability. During an upgrade the DaemonSet's own maxUnavailable is the availability control: 10% on a 100-node cluster upgrades ten nodes of networking at once, and a bad CNI build takes a tenth of the cluster off the network before you can stop it. OnDelete plus a staged loop over node groups is the slower, safer pattern for that class of component.

6. Static pods are not DaemonSets

/etc/kubernetes/manifests on a kubelet produces static pods: owned by a Node, not scheduled, and invisible to kubectl rollout. Control-plane components are usually static pods in kubeadm clusters while add-ons like kube-proxy are DaemonSets. A static pod is updated by changing the file on the node and letting the kubelet restart it, so neither rollout undo nor revisionHistoryLimit applies — a confusion that costs an hour during control-plane upgrades.

Common Runtime & Configuration Mistakes

SymptomCauseFix
DESIRED below node countTaints or node affinity exclude nodesAdd toleration or fix labels
No pod on a new nodeNode template missing the selector labelFix the node group template
Pending: untolerated taintMissing taint toleranceExplicit toleration, not Exists
ContainerCreating foreverhostPort already boundUnique ports, or hostNetwork
Rollout never finishesSurge pod cannot schedule or is not ReadyUse maxUnavailable
Pod evicted under memory pressureNo requests, low priorityrequests, system-node-critical
Two agents on one nodemaxSurge with node-exclusive resourceDrop surge for that DaemonSet
DaemonSet shows 0 desiredSelector does not match template labelsAlign selector and pod labels

A DaemonSet is a set of node constraints with a pod attached. When a node lacks its pod, the answer is in the node's labels, the node's taints, or the pod's schedule event. When a rollout misbehaves, the answer is in maxSurge/maxUnavailable and whether the new pod can be Ready where it lands.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.