Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes StatefulSets: Ordinal Identities, Headless Services, and PVC Lifecycle

Why each StatefulSet pod keeps a name, a DNS record, and its own PVC; how OrderedReady gates startup on readiness; and how partition-based updates canary a single quorum member.

~5 min readKubernetes
#Kubernetes#StatefulSet#Storage#Operations
3D illustration of numbered StatefulSet pod cubes each bound to its own storage cylinder and headless DNS

Quick Summary

A StatefulSet is the workload for pods that need a stable identity: a fixed ordinal name (web-0, web-1), a predictable DNS record, and a volume that follows the pod across rescheduling. A Deployment gives you interchangeable replicas behind one Service; a StatefulSet gives you numbered members that each know who they are — which is exactly what a database, a broker, or a quorum-based cluster needs.

Two fields make the identity work. spec.serviceName points at a headless Service (clusterIP: None), which creates one DNS A record per pod instead of a load-balanced virtual IP. spec.volumeClaimTemplates creates one PVC per ordinal from the same template, so web-0 always mounts data-web-0.

Ordering is the other half. With the default podManagementPolicy: OrderedReady, pod N is created only after pod N-1 is Running and Ready, and scale-down removes from the highest ordinal. Updates are ordered too, from the highest ordinal down, and updateStrategy.rollingUpdate.partition pins ordinals at the old revision — the standard way to canary a quorum member.

Basic Syntax & Core Signatures

yaml
apiVersion: v1 kind: Service metadata: name: web namespace: shop spec: clusterIP: None # headless: DNS per pod, no virtual IP selector: { app: web } ports: - { name: http, port: 8080, targetPort: 8080 } --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web namespace: shop spec: serviceName: web # required for network identity replicas: 3 podManagementPolicy: OrderedReady # OrderedReady | Parallel minReadySeconds: 5 selector: matchLabels: { app: web } updateStrategy: type: RollingUpdate # RollingUpdate | OnDelete rollingUpdate: partition: 0 # ordinals >= partition get the new revision persistentVolumeClaimRetentionPolicy: whenScaled: Retain # Retain | Delete whenDeleted: Retain # Retain | Delete template: metadata: labels: { app: web } spec: terminationGracePeriodSeconds: 30 containers: - name: web image: registry.example.com/shop/web:1.9.3 ports: [{ name: http, containerPort: 8080 }] env: - name: POD_NAME valueFrom: { fieldRef: { fieldPath: metadata.name } } - name: PEERS value: "web-0.web.shop.svc.cluster.local,web-1.web.shop.svc.cluster.local" readinessProbe: httpGet: { path: /healthz, port: http } volumeMounts: - { name: data, mountPath: /var/lib/web } volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] storageClassName: gp3 resources: requests: { storage: 20Gi }
bash
# The controller names each claim <template>-<statefulset>-<ordinal>: # data-web-0, data-web-1, data-web-2 kubectl -n shop get pvc -l app=web kubectl -n shop rollout status statefulset/web --timeout=5m

Key Comparison Table

AspectDeploymentStatefulSet
Pod namesRandom suffix (api-7d9f-2xk4p)Ordinal (web-0, web-1)
DNSOne Service VIP or one headless setpod-N.service.ns.svc per pod
StorageShared or noneOne PVC per ordinal
Start / scale-down orderArbitraryOrdinal order (default)
Pause mechanismmaxUnavailablepartition ordinal threshold
Typical useStateless HTTP servicesDatabases, Kafka, ZooKeeper
FieldDefaultEffect
podManagementPolicyOrderedReadySerial create/delete, readiness-gated
updateStrategy.typeRollingUpdateReverse-ordinal, one pod at a time
rollingUpdate.partition0Ordinals below it are frozen at the old revision
revisionHistoryLimit10ControllerRevisions kept for rollback
persistentVolumeClaimRetentionPolicyRetainWhether PVCs are deleted with scale-down or deletion
SymptomMeaning
web-2 stays Pending, web-1 never startsOrderedReady waiting on a lower ordinal
Pod Running but 0/1Readiness failing; the next ordinal is blocked
PVC Pending, WaitForFirstConsumer classNormal until the pod schedules

Practical Edge Cases & Deep Dives

1. Identity, DNS, and the headless Service contract

With clusterIP: None the Service has no VIP; DNS returns one A record per ready pod and each pod additionally gets <pod-name>.<service>.<namespace>.svc.cluster.local. That per-pod name is the identity peers use for replication and quorum.

A plain pod needs spec.subdomain: web to get that name; StatefulSet pods get it through serviceName. Forget the headless Service and pods still start, but peers cannot resolve each other. Because headless DNS returns pod IPs directly, clients that cache DNS keep stale addresses — members should address peers by name and re-resolve on failure, not pin an IP.

2. Volume lifecycle: PVCs outlive the pods and the StatefulSet

volumeClaimTemplates is not a "volume per pod" convenience; each PVC is a separate object that survives pod deletion, scale-down, and even StatefulSet deletion under the default Retain.

bash
kubectl -n shop scale statefulset web --replicas=1 kubectl -n shop get pvc data-web-2 # still there, bound to no pod kubectl -n shop scale statefulset web --replicas=3 # web-2 re-attaches it

Recreating web with replicas: 3 re-attaches data-web-2 to the new web-2, which is what makes restart-safe state possible. It also means churn never frees disk: scale-up/down cycles leave orphaned claims behind. Set whenScaled: Delete when the data is disposable, and audit orphans with kubectl get pvc against the live pod list. volumeClaimTemplates is immutable, so a size bump is done per PVC (with allowVolumeExpansion) or by migration.

3. Ordered startup, and what a failing readiness probe really blocks

text
web-0 Pending -> Running 0/1 (readiness failing) web-1 not created (OrderedReady) web-2 not created

OrderedReady creates pod N only after pod N-1 is Ready, and deletes in reverse. That is a correctness feature for quorum members and a trap when readiness depends on a peer: a two-member cluster where each waits for the other never starts. That deadlock is a code fix (readiness meaning "alive enough to form quorum"), not a YAML fix; podManagementPolicy: Parallel gets both pods running and leaves convergence to the application.

Scale-down is equally orderly: 3 -> 1 terminates web-2, waits for it to be gone, then web-1, each with the full terminationGracePeriodSeconds. A slow shutdown therefore looks frozen for minutes per member; kubectl describe pod shows Deletion Timestamp while the container drains, which is how you tell draining from stuck.

4. Rolling updates, partition canaries, and OnDelete

An update proceeds from the highest ordinal down, recreating each pod with the new pod-template-hash revision and waiting for readiness before touching the next. partition: N freezes all ordinals below N at the current revision:

bash
# Canary: upgrade only web-2, then move the boundary down kubectl -n shop patch statefulset web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}' kubectl -n shop set image statefulset/web web=registry.example.com/shop/web:2.0.0 kubectl -n shop rollout status statefulset/web kubectl -n shop patch statefulset web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}' # OnDelete: you delete pods yourself, in the order you choose kubectl -n shop patch statefulset web -p '{"spec":{"updateStrategy":{"type":"OnDelete"}}}' kubectl -n shop delete pod web-2 # recreated at the new revision

A pod that never becomes Ready stalls the rollout at that ordinal indefinitely, with no progressDeadlineSeconds equivalent. Pass --timeout to rollout status so automation fails instead of hanging, and roll back with kubectl rollout undo --to-revision=N, which itself follows ordinal order.

5. The selector and serviceName fields are immutable

text
The StatefulSet "web" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinal', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

selector, serviceName, and volumeClaimTemplates cannot be edited. A label typo or a Service rename therefore needs kubectl delete statefulset web --cascade=orphan, an edited manifest, and a re-apply so the orphaned pods are adopted by matching labels. Orphaning keeps the running pods and their PVCs alive, which is the safe route for a stateful workload.

6. When a StatefulSet pod cannot be rescheduled

text
Warning FailedAttachVolume Multi-Attach error for volume "pvc-6c1f..." Volume is already used by pod(s) web-1 on node-ip-10-0-3-77

An RWO PVC attaches to one node at a time. If a node dies without detaching (the cloud timeout is typically minutes) the replacement pod waits. Confirm with kubectl get volumeattachment before forcing anything. Under WaitForFirstConsumer binding a PVC also stays Pending until a pod using it schedules — expected, not broken.

Common Runtime & Configuration Mistakes

SymptomCauseFix
Peers cannot resolve web-0Service missing or not headlessclusterIP: None, matching serviceName
web-1 never createdweb-0 not Readykubectl describe pod web-0
Rollout stopped at web-2That ordinal's probe failsrollout undo, fix probe
Disk keeps filling after scale-downPVCs retained by defaultwhenScaled: Delete, or prune claims
Multi-Attach errorRWO volume attached to a dead nodeWait for detach, check volumeattachment
spec.selector: Invalid valueSelector editedRecreate with --cascade=orphan

The mental model that prevents most of these: a StatefulSet is an ordering machine around a stable name-to-storage mapping. Every ordinal is a separate identity that keeps its own claim, and the controller will not move past an ordinal that is not ready. When something looks stuck, ask which ordinal is holding the line and read that pod's conditions first.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.