Kubernetes StatefulSets: Ordinal Identities, Headless Services, and PVC Lifecycle
Why each StatefulSet pod keeps a name, a DNS record, and its own PVC; how OrderedReady gates startup on readiness; and how partition-based updates canary a single quorum member.

Quick Summary
A StatefulSet is the workload for pods that need a stable identity: a fixed ordinal name (web-0, web-1), a predictable DNS record, and a volume that follows the pod across rescheduling. A Deployment gives you interchangeable replicas behind one Service; a StatefulSet gives you numbered members that each know who they are — which is exactly what a database, a broker, or a quorum-based cluster needs.
Two fields make the identity work. spec.serviceName points at a headless Service (clusterIP: None), which creates one DNS A record per pod instead of a load-balanced virtual IP. spec.volumeClaimTemplates creates one PVC per ordinal from the same template, so web-0 always mounts data-web-0.
Ordering is the other half. With the default podManagementPolicy: OrderedReady, pod N is created only after pod N-1 is Running and Ready, and scale-down removes from the highest ordinal. Updates are ordered too, from the highest ordinal down, and updateStrategy.rollingUpdate.partition pins ordinals at the old revision — the standard way to canary a quorum member.
Basic Syntax & Core Signatures
apiVersion: v1
kind: Service
metadata:
name: web
namespace: shop
spec:
clusterIP: None # headless: DNS per pod, no virtual IP
selector: { app: web }
ports:
- { name: http, port: 8080, targetPort: 8080 }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
namespace: shop
spec:
serviceName: web # required for network identity
replicas: 3
podManagementPolicy: OrderedReady # OrderedReady | Parallel
minReadySeconds: 5
selector:
matchLabels: { app: web }
updateStrategy:
type: RollingUpdate # RollingUpdate | OnDelete
rollingUpdate:
partition: 0 # ordinals >= partition get the new revision
persistentVolumeClaimRetentionPolicy:
whenScaled: Retain # Retain | Delete
whenDeleted: Retain # Retain | Delete
template:
metadata:
labels: { app: web }
spec:
terminationGracePeriodSeconds: 30
containers:
- name: web
image: registry.example.com/shop/web:1.9.3
ports: [{ name: http, containerPort: 8080 }]
env:
- name: POD_NAME
valueFrom: { fieldRef: { fieldPath: metadata.name } }
- name: PEERS
value: "web-0.web.shop.svc.cluster.local,web-1.web.shop.svc.cluster.local"
readinessProbe:
httpGet: { path: /healthz, port: http }
volumeMounts:
- { name: data, mountPath: /var/lib/web }
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: gp3
resources:
requests: { storage: 20Gi }# The controller names each claim <template>-<statefulset>-<ordinal>:
# data-web-0, data-web-1, data-web-2
kubectl -n shop get pvc -l app=web
kubectl -n shop rollout status statefulset/web --timeout=5mKey Comparison Table
| Aspect | Deployment | StatefulSet |
|---|---|---|
| Pod names | Random suffix (api-7d9f-2xk4p) | Ordinal (web-0, web-1) |
| DNS | One Service VIP or one headless set | pod-N.service.ns.svc per pod |
| Storage | Shared or none | One PVC per ordinal |
| Start / scale-down order | Arbitrary | Ordinal order (default) |
| Pause mechanism | maxUnavailable | partition ordinal threshold |
| Typical use | Stateless HTTP services | Databases, Kafka, ZooKeeper |
| Field | Default | Effect |
|---|---|---|
podManagementPolicy | OrderedReady | Serial create/delete, readiness-gated |
updateStrategy.type | RollingUpdate | Reverse-ordinal, one pod at a time |
rollingUpdate.partition | 0 | Ordinals below it are frozen at the old revision |
revisionHistoryLimit | 10 | ControllerRevisions kept for rollback |
persistentVolumeClaimRetentionPolicy | Retain | Whether PVCs are deleted with scale-down or deletion |
| Symptom | Meaning |
|---|---|
web-2 stays Pending, web-1 never starts | OrderedReady waiting on a lower ordinal |
Pod Running but 0/1 | Readiness failing; the next ordinal is blocked |
PVC Pending, WaitForFirstConsumer class | Normal until the pod schedules |
Practical Edge Cases & Deep Dives
1. Identity, DNS, and the headless Service contract
With clusterIP: None the Service has no VIP; DNS returns one A record per ready pod and each pod additionally gets <pod-name>.<service>.<namespace>.svc.cluster.local. That per-pod name is the identity peers use for replication and quorum.
A plain pod needs spec.subdomain: web to get that name; StatefulSet pods get it through serviceName. Forget the headless Service and pods still start, but peers cannot resolve each other. Because headless DNS returns pod IPs directly, clients that cache DNS keep stale addresses — members should address peers by name and re-resolve on failure, not pin an IP.
2. Volume lifecycle: PVCs outlive the pods and the StatefulSet
volumeClaimTemplates is not a "volume per pod" convenience; each PVC is a separate object that survives pod deletion, scale-down, and even StatefulSet deletion under the default Retain.
kubectl -n shop scale statefulset web --replicas=1
kubectl -n shop get pvc data-web-2 # still there, bound to no pod
kubectl -n shop scale statefulset web --replicas=3 # web-2 re-attaches itRecreating web with replicas: 3 re-attaches data-web-2 to the new web-2, which is what makes restart-safe state possible. It also means churn never frees disk: scale-up/down cycles leave orphaned claims behind. Set whenScaled: Delete when the data is disposable, and audit orphans with kubectl get pvc against the live pod list. volumeClaimTemplates is immutable, so a size bump is done per PVC (with allowVolumeExpansion) or by migration.
3. Ordered startup, and what a failing readiness probe really blocks
web-0 Pending -> Running 0/1 (readiness failing)
web-1 not created (OrderedReady)
web-2 not createdOrderedReady creates pod N only after pod N-1 is Ready, and deletes in reverse. That is a correctness feature for quorum members and a trap when readiness depends on a peer: a two-member cluster where each waits for the other never starts. That deadlock is a code fix (readiness meaning "alive enough to form quorum"), not a YAML fix; podManagementPolicy: Parallel gets both pods running and leaves convergence to the application.
Scale-down is equally orderly: 3 -> 1 terminates web-2, waits for it to be gone, then web-1, each with the full terminationGracePeriodSeconds. A slow shutdown therefore looks frozen for minutes per member; kubectl describe pod shows Deletion Timestamp while the container drains, which is how you tell draining from stuck.
4. Rolling updates, partition canaries, and OnDelete
An update proceeds from the highest ordinal down, recreating each pod with the new pod-template-hash revision and waiting for readiness before touching the next. partition: N freezes all ordinals below N at the current revision:
# Canary: upgrade only web-2, then move the boundary down
kubectl -n shop patch statefulset web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
kubectl -n shop set image statefulset/web web=registry.example.com/shop/web:2.0.0
kubectl -n shop rollout status statefulset/web
kubectl -n shop patch statefulset web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
# OnDelete: you delete pods yourself, in the order you choose
kubectl -n shop patch statefulset web -p '{"spec":{"updateStrategy":{"type":"OnDelete"}}}'
kubectl -n shop delete pod web-2 # recreated at the new revisionA pod that never becomes Ready stalls the rollout at that ordinal indefinitely, with no progressDeadlineSeconds equivalent. Pass --timeout to rollout status so automation fails instead of hanging, and roll back with kubectl rollout undo --to-revision=N, which itself follows ordinal order.
5. The selector and serviceName fields are immutable
The StatefulSet "web" is invalid: spec: Forbidden: updates to statefulset spec
for fields other than 'replicas', 'ordinal', 'template', 'updateStrategy',
'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbiddenselector, serviceName, and volumeClaimTemplates cannot be edited. A label typo or a Service rename therefore needs kubectl delete statefulset web --cascade=orphan, an edited manifest, and a re-apply so the orphaned pods are adopted by matching labels. Orphaning keeps the running pods and their PVCs alive, which is the safe route for a stateful workload.
6. When a StatefulSet pod cannot be rescheduled
Warning FailedAttachVolume Multi-Attach error for volume "pvc-6c1f..."
Volume is already used by pod(s) web-1 on node-ip-10-0-3-77An RWO PVC attaches to one node at a time. If a node dies without detaching (the cloud timeout is typically minutes) the replacement pod waits. Confirm with kubectl get volumeattachment before forcing anything. Under WaitForFirstConsumer binding a PVC also stays Pending until a pod using it schedules — expected, not broken.
Common Runtime & Configuration Mistakes
| Symptom | Cause | Fix |
|---|---|---|
Peers cannot resolve web-0 | Service missing or not headless | clusterIP: None, matching serviceName |
web-1 never created | web-0 not Ready | kubectl describe pod web-0 |
Rollout stopped at web-2 | That ordinal's probe fails | rollout undo, fix probe |
| Disk keeps filling after scale-down | PVCs retained by default | whenScaled: Delete, or prune claims |
Multi-Attach error | RWO volume attached to a dead node | Wait for detach, check volumeattachment |
spec.selector: Invalid value | Selector edited | Recreate with --cascade=orphan |
The mental model that prevents most of these: a StatefulSet is an ordering machine around a stable name-to-storage mapping. Every ordinal is a separate identity that keeps its own claim, and the controller will not move past an ordinal that is not ready. When something looks stuck, ask which ordinal is holding the line and read that pod's conditions first.
Try matching browser tools
All toolsSmall utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Kubernetes guides
All Kubernetes guideskubectl Output: jsonpath, custom-columns, and the Debugging Commands Worth Knowing
jsonpath for one value in a script, custom-columns for tables people read, and kubectl debug when the container has no shell. Includes the three quoting layers that break jsonpath.
Kubernetes Namespaces, ResourceQuotas, and LimitRanges
A namespace is only a boundary when quota and limit range make it one. Aggregate caps, per-pod defaults, the admission errors they produce, and namespaces stuck Terminating.
Kubernetes securityContext, Capabilities, and Pod Security Standards
The five fields that remove most container escapes, which of them only exist at container level, and how the restricted profile enforces the same settings at admission.
Kubernetes PodDisruptionBudgets and Graceful Node Drains
A PDB paces voluntary disruption only. The percentage rounding that silently blocks every drain, the eviction API versus kubectl delete, and unhealthy pods that lock a node.
More in Kubernetes
All Kubernetes guidesThe guides filed next to this one in Kubernetes, in library order.