Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

PersistentVolumes, PVCs, StorageClasses, and Access Modes (RWO/RWX)

RWO is one node, not one pod. WaitForFirstConsumer is not a stuck PVC. Binding, dynamic provisioning, expansion, reclaim policies, and the Multi-Attach error that stops a second replica.

~5 min readKubernetes
#Kubernetes#Storage#StatefulSets
3D illustration of a storage class gear provisioning a disk volume that binds to a claim attached to a pod

Quick Summary

A PersistentVolume (PV) is storage described as a cluster resource; a PersistentVolumeClaim (PVC) is a request for storage by a workload; a StorageClass describes how to create PVs on demand. With dynamic provisioning, a workload declares a PVC, the StorageClass's CSI driver creates the underlying disk, a PV is generated to represent it, and the PVC binds to that PV. With static provisioning, an administrator creates PVs by hand and PVCs bind to the matching one.

Binding is a match on three things: capacity (the PV must be at least the request), access modes (the PV must support the requested mode), and storage class (unless the PVC pins a specific volumeName). volumeBindingMode: WaitForFirstConsumer delays provisioning until a pod using the PVC is scheduled, which is what keeps the disk in the same zone as the node — the reason a Pending PVC that becomes bound only after a pod exists is correct behaviour, not a bug.

Access modes describe node-level attachment, not application semantics: ReadWriteOnce is one node, ReadOnlyMany and ReadWriteMany require a shared filesystem (NFS, CephFS, EFS, Azure Files), and ReadWriteOncePod is one pod. This guide covers the PV/PVC/StorageClass fields, binding and provisioning flows, expansion and reclaim policies, the Pending PVC debugging sequence, and the StatefulSet volume patterns.

Basic Syntax & Core Signatures

yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-ssd annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: ebs.csi.aws.com # or pd.csi.storage.gke.io, disk.csi.azure.com, ... parameters: type: gp3 fsType: ext4 reclaimPolicy: Delete # Delete | Retain volumeBindingMode: WaitForFirstConsumer # Immediate | WaitForFirstConsumer allowVolumeExpansion: true
yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data spec: accessModes: [ReadWriteOnce] storageClassName: fast-ssd volumeMode: Filesystem # or Block resources: requests: storage: 20Gi
yaml
apiVersion: v1 kind: PersistentVolume # static provisioning: created by an admin metadata: name: pv-nfs-01 spec: capacity: storage: 100Gi accessModes: [ReadWriteMany] persistentVolumeReclaimPolicy: Retain storageClassName: nfs nfs: server: 10.0.0.20 path: /exports/app mountOptions: [nfsvers=4.1, hard, timeo=600]
yaml
# StatefulSet: one PVC per replica, named <template>-<statefulset>-<ordinal> volumeClaimTemplates: - metadata: { name: data } spec: accessModes: [ReadWriteOnce] storageClassName: fast-ssd resources: { requests: { storage: 50Gi } }
bash
kubectl get storageclass kubectl get pv,pvc -A kubectl describe pvc data kubectl describe pv pv-nfs-01 kubectl get pvc data -o jsonpath='{.spec.volumeName}{"\n"}{.status.phase}{"\n"}' kubectl patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'

Key Comparison Table

ModeShortMeaningRequires
ReadWriteOnceRWORead-write by a single nodeBlock storage (EBS, PD, local)
ReadOnlyManyROXRead-only by many nodesShared filesystem support
ReadWriteManyRWXRead-write by many nodesNFS, CephFS, EFS, Azure Files
ReadWriteOncePodRWOPRead-write by a single podCSI driver support
SettingWhereEffect
reclaimPolicy: DeleteStorageClass / PVUnderlying disk deleted with the PVC
reclaimPolicy: RetainStorageClass / PVPV goes Released, data kept, manual cleanup
volumeBindingMode: ImmediateStorageClassProvision as soon as the PVC exists
volumeBindingMode: WaitForFirstConsumerStorageClassProvision when a pod is scheduled (zone-aware)
allowVolumeExpansionStorageClassPVC can grow
volumeMode: BlockPVCRaw block device, no filesystem
storageClassName: ""PVCNo dynamic provisioning; bind to a classless PV
PVC phaseMeaning
PendingNot bound yet: no PV, waiting for a consumer, or the provisioner failed
BoundAttached to a PV
LostThe bound PV disappeared

Practical Edge Cases & Deep Dives

1. A Pending PVC with WaitForFirstConsumer is not broken

bash
$ kubectl get pvc data NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS data Pending fast-ssd $ kubectl describe pvc data | tail -3 Events: <none> # no events at all

With WaitForFirstConsumer, no PV is created until a pod references the PVC and is scheduled, because only then is the node (and its zone) known. describe pvc shows no events and the provisioner logs show nothing, which looks like a failure and is not. Create the pod, and the PVC binds. With Immediate, a Pending PVC means the opposite: the provisioner ran and failed, and its error is in the events (waiting for a volume to be created) and in the CSI controller's logs.

bash
kubectl get events --field-selector involvedObject.name=data --sort-by=.lastTimestamp kubectl -n kube-system logs deploy/ebs-csi-controller -c csi-provisioner --tail=50

2. RWO is per node, not per pod

text
Two pods on the SAME node mounting the same RWO PVC: allowed. Two pods on DIFFERENT nodes mounting the same RWO PVC: the second stays ContainerCreating with a Multi-Attach error.
bash
$ kubectl describe pod api-2 | grep -A2 Multi-Attach Warning FailedAttachVolume ... Multi-Attach error for volume "pvc-3f2a..." Volume is already exclusively attached to one node and can't be attached to another

This is the classic consequence of scaling a Deployment that mounts a persistent disk: the first replica schedules on one node, the second on another, and the second pod never starts. The answers are a workload that does not need shared storage (object storage, a database), ReadWriteMany on a shared filesystem, or a StatefulSet where each replica has its own PVC. A Deployment with an RWO PVC and more than one replica is nearly always a mistake.

3. Retain leaves a Released PV that nothing can bind

bash
$ kubectl get pv NAME CAPACITY STATUS CLAIM RECLAIMPOLICY pv-data 100Gi Released default/data Retain $ kubectl delete pvc data $ kubectl get pv # still there, still Released, still holding the data

Retain is the right policy for data that must survive an accidental PVC deletion — and it means the PV keeps a claimRef to the deleted claim, so no new PVC can bind to it. Reclaiming it for reuse is a deliberate two-step:

bash
kubectl patch pv pv-data -p '{"spec":{"claimRef": null}}' kubectl delete pv pv-data # after copying the data somewhere if it is needed

Deleting the PV after removing the reference deletes the underlying disk according to the driver, so copy first. The reverse mistake is Delete on a StorageClass holding a database: kubectl delete pvc then removes the disk, and no amount of Kubernetes troubleshooting brings it back.

4. Growing a PVC: what expansion actually requires

bash
$ kubectl patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}' $ kubectl get pvc data NAME STATUS VOLUME CAPACITY ACCESS MODES data Bound pvc-... 20Gi RWO # request updated, capacity not yet $ kubectl describe pvc data | grep -i -A2 'FileSystemResizePending'

Three conditions must hold: the StorageClass has allowVolumeExpansion: true, the driver supports expansion, and the filesystem supports online growth (ext4/xfs do; some drivers need the pod restarted to finish the resize). Shrinking is never supported — a smaller request is rejected by the API server — so a mistyped size costs a new PVC and a data copy. With WaitForFirstConsumer, the resize similarly waits for the next pod start on some drivers.

5. StatefulSets, volumeClaimTemplates, and the PVCs that outlive the app

bash
$ kubectl get pvc -l app=db NAME STATUS VOLUME CAPACITY data-db-0 Bound pvc-9a1c 50Gi data-db-1 Bound pvc-b3d2 50Gi

Each replica gets a stable PVC named from the template, the StatefulSet and the ordinal; deleting the StatefulSet does not delete those PVCs (by design, to protect data), so a helm uninstall leaves disks behind and a reinstall rebinds to them. Scaling down keeps the PVCs of the removed ordinals, which is why scaling back up reattaches the same data. Cleanup is explicit:

bash
kubectl delete pvc -l app=db # after the StatefulSet is gone

6. The default class, and the PVC that picks the wrong one

bash
$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE fast-ssd (default) ebs.csi.aws.com Delete WaitForFirstConsumer slow-hdd ebs.csi.aws.com Retain Immediate $ kubectl get pvc data -o jsonpath='{.spec.storageClassName}{"\n"}' slow-hdd

A PVC with no storageClassName gets the default class at creation time; if the cluster's default changes later, existing PVCs keep the class they were created with. Setting storageClassName: "" explicitly opts out of dynamic provisioning entirely, which is what static PVs without a class need. Two classes marked default make the cluster choose arbitrarily, and the API server only warns.

Common Runtime & Configuration Mistakes

SymptomCauseFix
PVC Pending with WaitForFirstConsumerNo consumer pod yetCreate the pod; this is expected
PVC Pending with eventsProvisioner error, quota, unsupported parametersRead the events, then the CSI controller logs
PVC Pending, no events, ImmediateNo matching PV for a static claimCapacity, access mode, class, selector
Multi-Attach error on a second podRWO attached to another nodeOne replica, RWX storage, or a StatefulSet
FailedMount on a podPVC not bound, or the node cannot reach the storagedescribe pod, CSI node plugin logs
PVC stuck TerminatingA pod or finalizer still uses itDelete the pod first
Resize rejectedallowVolumeExpansion falseEnable it on the StorageClass
Capacity unchanged after resizeFilesystem resize pendingRestart the pod, check driver support
Shrink rejectedNot supported by any driverNew smaller PVC and copy
PV Released and unboundableRetain with a stale claimRefClear claimRef, then reuse or delete
Data gone after helm uninstallreclaimPolicy: DeleteRetain for stateful data
RWX claim never bindsStorage backend has no shared filesystemNFS/CephFS/EFS class
PVC binds to an unexpected classDefault class, or an explicit one in the templateSet storageClassName explicitly

Storage in Kubernetes is safe when the policy is chosen before the data exists: Retain for anything that cannot be regenerated, WaitForFirstConsumer for zone-aware disks, RWO with a single replica (or a StatefulSet) rather than a Deployment, and expansion enabled from the start. Almost every storage incident is a Pending or Multi-Attach that traces back to one of those four settings being the default instead of a decision.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.