Kubernetes Multi-Attach error: the volume is still attached elsewhere
One claim, two nodes. Find the current holder, clear a stale attachment after a node failure, and give the volume a single writer.

Introduction & Quick Summary
A pod stays in ContainerCreating and kubectl describe pod reports Multi-Attach error for volume "pvc-8f2c..." Volume is already used by pod(s) web-7d9f-abcde on node ip-10-0-3-14, or the more common wrapper Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-2xq7]: timed out waiting for the condition. The application is healthy in every other respect; it simply never starts.
Both messages mean the same thing: the volume's access mode allows one node at a time, and the storage plugin believes another node still has it. The old consumer may be a terminating pod, a pod on a node that died without detaching, a second replica of a Deployment sharing one PVC, or a StatefulSet rollout that briefly runs two pods that both need a ReadWriteOnce claim.
The order that resolves it fastest:
- Find the current holder:
kubectl get volumeattachment, andkubectl get pod -o widefor the pods named in the error. - If the holder is terminating on a dead node, delete the stuck pod and then the stale
VolumeAttachment, so the attach/detach controller can finish. - Fix the shape: one replica per
ReadWriteOnceclaim,ReadWriteOncePodwhen the CSI driver supports it, and a rollout strategy that does not overlap pods on different nodes.
Recreating the PVC does not help and can lose data. --force deleting a pod without clearing the attachment usually reproduces the error on the next node.
Core Mechanics, Signatures & Reproducible Setup
Kubernetes splits storage into a control-plane attach step and a node-local mount step. The attach/detach controller, working with the CSI driver, attaches a volume to one node at a time for ReadWriteOnce and ReadWriteOncePod. The kubelet on that node then stages and mounts it. Attachments are recorded as VolumeAttachment objects, and they are only removed when the controller observes that no pod needs the volume any more.
kubectl get pvc data -o jsonpath='{.spec.accessModes}{"\n"}{.status.phase}{"\n"}'
kubectl get volumeattachment -o custom-columns=\
NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached
kubectl get pods -A -o wide --field-selector spec.nodeName=ip-10-0-3-14 | grep -i webA minimal reproduction on a multi-node cluster with a CSI driver that enforces single attach:
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
replicas: 2 # two pods, one ReadWriteOnce claim
selector: { matchLabels: { app: web } }
template:
metadata: { labels: { app: web } }
spec:
containers:
- name: app
image: ghcr.io/acme/web:1.4.0
volumeMounts: [{ name: data, mountPath: /data }]
volumes:
- name: data
persistentVolumeClaim: { claimName: data } # RWO, single nodeThe second replica lands on another node, the driver refuses the second attach, and both pods report the error until the scheduler gives up on one of them.
Key Comparison Table
| Access mode | Concurrent pods | Concurrent nodes | Use for |
|---|---|---|---|
| ReadWriteOnce | many on one node | one | a Deployment with one replica |
| ReadWriteOncePod | one | one | stateful singletons, strict fencing |
| ReadOnlyMany | many | many | shared assets |
| ReadWriteMany | many | many | NFS, EFS, CephFS only |
| ReadWriteOnce with block device | one attach | one | databases |
| Observed state | Meaning | Action |
|---|---|---|
| Two pods on different nodes, one PVC | invalid topology | scale to one, or use a per-pod claim |
| Old pod Terminating on the same node | detach not yet complete | wait, then delete the pod |
| Node NotReady and pod gone | stale attachment | delete the VolumeAttachment |
| PVC Pending, pod unscheduled | no PV bound, different problem | check the StorageClass and events |
| Mount fails but attach succeeded | node plugin or filesystem error | check kubelet and CSI logs |
| Strategy | Overlap during rollout | Downtime | Suitable for |
|---|---|---|---|
| Deployment, one replica, Recreate | none | yes, brief | single-writer apps |
| StatefulSet with per-pod PVCs | none | per pod | databases, queues |
| Deployment, two replicas, RWX | none | none | shared read-write storage |
| Deployment, two replicas, RWO | yes, fails | not achievable | never |
Practical Edge Cases, Error Fixes & Deep Dive Code Examples
1. A Deployment with more than one replica and one RWO claim
This is the most common cause in application manifests copied from a single-replica tutorial. Either give each pod its own claim, which only a StatefulSet does properly, or accept a single writer with a rollout that never overlaps:
spec:
replicas: 1
strategy:
type: Recreate # old pod is gone before the new one starts
template:
spec:
containers:
- name: app
volumeMounts: [{ name: data, mountPath: /data }]Recreate trades availability for correctness. A rolling update with two replicas on a shared RWO volume cannot be made correct; the second pod will always be scheduled before the first is gone.
2. StatefulSet rollouts that still collide
StatefulSet creates one PVC per ordinal and rolls pods in reverse order by default, so a collision here usually means the pods are pinned to a node pool with a single node, or a PodDisruptionBudget is keeping the old pod alive while the new one is already scheduled:
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate: { partition: 0 }
podManagementPolicy: OrderedReady # default, safest for single attachCheck whether the old pod is actually terminating: kubectl get pod web-1 -o jsonpath='{.metadata.deletionTimestamp}{"\n"}' shows a value only while it is being deleted. A pod stuck there with a finalizer holds the attachment.
3. Clearing a stale attachment after a node failure
When a node disappears, the controller cannot confirm the detach and the VolumeAttachment stays. Verify that no pod on that node still needs the volume, then remove the attachment object so the controller retries elsewhere:
kubectl get volumeattachment | awk '$5=="false"{print $1}'
kubectl delete volumeattachment csi-3f9a... # for the node that is gone
kubectl describe pvc data | tail -20 # events show the new attach attemptIf the driver is a cloud block store, the provider side can lag: an EBS volume may still report in-use on the old instance. Confirm at the provider before deleting anything further, and detach there if the instance is truly gone.
4. Forcing a stuck pod is not the fix
kubectl delete pod --force --grace-period=0 removes the API object, not the attachment. The next pod on a new node hits the same error, now with no pod to point at, which makes the incident harder to read:
# prefer a normal delete with a bounded wait
kubectl delete pod web-7d9f-abcde --timeout=120s
# only after the node is proven gone
kubectl delete volumeattachment <name>5. Using ReadWriteOncePod for real fencing
ReadWriteOncePod restricts a claim to a single pod cluster-wide, which catches the misconfiguration at scheduling time instead of at attach time and gives a clearer message. It needs a CSI driver that supports single-pod access, and the claim's mode cannot be changed after binding:
kind: PersistentVolumeClaim
spec:
accessModes: ["ReadWriteOncePod"]
resources: { requests: { storage: 20Gi } }Migrate by creating a new claim and copying data, or by recreating the workload with a fresh claim, not by editing the existing one.
6. Reading events instead of guessing
The event stream contains the sequence that explains the failure; the pod's status only shows the last line.
kubectl get events --sort-by=.lastTimestamp | grep -Ei 'attach|mount|volume' | tail -20
kubectl logs -n kube-system ds/ebs-csi-node --tail=50 | grep -i attachFor manifests you are about to change, keep the PVC and the workload in one file and validate it before applying: paste the YAML into YAML Formatter to catch indentation and duplicate keys, which otherwise surface as a different, misleading failure.
Common Compilation/Runtime Mistakes & Version Gotchas
- Scaling a single-writer Deployment to two "for zero-downtime deploys". There is no configuration that makes an RWO block device writable from two nodes; use
Recreateor move the state out. - Assuming ReadWriteOnce means one pod. It means one node; pods on the same node can mount it simultaneously.
- Recreating the PVC to clear the error. The PVC is bound to a PV and the data lives there; deleting it can orphan or destroy the volume.
- Deleting pods before checking attachments. The pod is often not the holder; the
VolumeAttachmentis. - Ignoring zone topology. A volume in one availability zone cannot attach to a node in another; the scheduler must be constrained with
volumeBindingMode: WaitForFirstConsumer. - Allowing
topologySpreadConstraintsto scatter a StatefulSet across zones. Each ordinal then lands in a zone its volume is not in. - Old CSI driver versions. Early EBS CSI releases had detach timeouts that left attachments for minutes; if detach consistently lags, the driver version is worth checking before the manifests.
Primary references: Persistent Volumes documents the access modes and their node-level meaning, and the Volumes page covers the attach and mount lifecycle that the VolumeAttachment objects track.
Official references
primary documentationThe Kubernetes documentation that defines the behaviour this guide explains.
Try matching browser tools
All toolsSmall utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Kubernetes guides
All Kubernetes guidesKubernetes 1.35: kubelet will not start on a cgroup v1 node
A 1.35 kubelet refuses a cgroup v1 host, and the removed pod-infra flag stops custom images. Check the kernel command line before the node pool.
Helm: another operation (install/upgrade/rollback) is in progress
Helm 3 stores the lock inside the release secret. Read the pending status, roll back to the last good revision, or clear the record deliberately.
Helm upgrade failed: field is immutable on Service and Job
Some fields cannot be patched at all. Here is which ones, why --force does not help, and the safe recreate procedure for Services, Jobs and StatefulSets.
Kubernetes: Pod stuck in Terminating (finalizers and grace periods)
A pod disappears only when finalizers clear and containers stop. Read deletionTimestamp, find the blocker, and force-delete only a dead node.
More in Kubernetes
All Kubernetes guidesThe guides filed next to this one in Kubernetes, in library order.