Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Multi-Attach error: the volume is still attached elsewhere

One claim, two nodes. Find the current holder, clear a stale attachment after a node failure, and give the volume a single writer.

~6 min readKubernetes
#Kubernetes#Storage#PVC#Troubleshooting
One storage block reached by two node pads with the second cable blocked by an amber plate

Introduction & Quick Summary

A pod stays in ContainerCreating and kubectl describe pod reports Multi-Attach error for volume "pvc-8f2c..." Volume is already used by pod(s) web-7d9f-abcde on node ip-10-0-3-14, or the more common wrapper Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-2xq7]: timed out waiting for the condition. The application is healthy in every other respect; it simply never starts.

Both messages mean the same thing: the volume's access mode allows one node at a time, and the storage plugin believes another node still has it. The old consumer may be a terminating pod, a pod on a node that died without detaching, a second replica of a Deployment sharing one PVC, or a StatefulSet rollout that briefly runs two pods that both need a ReadWriteOnce claim.

The order that resolves it fastest:

  1. Find the current holder: kubectl get volumeattachment, and kubectl get pod -o wide for the pods named in the error.
  2. If the holder is terminating on a dead node, delete the stuck pod and then the stale VolumeAttachment, so the attach/detach controller can finish.
  3. Fix the shape: one replica per ReadWriteOnce claim, ReadWriteOncePod when the CSI driver supports it, and a rollout strategy that does not overlap pods on different nodes.

Recreating the PVC does not help and can lose data. --force deleting a pod without clearing the attachment usually reproduces the error on the next node.

Core Mechanics, Signatures & Reproducible Setup

Kubernetes splits storage into a control-plane attach step and a node-local mount step. The attach/detach controller, working with the CSI driver, attaches a volume to one node at a time for ReadWriteOnce and ReadWriteOncePod. The kubelet on that node then stages and mounts it. Attachments are recorded as VolumeAttachment objects, and they are only removed when the controller observes that no pod needs the volume any more.

bash
kubectl get pvc data -o jsonpath='{.spec.accessModes}{"\n"}{.status.phase}{"\n"}' kubectl get volumeattachment -o custom-columns=\ NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached kubectl get pods -A -o wide --field-selector spec.nodeName=ip-10-0-3-14 | grep -i web

A minimal reproduction on a multi-node cluster with a CSI driver that enforces single attach:

apiVersion: apps/v1 kind: Deployment metadata: { name: web } spec: replicas: 2 # two pods, one ReadWriteOnce claim selector: { matchLabels: { app: web } } template: metadata: { labels: { app: web } } spec: containers: - name: app image: ghcr.io/acme/web:1.4.0 volumeMounts: [{ name: data, mountPath: /data }] volumes: - name: data persistentVolumeClaim: { claimName: data } # RWO, single node

The second replica lands on another node, the driver refuses the second attach, and both pods report the error until the scheduler gives up on one of them.

Key Comparison Table

Access modeConcurrent podsConcurrent nodesUse for
ReadWriteOncemany on one nodeonea Deployment with one replica
ReadWriteOncePodoneonestateful singletons, strict fencing
ReadOnlyManymanymanyshared assets
ReadWriteManymanymanyNFS, EFS, CephFS only
ReadWriteOnce with block deviceone attachonedatabases
Observed stateMeaningAction
Two pods on different nodes, one PVCinvalid topologyscale to one, or use a per-pod claim
Old pod Terminating on the same nodedetach not yet completewait, then delete the pod
Node NotReady and pod gonestale attachmentdelete the VolumeAttachment
PVC Pending, pod unscheduledno PV bound, different problemcheck the StorageClass and events
Mount fails but attach succeedednode plugin or filesystem errorcheck kubelet and CSI logs
StrategyOverlap during rolloutDowntimeSuitable for
Deployment, one replica, Recreatenoneyes, briefsingle-writer apps
StatefulSet with per-pod PVCsnoneper poddatabases, queues
Deployment, two replicas, RWXnonenoneshared read-write storage
Deployment, two replicas, RWOyes, failsnot achievablenever

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. A Deployment with more than one replica and one RWO claim

This is the most common cause in application manifests copied from a single-replica tutorial. Either give each pod its own claim, which only a StatefulSet does properly, or accept a single writer with a rollout that never overlaps:

spec: replicas: 1 strategy: type: Recreate # old pod is gone before the new one starts template: spec: containers: - name: app volumeMounts: [{ name: data, mountPath: /data }]

Recreate trades availability for correctness. A rolling update with two replicas on a shared RWO volume cannot be made correct; the second pod will always be scheduled before the first is gone.

2. StatefulSet rollouts that still collide

StatefulSet creates one PVC per ordinal and rolls pods in reverse order by default, so a collision here usually means the pods are pinned to a node pool with a single node, or a PodDisruptionBudget is keeping the old pod alive while the new one is already scheduled:

spec: updateStrategy: type: RollingUpdate rollingUpdate: { partition: 0 } podManagementPolicy: OrderedReady # default, safest for single attach

Check whether the old pod is actually terminating: kubectl get pod web-1 -o jsonpath='{.metadata.deletionTimestamp}{"\n"}' shows a value only while it is being deleted. A pod stuck there with a finalizer holds the attachment.

3. Clearing a stale attachment after a node failure

When a node disappears, the controller cannot confirm the detach and the VolumeAttachment stays. Verify that no pod on that node still needs the volume, then remove the attachment object so the controller retries elsewhere:

bash
kubectl get volumeattachment | awk '$5=="false"{print $1}' kubectl delete volumeattachment csi-3f9a... # for the node that is gone kubectl describe pvc data | tail -20 # events show the new attach attempt

If the driver is a cloud block store, the provider side can lag: an EBS volume may still report in-use on the old instance. Confirm at the provider before deleting anything further, and detach there if the instance is truly gone.

4. Forcing a stuck pod is not the fix

kubectl delete pod --force --grace-period=0 removes the API object, not the attachment. The next pod on a new node hits the same error, now with no pod to point at, which makes the incident harder to read:

bash
# prefer a normal delete with a bounded wait kubectl delete pod web-7d9f-abcde --timeout=120s # only after the node is proven gone kubectl delete volumeattachment <name>

5. Using ReadWriteOncePod for real fencing

ReadWriteOncePod restricts a claim to a single pod cluster-wide, which catches the misconfiguration at scheduling time instead of at attach time and gives a clearer message. It needs a CSI driver that supports single-pod access, and the claim's mode cannot be changed after binding:

kind: PersistentVolumeClaim spec: accessModes: ["ReadWriteOncePod"] resources: { requests: { storage: 20Gi } }

Migrate by creating a new claim and copying data, or by recreating the workload with a fresh claim, not by editing the existing one.

6. Reading events instead of guessing

The event stream contains the sequence that explains the failure; the pod's status only shows the last line.

bash
kubectl get events --sort-by=.lastTimestamp | grep -Ei 'attach|mount|volume' | tail -20 kubectl logs -n kube-system ds/ebs-csi-node --tail=50 | grep -i attach

For manifests you are about to change, keep the PVC and the workload in one file and validate it before applying: paste the YAML into YAML Formatter to catch indentation and duplicate keys, which otherwise surface as a different, misleading failure.

Common Compilation/Runtime Mistakes & Version Gotchas

  • Scaling a single-writer Deployment to two "for zero-downtime deploys". There is no configuration that makes an RWO block device writable from two nodes; use Recreate or move the state out.
  • Assuming ReadWriteOnce means one pod. It means one node; pods on the same node can mount it simultaneously.
  • Recreating the PVC to clear the error. The PVC is bound to a PV and the data lives there; deleting it can orphan or destroy the volume.
  • Deleting pods before checking attachments. The pod is often not the holder; the VolumeAttachment is.
  • Ignoring zone topology. A volume in one availability zone cannot attach to a node in another; the scheduler must be constrained with volumeBindingMode: WaitForFirstConsumer.
  • Allowing topologySpreadConstraints to scatter a StatefulSet across zones. Each ordinal then lands in a zone its volume is not in.
  • Old CSI driver versions. Early EBS CSI releases had detach timeouts that left attachments for minutes; if detach consistently lags, the driver version is worth checking before the manifests.

Primary references: Persistent Volumes documents the access modes and their node-level meaning, and the Volumes page covers the attach and mount lifecycle that the VolumeAttachment objects track.

Official references

primary documentation

The Kubernetes documentation that defines the behaviour this guide explains.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.