ImagePullBackOff and ErrImagePull: Reading the Real Pull Error
ImagePullBackOff hides five different failures. The event text names which one: wrong reference, missing credentials, wrong architecture, unreachable registry, or a rate limit.

Introduction & Quick Summary
Symptom. kubectl get pods shows ErrImagePull once and then ImagePullBackOff on every subsequent retry. kubectl describe pod ends with one of: Failed to pull image "app:1.4.2": rpc error: code = NotFound desc = failed to pull and unpack image ...: not found, unauthorized: authentication required, manifest unknown, no matching manifest for linux/arm64/v8 in the manifest list entries, dial tcp: i/o timeout, or toomanyrequests: You have reached your pull rate limit.
Cause. The kubelet on the node cannot obtain the image. Either the reference is wrong (tag, digest, registry host), the registry requires credentials the pod does not carry, the image has no layer for the node's architecture, the node cannot reach or resolve the registry, or the registry is rate-limiting anonymous pulls. ImagePullBackOff is just the backoff state after a failed pull: 10 seconds, doubling to 5 minutes.
Fix. Read the event text, because it names the exact class. NotFound and manifest unknown are reference problems; unauthorized is imagePullSecrets; no matching manifest is a build and platform problem; timeouts and DNS are node networking; toomanyrequests is authentication or a pull-through cache.
Core Mechanics, Signatures & Reproducible Setup
Before any container starts, the node's container runtime resolves the image reference, authenticates to the registry, downloads the manifest for the node's OS and architecture, then fetches and unpacks layers. Failure at any of those steps produces a pod event and the ImagePullBackOff waiting state. imagePullPolicy decides whether this happens at all: IfNotPresent (default for tagged images) skips the pull when a local copy exists, Always (default for latest and for digests without a tag) pulls every start, and Never expects the image on the node.
Reproduce each class in one namespace:
kubectl create ns pull-demo
# 1. wrong tag
kubectl run bad-tag --image=nginx:1.99.0-nope -n pull-demo
# 2. private image without credentials
kubectl run private --image=ghcr.io/example/private-api:1.0.0 -n pull-demo
# 3. valid image, wrong architecture (on an amd64 node)
kubectl run arm --image=--platform-suffix-example -n pull-demo
kubectl get events -n pull-demo --sort-by=.lastTimestampRead the event rather than the pod status; the status only says ImagePullBackOff:
kubectl describe pod bad-tag -n pull-demo | tail -5
# Failed to pull image "nginx:1.99.0-nope": ... not foundKey Comparison Table
| Event text | Class | Fix |
|---|---|---|
not found / manifest unknown | Wrong tag or repository | Correct the reference, verify with docker manifest inspect |
unauthorized: authentication required | No usable credentials | imagePullSecrets or node-level registry config |
no matching manifest for linux/arm64 | Architecture mismatch | Multi-arch build, or nodeSelector for the matching arch |
dial tcp … i/o timeout, no such host | Node cannot reach or resolve the registry | Egress, DNS, proxy, private endpoint |
x509: certificate signed by unknown authority | Private CA not trusted on the node | Install the CA in the node's trust store |
toomanyrequests | Registry rate limit | Authenticate; add a pull-through cache |
ErrImageNeverPull | imagePullPolicy: Never and no local image | Change policy or preload the image |
| Policy | When it pulls | Use for |
|---|---|---|
Always | Every container start | latest, mutable tags in demos |
IfNotPresent | Only when absent locally | Immutable version tags |
Never | Never | Air-gapped nodes with preloaded images |
Practical Edge Cases, Error Fixes & Deep Dive Code Examples
1. Proving the reference is correct
# Does the tag exist, and for which platforms?
docker manifest inspect nginx:1.27-alpine | head -20
# Or without Docker:
curl -s "https://registry-1.docker.io/v2/library/nginx/manifests/1.27-alpine" \
-H "Accept: application/vnd.oci.image.index.v1+json" | head -30A private registry returns 401 from that curl before it tells you whether the tag exists, so an unauthorized response to a probe says nothing about existence. Use the registry's UI or crane digest repo/image:tag with credentials.
2. imagePullSecrets that the pod actually uses
kubectl create secret docker-registry regcred \
--docker-server=ghcr.io \
--docker-username=<user> \
--docker-password=<token> \
-n pull-demospec:
imagePullSecrets:
- name: regcred
containers:
- name: api
image: ghcr.io/example/private-api:1.0.0The secret must be a kubernetes.io/dockerconfigjson type and live in the pod's namespace. Attaching it to the namespace's default ServiceAccount removes the per-pod field:
kubectl patch serviceaccount default -n pull-demo \
-p '{"imagePullSecrets":[{"name":"regcred"}]}'A common failure is a secret created for https://ghcr.io instead of ghcr.io; the auth key in .dockerconfigjson must match the registry host exactly. Inspect it:
kubectl get secret regcred -n pull-demo -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | head -c 3003. Architecture mismatch after moving to ARM nodes
# Build for both architectures from an amd64 host
docker buildx create --use --name multi
docker buildx build --platform linux/amd64,linux/arm64 \
-t ghcr.io/example/private-api:1.0.0 --push .Then confirm the index lists both:
docker buildx imagetools inspect ghcr.io/example/private-api:1.0.0If a base image itself has no ARM64 variant, the build fails earlier with no match for platform in manifest. Pinning workloads with kubernetes.io/arch node selectors is a workaround, not a fix; it creates a cluster that only runs on one architecture.
4. Node-level registry configuration
Cluster-wide credentials belong in the container runtime config, not in every manifest. For containerd, the node's /etc/containerd/config.toml gets a registry host block with an auth path, and kubelet needs a restart:
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"# /etc/containerd/certs.d/registry.internal/hosts.toml
server = "https://registry.internal"
[host."https://registry.internal"]
capabilities = ["pull", "resolve"]
ca = "/etc/containerd/certs.d/registry.internal/ca.crt"On managed clusters this is a node pool setting, not a file you edit; check the provider's documented mechanism before debugging a manifest that cannot fix it.
5. Rate limits and pull-through caches
Docker Hub limits anonymous pulls per source IP, and a cluster behind one NAT gateway reaches that limit quickly during a rolling restart. Two fixes: authenticate (a free account raises the limit substantially), or run a pull-through cache so the cluster pulls from a local registry mirror.
# kubelet flag equivalent on self-managed nodes
# --registry-mirror=https://mirror.internalDetection is easy: the event says toomanyrequests, and crictl pull nginx:1.27 on the node fails identically while a different image pulls fine.
6. ErrImageNeverPull and preloaded images
imagePullPolicy: Never produces no pull at all, so the error is Container image "app:1.4.2" is not present with pull policy of Never. This is the correct policy for air-gapped clusters; the workflow is to preload the image on every node (ctr -n k8s.io images import app.tar or a DaemonSet that pulls from an internal mirror) and verify with crictl images. If only some pods fail, compare nodeName across the failing pods — the image is missing on specific nodes, which usually means a DaemonSet preloader did not finish on them.
Common Compilation/Runtime Mistakes & Version Gotchas
| Mistake | Symptom | Correct approach |
|---|---|---|
Secret created for https://host instead of host | unauthorized despite a valid secret | Match the host key exactly |
| Secret in the wrong namespace | unauthorized | Create it in the pod's namespace |
| Tag rebuilt and pushed, but pods use a local copy | Old code runs | Immutable tags, imagePullPolicy: Always |
Digest reference with Always | Unnecessary pulls and rate limits | IfNotPresent on a digest |
Assuming latest is current | Stale image on nodes | Never deploy latest |
| Amd64 image on an ARM node | no matching manifest | Multi-arch build |
| Registry behind a private CA | x509: certificate signed by unknown authority | Trust the CA on the node |
Version notes: since Kubernetes 1.24 Docker Engine is no longer the default runtime, so daemon-level credential helpers (~/.docker/config.json) on a node have no effect — the node's containerd config is what matters. imagePullPolicy defaults differ by reference form: a bare image: nginx resolves to latest and Always, while nginx:1.27 defaults to IfNotPresent, so an identical manifest can behave differently after a tag is added. imagePullPolicy: Always on a digest reference still contacts the registry on every container start, which is a common source of toomanyrequests in clusters that pin by digest and restart often.
For the mechanical checks, the base64 tool decodes a .dockerconfigjson to confirm the registry host and username, the JSON formatter turns a docker manifest inspect dump or a pod -o json into something readable, and the YAML validator catches a spec.imagePullSecrets indented under the wrong key — a typo that produces unauthorized with a secret that is perfectly valid.
Try matching browser tools
All toolsSmall utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Kubernetes guides
All Kubernetes guidesCrashLoopBackOff: Reading Exit Codes and Debugging Restart Loops
The backoff is only the waiting state. The evidence is in Last State, the exit code, and the previous container's last log line, and each points to a different fix.
kubectl Output: jsonpath, custom-columns, and the Debugging Commands Worth Knowing
jsonpath for one value in a script, custom-columns for tables people read, and kubectl debug when the container has no shell. Includes the three quoting layers that break jsonpath.
Kubernetes Namespaces, ResourceQuotas, and LimitRanges
A namespace is only a boundary when quota and limit range make it one. Aggregate caps, per-pod defaults, the admission errors they produce, and namespaces stuck Terminating.
Kubernetes StatefulSets: Ordinal Identities, Headless Services, and PVC Lifecycle
Why each StatefulSet pod keeps a name, a DNS record, and its own PVC; how OrderedReady gates startup on readiness; and how partition-based updates canary a single quorum member.
More in Kubernetes
All Kubernetes guidesThe guides filed next to this one in Kubernetes, in library order.
- Horizontal Pod Autoscaler (HPA): Scaling Metrics and Stabilization Windows
- Kubernetes Deployments: Rolling Updates, Rollbacks, and maxUnavailable
- Kubernetes Ingress: Path Types, Controller Annotations, and TLS Termination
- Kubernetes Jobs and CronJobs: Completions, Parallelism, Backoff, and Missed Schedules