Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

ImagePullBackOff and ErrImagePull: Reading the Real Pull Error

ImagePullBackOff hides five different failures. The event text names which one: wrong reference, missing credentials, wrong architecture, unreachable registry, or a rate limit.

~5 min readKubernetes
#Kubernetes#Containers#Registry#Troubleshooting
3D illustration of a node cube pulling an image layer stack through a locked registry gate

Introduction & Quick Summary

Symptom. kubectl get pods shows ErrImagePull once and then ImagePullBackOff on every subsequent retry. kubectl describe pod ends with one of: Failed to pull image "app:1.4.2": rpc error: code = NotFound desc = failed to pull and unpack image ...: not found, unauthorized: authentication required, manifest unknown, no matching manifest for linux/arm64/v8 in the manifest list entries, dial tcp: i/o timeout, or toomanyrequests: You have reached your pull rate limit.

Cause. The kubelet on the node cannot obtain the image. Either the reference is wrong (tag, digest, registry host), the registry requires credentials the pod does not carry, the image has no layer for the node's architecture, the node cannot reach or resolve the registry, or the registry is rate-limiting anonymous pulls. ImagePullBackOff is just the backoff state after a failed pull: 10 seconds, doubling to 5 minutes.

Fix. Read the event text, because it names the exact class. NotFound and manifest unknown are reference problems; unauthorized is imagePullSecrets; no matching manifest is a build and platform problem; timeouts and DNS are node networking; toomanyrequests is authentication or a pull-through cache.

Core Mechanics, Signatures & Reproducible Setup

Before any container starts, the node's container runtime resolves the image reference, authenticates to the registry, downloads the manifest for the node's OS and architecture, then fetches and unpacks layers. Failure at any of those steps produces a pod event and the ImagePullBackOff waiting state. imagePullPolicy decides whether this happens at all: IfNotPresent (default for tagged images) skips the pull when a local copy exists, Always (default for latest and for digests without a tag) pulls every start, and Never expects the image on the node.

Reproduce each class in one namespace:

bash
kubectl create ns pull-demo # 1. wrong tag kubectl run bad-tag --image=nginx:1.99.0-nope -n pull-demo # 2. private image without credentials kubectl run private --image=ghcr.io/example/private-api:1.0.0 -n pull-demo # 3. valid image, wrong architecture (on an amd64 node) kubectl run arm --image=--platform-suffix-example -n pull-demo kubectl get events -n pull-demo --sort-by=.lastTimestamp

Read the event rather than the pod status; the status only says ImagePullBackOff:

bash
kubectl describe pod bad-tag -n pull-demo | tail -5 # Failed to pull image "nginx:1.99.0-nope": ... not found

Key Comparison Table

Event textClassFix
not found / manifest unknownWrong tag or repositoryCorrect the reference, verify with docker manifest inspect
unauthorized: authentication requiredNo usable credentialsimagePullSecrets or node-level registry config
no matching manifest for linux/arm64Architecture mismatchMulti-arch build, or nodeSelector for the matching arch
dial tcp … i/o timeout, no such hostNode cannot reach or resolve the registryEgress, DNS, proxy, private endpoint
x509: certificate signed by unknown authorityPrivate CA not trusted on the nodeInstall the CA in the node's trust store
toomanyrequestsRegistry rate limitAuthenticate; add a pull-through cache
ErrImageNeverPullimagePullPolicy: Never and no local imageChange policy or preload the image
PolicyWhen it pullsUse for
AlwaysEvery container startlatest, mutable tags in demos
IfNotPresentOnly when absent locallyImmutable version tags
NeverNeverAir-gapped nodes with preloaded images

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. Proving the reference is correct

bash
# Does the tag exist, and for which platforms? docker manifest inspect nginx:1.27-alpine | head -20 # Or without Docker: curl -s "https://registry-1.docker.io/v2/library/nginx/manifests/1.27-alpine" \ -H "Accept: application/vnd.oci.image.index.v1+json" | head -30

A private registry returns 401 from that curl before it tells you whether the tag exists, so an unauthorized response to a probe says nothing about existence. Use the registry's UI or crane digest repo/image:tag with credentials.

2. imagePullSecrets that the pod actually uses

bash
kubectl create secret docker-registry regcred \ --docker-server=ghcr.io \ --docker-username=<user> \ --docker-password=<token> \ -n pull-demo
yaml
spec: imagePullSecrets: - name: regcred containers: - name: api image: ghcr.io/example/private-api:1.0.0

The secret must be a kubernetes.io/dockerconfigjson type and live in the pod's namespace. Attaching it to the namespace's default ServiceAccount removes the per-pod field:

bash
kubectl patch serviceaccount default -n pull-demo \ -p '{"imagePullSecrets":[{"name":"regcred"}]}'

A common failure is a secret created for https://ghcr.io instead of ghcr.io; the auth key in .dockerconfigjson must match the registry host exactly. Inspect it:

bash
kubectl get secret regcred -n pull-demo -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | head -c 300

3. Architecture mismatch after moving to ARM nodes

bash
# Build for both architectures from an amd64 host docker buildx create --use --name multi docker buildx build --platform linux/amd64,linux/arm64 \ -t ghcr.io/example/private-api:1.0.0 --push .

Then confirm the index lists both:

bash
docker buildx imagetools inspect ghcr.io/example/private-api:1.0.0

If a base image itself has no ARM64 variant, the build fails earlier with no match for platform in manifest. Pinning workloads with kubernetes.io/arch node selectors is a workaround, not a fix; it creates a cluster that only runs on one architecture.

4. Node-level registry configuration

Cluster-wide credentials belong in the container runtime config, not in every manifest. For containerd, the node's /etc/containerd/config.toml gets a registry host block with an auth path, and kubelet needs a restart:

toml
[plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d"
toml
# /etc/containerd/certs.d/registry.internal/hosts.toml server = "https://registry.internal" [host."https://registry.internal"] capabilities = ["pull", "resolve"] ca = "/etc/containerd/certs.d/registry.internal/ca.crt"

On managed clusters this is a node pool setting, not a file you edit; check the provider's documented mechanism before debugging a manifest that cannot fix it.

5. Rate limits and pull-through caches

Docker Hub limits anonymous pulls per source IP, and a cluster behind one NAT gateway reaches that limit quickly during a rolling restart. Two fixes: authenticate (a free account raises the limit substantially), or run a pull-through cache so the cluster pulls from a local registry mirror.

yaml
# kubelet flag equivalent on self-managed nodes # --registry-mirror=https://mirror.internal

Detection is easy: the event says toomanyrequests, and crictl pull nginx:1.27 on the node fails identically while a different image pulls fine.

6. ErrImageNeverPull and preloaded images

imagePullPolicy: Never produces no pull at all, so the error is Container image "app:1.4.2" is not present with pull policy of Never. This is the correct policy for air-gapped clusters; the workflow is to preload the image on every node (ctr -n k8s.io images import app.tar or a DaemonSet that pulls from an internal mirror) and verify with crictl images. If only some pods fail, compare nodeName across the failing pods — the image is missing on specific nodes, which usually means a DaemonSet preloader did not finish on them.

Common Compilation/Runtime Mistakes & Version Gotchas

MistakeSymptomCorrect approach
Secret created for https://host instead of hostunauthorized despite a valid secretMatch the host key exactly
Secret in the wrong namespaceunauthorizedCreate it in the pod's namespace
Tag rebuilt and pushed, but pods use a local copyOld code runsImmutable tags, imagePullPolicy: Always
Digest reference with AlwaysUnnecessary pulls and rate limitsIfNotPresent on a digest
Assuming latest is currentStale image on nodesNever deploy latest
Amd64 image on an ARM nodeno matching manifestMulti-arch build
Registry behind a private CAx509: certificate signed by unknown authorityTrust the CA on the node

Version notes: since Kubernetes 1.24 Docker Engine is no longer the default runtime, so daemon-level credential helpers (~/.docker/config.json) on a node have no effect — the node's containerd config is what matters. imagePullPolicy defaults differ by reference form: a bare image: nginx resolves to latest and Always, while nginx:1.27 defaults to IfNotPresent, so an identical manifest can behave differently after a tag is added. imagePullPolicy: Always on a digest reference still contacts the registry on every container start, which is a common source of toomanyrequests in clusters that pin by digest and restart often.

For the mechanical checks, the base64 tool decodes a .dockerconfigjson to confirm the registry host and username, the JSON formatter turns a docker manifest inspect dump or a pod -o json into something readable, and the YAML validator catches a spec.imagePullSecrets indented under the wrong key — a typo that produces unauthorized with a secret that is perfectly valid.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.