Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Docker build: no space left on device while df shows free space

BuildKit cache lives outside docker system df. Measure it with buildx du, prune to a size budget and reorder the Dockerfile.

~5 min readDocker
#Docker#BuildKit#Disk Space#CI
A build layer stack crushed by a full disk gauge with an amber overflow, headline ENOSPC

Introduction & Quick Summary

Symptom. A build that worked yesterday dies mid-layer:

code
#12 [builder 4/7] RUN npm ci #12 0.842 npm error code ENOSPC #12 3.271 failed to solve: process "/bin/sh -c npm ci" did not complete successfully: exit code: 1 ERROR: failed to solve: ResourceExhausted: no space left on device

Yet df -h / shows tens of gigabytes free, or docker system df claims only a few gigabytes of images. A second symptom: docker build succeeds on a laptop and fails on a CI runner with a 14 GB disk after two or three builds.

Cause. BuildKit keeps its layer cache and build state somewhere that is not the working directory, usually /var/lib/docker, /var/lib/containerd, or a builder container's own volume. docker system df reports images, containers and volumes, and it under-reports builder cache, so the two numbers disagree. --no-cache does not delete the cache; it only refuses to read it, and every build adds more.

Fix. Look at builder usage directly, then prune with a size target instead of --all when you need to keep warm layers.

bash
docker buildx du # per-record usage, biggest first docker system df -v # images, containers, volumes, build cache docker buildx prune --keep-storage 20GB
json
// daemon.json: move the data root off a small root disk { "data-root": "/mnt/docker" }

Core Mechanics, Signatures & Reproducible Setup

Every layer you build is stored as a content-addressed record, and every COPY that changes produces a new one on top of a base image that is tens or hundreds of megabytes. A Node image with npm ci run before COPY of source code caches well; the same image built with COPY . . first stores a new full layer per commit. That is how a repository with 200 MB of source produces 20 GB of cache in a week.

Reproduce the growth to see where it goes:

bash
mkdir -p /tmp/enospc && cd /tmp/enospc printf 'FROM node:26.13.0-bookworm-slim\nWORKDIR /app\nCOPY . .\nRUN npm ci\n' > Dockerfile fallocate -l 400M blob.bin for i in $(seq 1 5); do echo "$i" >> blob.bin; docker build -t enospc:$i . >/dev/null; done docker buildx du | head -5

Each iteration stores a fresh copy of blob.bin, and the cache grows by roughly 400 MB per build. Reordering the Dockerfile so dependency installation precedes the source copy is the structural fix, not pruning:

dockerfile
FROM node:26.13.0-bookworm-slim WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci COPY . . RUN npm run build

Now blob.bin changes and only the last two layers are new.

Key Comparison Table

CommandReportsDoes not report
df -h /var/lib/dockerfilesystem free spacewho consumed it
docker system dfimages, containers, volumesmost builder cache
docker system df -vper-image and per-volume sizeslayer ownership detail
docker buildx dubuilder cache recordsimages pulled but unused
docker buildx prunereclaims builder cachedangling images and volumes
docker image prune -areclaims all unused imagescache, volumes
df -iinode exhaustionbyte usage
Where the space goesTypical sizeReclaim with
Build cache recordsGBs per active projectdocker buildx prune
Dangling image layershundreds of MB eachdocker image prune
Stopped containers and their writable layersvariesdocker container prune
Named volumes from removed containersoften the largestdocker volume prune
/var/lib/docker/overlay2 orphansgrows with interrupted buildsrestart the daemon, then prune

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. The two commands that tell the truth

docker system df is a summary that predates BuildKit; the builder's cache is a separate store. When the two numbers disagree, buildx du is authoritative:

bash
docker buildx ls # which builder is in use docker buildx du --verbose | sort -k1 -hr | head -20 docker buildx prune -f --keep-storage 10GB --filter until=168h

--keep-storage keeps the most recent records up to the size you name, which preserves the warm cache for the project you are actively building while dropping everyone else's. --filter until=168h drops anything untouched for a week.

2. Inodes, not bytes

ENOSPC with free space usually means inodes: millions of small files from node_modules in a hundred layers. Nothing in Docker's own output shows this.

bash
df -i /var/lib/docker # if IUse% is 100, find the directory with the most entries sudo find /var/lib/docker/overlay2 -maxdepth 2 -type d | wc -l

The fix is structural: fewer layers with fewer files, npm ci --omit=dev in the runtime stage, and a multi-stage build that never copies node_modules into the final image.

3. CI runners and the 14 GB disk

Hosted runners give you a fixed disk and no daemon configuration. A reliable pattern is a hard budget per build plus a cleanup step that runs before the build rather than after:

- name: Reclaim disk run: | docker buildx prune -af --keep-storage 0 || true docker image prune -af || true df -h / - name: Build run: docker buildx build --cache-to type=gha,mode=max --cache-from type=gha -t app .

Pushing the cache to a remote backend (type=gha, type=registry or S3) removes the need to keep it on the runner at all, which is the only version of this that survives a busy repository.

4. --no-cache is not a cleanup

docker build --no-cache skips reading the cache and still writes every new layer. Teams that "clean up" by rebuilding with --no-cache accelerate the growth they are trying to stop. Delete instead:

bash
docker buildx prune -af # everything, builder cache only docker system prune -af --volumes # images, containers, networks, volumes

--volumes is the flag that has destroyed real data: it removes named volumes that are not currently attached to a running container, including a development database someone stopped for lunch. Read the list first with docker volume ls -f dangling=true.

5. Moving the data root when the disk is too small

On a VM where / is 20 GB and /mnt is the real disk, the daemon's storage belongs on /mnt. Edit the daemon configuration and restart; every image has to be pulled again afterwards, so schedule it:

bash
sudo systemctl stop docker docker.socket sudo mkdir -p /mnt/docker sudo rsync -aHAX --info=progress2 /var/lib/docker/ /mnt/docker/ sudo tee /etc/docker/daemon.json >/dev/null <<'EOF' { "data-root": "/mnt/docker" } EOF sudo systemctl start docker docker info | grep -i "docker root dir"

Copying rather than starting empty keeps local images; starting empty is faster and safer if you can rebuild everything.

6. Build cache mounts that grow without limit

RUN --mount=type=cache,target=/root/.npm speeds up installs and is easy to leave unbounded. Cache mounts are not part of the image, so they are invisible in docker images and grow until the builder disk fills:

dockerfile
RUN --mount=type=cache,target=/root/.npm,sharing=locked \ npm ci --prefer-offline

Prune them with the builder, and in CI pass --cache-to a remote backend so the mount does not persist across jobs. When a build works locally and fails on the runner with ENOSPC during npm ci, the mount is the first suspect.

Common Compilation/Runtime Mistakes & Version Gotchas

  • Trusting docker system df for builder usage. It does not include most BuildKit cache.
  • Running docker system prune without --volumes and expecting the space back. Volumes are usually the largest single item.
  • Using --no-cache as cleanup. It adds layers instead of removing them.
  • Copying the whole context before installing dependencies. Every source change invalidates the heavy layer.
  • Ignoring .dockerignore. node_modules, .git and build output get sent to the daemon and can land in the image.
  • Pruning a shared builder during a deploy. Another build on the same builder can fail mid-layer; prune when the queue is empty.
  • Assuming a bigger disk fixes it. It postpones it; the Dockerfile is the cause.

Check the response your registry returns when a pull fails for quota reasons with the status code reference, and format a daemon configuration you are editing with the YAML and config validator before restarting the daemon with it.

Reference: Docker build garbage collection and docker buildx prune.

Official references

primary documentation

The Docker documentation that defines the behaviour this guide explains.

Try matching browser tools

All tools

Small utilities that fit Docker work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Docker guides

More in Docker

All Docker guides

The guides filed next to this one in Docker, in library order.