Two failures wear the same face: builds that rebuild everything every time, and builds that hand you a stale image. Both trace back to how layer caching keys work.
Fixing Docker Build Cache — Slow Builds and Stale Layers
Our CI image build crept from ninety seconds to seven minutes over about a year, and nobody noticed the day it happened because it never happened on one day. A dependency here, a reordered COPY there. Then a separate team burned an afternoon chasing a bug that turned out to be a stale layer serving an old config file that had been "fixed" three commits ago. Same subsystem, two opposite symptoms. Once you see how the cache actually keys layers, both stop being mysteries.
Every instruction in your Dockerfile produces a layer, and Docker computes a cache key for each one. For most instructions the key is the instruction text itself plus the state of the layers before it. For COPY and ADD it also includes a checksum of the files being copied. If the key matches a layer Docker already has, it reuses it and moves on. The moment one key misses, every instruction after it is invalidated too, because each layer's key depends on the one above it.
That cascade is the whole game. A cache miss on line 4 throws away the work on lines 5 through 20 even if those instructions never changed. So the question is never "is my cache working," it's "which line is busting it, and did I put that line in the worst possible spot."
The classic offender is copying your entire source tree before installing dependencies:
FROM node:20
WORKDIR /app
COPY . .
RUN npm install
RUN npm run build
CMD ["node", "dist/server.js"]
COPY . . checksums your whole working tree. Change one line in any source file and the key for that layer misses, which invalidates npm install and everything below. You reinstall every dependency on every commit, even though your package.json hasn't moved in weeks.
The fix is to order the Dockerfile so rarely-changing layers come first. Copy the lockfile and install before you copy source:
FROM node:20
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
CMD ["node", "dist/server.js"]
Now npm ci only reruns when package.json or package-lock.json changes. Edit a route handler and the install layer stays cached; Docker jumps straight to COPY . .. This one reordering is the highest-leverage change most Dockerfiles need, and it costs nothing.
The same logic applies per language. Python: copy requirements.txt and pip install before source. Go: copy go.mod and go.sum and run go mod download first. Rust with Cargo, Ruby with Bundler, same shape. Dependencies change rarely, source changes constantly, so dependencies go up top.
Even with good ordering, COPY . . will checksum everything in the build context. If your context includes node_modules, .git, build output, and log files, two things go wrong: the context upload is huge, and unrelated file changes bust the copy layer. A local node_modules timestamp shift can invalidate the copy even when no source changed.
A .dockerignore fixes both:
node_modules
.git
dist
*.log
.env
coverage
This trims what gets sent to the daemon and keeps churny local files out of the cache key. If you want the full rundown on what belongs in there, we wrote up .dockerignore separately. Treat it as load-bearing, not housekeeping.
Ordering keeps the install layer cached across commits, but the first build after a lockfile change still downloads every package from scratch. BuildKit cache mounts fix that by persisting the package manager's own cache directory across builds, outside the image layers:
# syntax=docker/dockerfile:1
FROM node:20
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
npm ci
COPY . .
RUN npm run build
The --mount=type=cache directory survives even when the layer itself is rebuilt. Add one dependency and npm ci reruns, but it pulls the unchanged packages from the warm cache instead of the network. For pip use target=/root/.cache/pip, for Go target=/root/.cache/go-build and /go/pkg/mod, for apt point at /var/cache/apt. This is the single biggest win on a cold-ish build, and it needs BuildKit, which is the default in current Docker.
CI runners start clean, so none of the above helps unless the cache lives somewhere the runner can reach. That's what registry cache is for. You export the build cache to a registry and import it on the next run:
docker buildx build \
--cache-from type=registry,ref=myrepo/app:buildcache \
--cache-to type=registry,ref=myrepo/app:buildcache,mode=max \
-t myrepo/app:latest --push .
mode=max exports cache for every stage, not just the final image, which matters for multi-stage builds. The runner pulls the cache metadata, matches layer keys, and skips work it did last time. Set this up once and your CI build behaves like your warm laptop.
The opposite failure. Docker happily reuses a layer whose key still matches but whose real-world dependency changed underneath it. The usual trap is RUN apt-get update on its own line: the key never changes, so Docker serves a months-old package index, then apt-get install pulls stale or missing versions. Always chain them in one instruction so they cache and bust together:
RUN apt-get update && apt-get install -y --no-install-recommends curl
The other stale trap is remote content Docker can't see, a git clone, a curl of a "latest" tarball. The instruction text is identical run to run, so the key matches and you get last week's download. When you genuinely need a fresh pull, reach for --no-cache on that build, or a cache-busting build arg for the one stage that needs it. Reserve --no-cache for debugging and for the rare content Docker can't checksum. Using it as your default just means you've given up on the cache entirely.
Multi-stage builds cache per stage, so a heavy builder stage stays warm while your slim runtime stage rebuilds cheaply:
FROM node:20 AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY . .
RUN npm run build
FROM node:20-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["node", "dist/server.js"]
Order the COPY lines in the final stage the same way, most stable first. And if you hit a build error that looks like a cache problem but isn't, our Docker troubleshooting guide covers the ones that masquerade as caching bugs.
Fix ordering first, because it's free and it fixes the most common case. Add a real .dockerignore the same afternoon. Then wire BuildKit cache mounts for your package manager and registry cache in CI, in that order, since the mounts help everywhere and the registry cache only helps runners. Leave --no-cache in the toolbox for the two or three things Docker genuinely can't see, and chain your apt-get lines so the stale case never bites. Do that and a seven-minute build goes back to ninety seconds without anyone noticing the day it happened.
