Docker Image Hardening for Production

When we audited our container images last spring, the average production image was 1.2 GB. The Trivy report showed 200+ CVEs across them, mostly inherited from base layers we hadn't picked deliberately. Cold-start times for autoscaled services were noticeably worse than they should have been because pulling a 1.2 GB image to a fresh node took ~12 seconds.

A quarter later we're at an average ~80 MB per image, ~4 CVEs each, and cold-start image-pull time is under 2 seconds. The work was unglamorous. None of it was clever.

What was wrong#

We did the audit in a spreadsheet. For each of our ~22 production images, we recorded: base image, total size, layer count, CVEs by severity, and what the image actually needed to run.

The patterns that fell out:

• 9 images used node:18 (full Debian-based, ~1 GB). They needed Node and curl. That's it.
• 5 images used python:3.11 (similarly full). Same story.
• 4 images used ubuntu:22.04 as a base for what amounted to a single Go binary.
• 2 images had inherited layers from a builder stage that nobody had cleaned up — so they shipped GCC, make, npm, and a node_modules tree alongside the actual app.
• 1 image had two copies of the application baked in, one in /app and one in /srv/app. Nobody knew why.

The opportunity was deleting things, not adding security tools.

The five changes that did most of the work#

1. Switch to distroless or minimal bases#

For the Node services we moved to gcr.io/distroless/nodejs20-debian12. For Go binaries we moved to gcr.io/distroless/static-debian12. For Python we used python:3.11-slim as an intermediate step (distroless Python is harder to use because of native deps); slim was good enough.

# Before
FROM node:18
WORKDIR /app
COPY . .
RUN npm install
CMD ["node", "server.js"]

# After
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
RUN npm run build

FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./
USER nonroot
CMD ["dist/server.js"]

Image size dropped from ~1 GB to ~140 MB on the Node services. CVEs from inherited layers dropped to near zero (distroless ships only what you need to run).

2. Multi-stage builds, ruthlessly#

Most of our images had been single-stage — build tools and runtime tools in the same final image. Splitting them was mechanical:

• Stage 1: FROM <full toolchain image> — install build tools, compile, produce artifacts
• Stage 2: FROM <minimal runtime image> — copy ONLY the artifacts and runtime deps from stage 1

The trap is COPY --from=builder /app /app. That tends to copy more than the artifact. The fix: COPY --from=builder /app/dist /app/dist, or COPY --from=builder /app/myapp /myapp. Be precise.

3. Run as non-root#

Most distroless images have a nonroot user (UID 65532) baked in. We added USER nonroot to every Dockerfile. Combined with runAsNonRoot: true in the Pod's securityContext, anything that tries to escalate fails immediately.

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

Read-only root filesystem caught a few apps that were writing temp files in surprising places — we mounted explicit emptyDir volumes for the directories they actually needed and called the rest a bug.

4. Pin base images by digest, not tag#

FROM node:20-alpine is a moving target. Today's 20-alpine is different from yesterday's. We pin to the digest:

FROM node:20-alpine@sha256:1a7d234c8b... AS builder
FROM gcr.io/distroless/nodejs20-debian12@sha256:9fa2f99... AS runtime

Dependabot bumps these via PR. We see the digest change in the diff. If a base image bumps in a way that breaks our build, we see it explicitly instead of being surprised next Tuesday.

5. Scan in CI, not as a one-off#

We added Trivy to every image build. The CI fails on CRITICAL or HIGH CVEs unless an explicit allowlist entry is set:

- name: Scan image
  uses: aquasecurity/trivy-action@<pinned-sha>
  with:
    image-ref: ${{ steps.build.outputs.tag }}
    severity: CRITICAL,HIGH
    exit-code: 1
    ignore-unfixed: true   # don't fail on CVEs without a patch yet
    trivyignores: .trivyignore

The .trivyignore file lists CVEs we've consciously accepted, with a comment explaining why and an expiration date for review.

This stopped us from shipping new HIGH CVEs. The existing ones got cleaned up by re-baselining as we did the migration to distroless.

What broke during the migration#

Two things, both fixable.

The Python migration to python:3.11-slim broke a service that needed libpq-dev (for psycopg2 to compile). The fix was to install it explicitly in the build stage and use psycopg2-binary instead of psycopg2 so the runtime image didn't need libpq. Took a couple of hours.

The Go service migration to distroless broke a service that was shelling out to curl for a health check. We replaced the curl call with a small Go HTTP request inside the binary. Cleaner anyway. Took an hour.

What we don't do#

• We don't add a shell to distroless images for "debugging." kubectl debug with an ephemeral container is the right tool; baking a shell into production images defeats the point of distroless.
• We don't sign images with cosign yet. It's on the roadmap but we want to land registry-side image scanning + admission control first.
• We don't try to get to zero CVEs. Some HIGH CVEs in OS layers don't have patches yet; we accept and review monthly. The goal is "explicitly known and tracked," not "zero."

Numbers after the migration#

The cold-start improvement was an unexpected bonus. Pulling 80 MB instead of 1.2 GB shaved 10+ seconds off autoscaling response time. For traffic spikes, that's the difference between absorbing the spike and dropping requests.

The 80/20 list for a team starting fresh#

In order of return:

1. Move to distroless (or -slim) base images. This is the single biggest move.
2. Multi-stage builds with precise COPY targets.
3. Run as non-root with read-only root filesystem.
4. Pin base images by digest. Use Dependabot.
5. Scan in CI and fail on HIGH CVEs (with an allowlist for tracked exceptions).

The first two get you 80% of the size reduction. Number three and four are mechanical and prevent regression. Number five enforces the bar.

What I wouldn't bother with on day one: image signing, custom Trivy DB, content trust workflows. They're great once you've done the basics; they don't help if your base image is still ubuntu:latest.