Skip to main content
Container Security Scanning: Protecting Your Docker Images

Container Security Scanning: Protecting Your Docker Images

KU
· Founder & Chief Editor
• 8 min read•Updated •

We scan every container image in CI and at runtime. Trivy + Cosign + admission controllers. The setup that earns its place and what we wish we'd known.

Container Security Scanning: A Working Setup

Every container image we build goes through security scanning. The scanners flag CVEs in base images and dependencies; the policies determine what blocks deploys; the runtime checks verify what's actually running matches what was scanned. After tuning the system over a few years, this is what we run, with the production reasons each piece earns its place.

What we want from container scanning

The goals:

  1. Block deploys of known-vulnerable images. If a critical CVE is in our base image, we want to know before it hits prod, not after.
  2. Detect drift. What's running in prod should match what was scanned. If it doesn't, something is off.
  3. Surface vulnerabilities to teams so they can prioritize fixes.
  4. Verify image provenance. The image running was actually built by our CI from our code.

Each of these maps to specific tooling.

The pipeline

A typical image's security journey:

code
Build → Scan (CI) → Sign → Push to registry → Verify on deploy → Run

In detail:

  1. CI builds the image
  2. Trivy scans it for vulnerabilities; CI fails if scan exceeds thresholds
  3. Cosign signs the image
  4. Image pushed to ECR with the signature
  5. Argo CD pulls the image to deploy
  6. Admission controller verifies the signature before allowing the pod
  7. Falco watches runtime behavior

If anything fails along the way, the deploy is blocked. The unhappy path doesn't give you "a slightly less secure container" — it gives you no deploy.

Trivy: scanning at build time

Trivy is the open-source scanner we use. It checks:

  • OS packages (apt/apk databases, etc.) against vulnerability databases
  • Language-specific packages (npm, pip, Go modules) against their respective databases
  • Configuration files (Dockerfile, K8s YAML) for misconfigurations
  • Secrets (accidentally committed credentials)

A typical Trivy run on a container takes 10-30 seconds. Fast enough to fit in CI.

The scan in CI:

Bash
trivy image \
  --severity HIGH,CRITICAL \
  --exit-code 1 \
  --ignore-unfixed \
  --format table \
  $IMAGE_TAG

What this means:

  • Only HIGH and CRITICAL severities (we triage MEDIUM separately)
  • Exit code 1 → CI fails on findings
  • --ignore-unfixed → only fail on CVEs that have a fix available (don't block on unfixable issues, which are usually old CVEs in stable distros)

Threshold tuning

The naive setup is "any HIGH/CRITICAL CVE blocks the deploy." This is too strict; you'll be unable to ship anything because some library will always have a recent CVE.

Our thresholds:

  • CRITICAL CVE with fix: blocks deploy.
  • HIGH CVE with fix: blocks deploy unless explicit allowlist entry exists.
  • MEDIUM CVE: surfaces in dashboard; doesn't block.
  • CVEs without fix: surfaced but not blocking. We track upstream; rebuild when a fix becomes available.
  • HIGH CVE in dev/test dependencies: doesn't block (only flagged for runtime/prod path).

The allowlist is non-trivial. Each entry has an owner, a reason, and an expiry date. An allowlist entry that's been there for 6 months gets re-reviewed.

Image base choice matters

The base image determines most of your CVE surface. Switching from ubuntu:22.04 to gcr.io/distroless/static cut our average HIGH/CRITICAL CVE count from ~25 to ~2. Because distroless contains far fewer packages.

Our base image rules:

  • Default: distroless or alpine (alpine for shells we sometimes need, distroless when we don't)
  • If a full distro is needed: use the slim variant (debian:slim, not debian)
  • Pin to specific tags (debian:12.4-slim, not debian:slim)
  • Rebuild nightly to pick up upstream patches

The "rebuild nightly" point is important. Even if your code doesn't change, the upstream base might have new CVE fixes. Our nightly rebuild catches these without manual intervention.

Cosign: signing and verifying

Trivy tells us what's IN the image. Cosign tells us if it's the image we built.

In CI, after building and scanning, we sign:

Bash
cosign sign --key cosign.key $IMAGE_TAG

The signature is stored in the registry alongside the image (Cosign uses a sidecar tag).

At deploy time, an admission controller (Kyverno or Gatekeeper) verifies the signature:

YAML
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  rules:
    - name: verify-cosign
      match:
        resources:
          kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "our-ecr-repo/*"
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      ...
                      -----END PUBLIC KEY-----

A pod whose image isn't signed by our key gets rejected at admission. This is the runtime defense against "someone pushes an unauthorized image to ECR."

We started with this in audit-only mode for a few weeks (just logging violations) before enforcing. Found a few internal images that weren't going through our signing pipeline; fixed those before turning enforcement on.

SBOM generation

Software Bill of Materials: a structured list of all components in an image. We generate one per image with Syft:

Bash
syft $IMAGE_TAG -o cyclonedx-json > sbom.json

The SBOM is stored alongside the image (also as a Cosign attestation).

Use cases:

  • Compliance audits ("show us what's in this image")
  • Vulnerability response ("find all images that contain log4j 2.14")
  • Supply-chain analysis

We don't use SBOMs much day-to-day, but having them means we're prepared for compliance asks and incidents.

Runtime detection: Falco

Build-time scanning catches what's IN the image. Runtime detection catches what the image DOES.

Falco is the open-source runtime security tool we use. It watches kernel events for suspicious behavior:

  • Container running as root (when it shouldn't)
  • Sensitive file reads (/etc/shadow, ~/.aws/credentials)
  • Suspicious process exec (e.g., nc -e /bin/sh — reverse shell)
  • Network connections to suspicious destinations
  • Container privilege escalation

Falco rules are extensive out-of-the-box; we tune to reduce noise.

When a Falco rule fires:

  • Critical severity → page the on-call
  • High severity → ticket for security team
  • Medium → log for review

Expect most high-severity Falco hits to be false positives (false positives — usually a CI runner doing something unusual). The visibility is still worth the effort.

What's hard about scanning

Things that bite teams:

False positives. Some CVEs are flagged but unexploitable in your context (you don't use the vulnerable function). Without an allowlist mechanism, you fail builds for issues that don't matter. Allowlist with reason + expiry helps.

Vuln database coverage. Trivy's database is good for major distros and ecosystems. For obscure dependencies, coverage is patchier. We use multiple scanners (Trivy + Snyk for languages it covers better) on critical images.

Build-time vs runtime mismatch. Image is scanned at build; some packages are added at runtime (shouldn't be, but sometimes are). Runtime detection (Falco) helps catch these.

Speed. Scanning a large image (~1GB) can take 30+ seconds. For small frequent CI runs this is significant. Caching layers helps; switching to smaller base images helps more.

Noise after a major CVE announcement. When log4shell hit, every Java image flagged. All hands on deck for a couple of days. The pipeline survived but barely; we built tooling to do bulk allowlisting / bulk re-scanning afterward.

Specific incidents

Times scanning saved us:

A base image upgrade introduced a new HIGH CVE. We were updating from node:18-alpine to node:20-alpine. The newer alpine had a temporarily-vulnerable musl. CI blocked the deploy; we waited a week for the upstream fix; rebuilt; deploy went through. Without scanning, the vulnerability would have been in production.

A typosquatted npm package made it into a dev dependency. A package similar to a real one, with malicious code. Trivy flagged unusual behavior in the npm scan. We removed the dependency; reported it upstream.

An old image was still being deployed in prod. A service hadn't been redeployed in 8 months; its image had accumulated several CVE patches in upstream. Our nightly rebuild + scan flagged this; we kicked off a deploy of the fresh image.

Cost

Self-hosted setup:

  • Trivy: free (CLI tool, runs in CI)
  • Cosign: free (CLI tool, runs in CI)
  • Syft: free
  • Falco: free open source, ~$200/month for the supporting infrastructure
  • Admission controllers: free
  • Engineer time: ~3 hours/week ongoing

Total marginal cost: ~$200/month. The engineer time is real but small.

Compared to commercial alternatives (Snyk, Aqua, Sysdig): the commercial tools have better UX and more features but cost $5-20k+/month for our scale. The open-source stack is good enough for our needs.

What I'd tell a team starting

Scan in CI; block on critical findings. The simplest setup catches a lot.

Use distroless or slim base images. Most of your CVE surface comes from the base; smaller bases = fewer CVEs.

Sign images and verify at admission. Defense against unauthorized images. Cosign + Kyverno is the standard.

Allowlist with expiry, not "ignore forever." Each exception is reviewed periodically.

Rebuild nightly. Catches upstream patches without manual intervention.

Add runtime detection eventually. Falco or similar for visibility into what's running.

Don't try to fix everything immediately. Triage by severity; address the actionable items; track the rest.

Container security isn't one big thing. It's a pipeline with multiple checkpoints, each catching a different class of issue. The pieces are well-known; the discipline is in keeping the thresholds reasonable, the allowlist clean, and the rebuilds happening. Once the system is running, it does its work quietly. The win is the bad images that never made it to production — which you don't see, because they didn't.

React
Share this post
Cite this article
Plain text

Kiril Urbonas (2025). Container Security Scanning: Protecting Your Docker Images. DevOpsNess. Updated 2026-11-20. https://www.devopsness.com/blog/container-security-scanning-protecting-docker-images

Markdown

[Container Security Scanning: Protecting Your Docker Images](https://www.devopsness.com/blog/container-security-scanning-protecting-docker-images) (Kiril Urbonas, DevOpsNess, updated 2026-11-20)

KU

Kiril Urbonas

Founder & Chief Editor

608 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.