# Handling Vulnerabilities in Production: What We Actually Do

URL: https://www.devopsness.com/blog/handling-vulnerabilities-in-production
Author: Kiril Urbonas
Published: 2026-05-16
Updated: 2026-10-31
Categories: DevOps
Tags: Security, Vulnerabilities, CVE, Patching, DevOps

Summary: You always have known vulnerabilities. The question is how you triage, patch, and respond. A KEV-and-EPSS triage discipline that holds up through incidents and routine work.

---
# Handling Vulnerabilities in Production: What We Actually Do

Triage every CVE on three things: is it exploited in the wild (CISA's KEV catalog, FIRST's EPSS score), is the vulnerable code reachable, and what sits behind it. Known-exploited plus internet-facing means fix within 24 hours. Everything else follows a severity-based patching SLO, and every exception gets a named owner and an expiry date.

> [!VERIFIED]
> Trivy v0.76.1, CVSS v4.0, CISA KEV catalog version 2026.10.30 (1,766 entries), against first.org, cisa.gov and the trivy.dev filtering docs. Checked October 2026.

If you run non-trivial software, you have known vulnerabilities right now: open CVEs in OS packages, an unpatched advisory in some npm or Python dependency, a bundled library upstream abandoned. That is normal. The question is whether you have a system for handling them. This is the one we run after a few real incidents and a lot of routine maintenance.

## Scanner output is a baseline, not a crisis

Scanners will report hundreds of findings. Teams that treat every red row as urgent burn out, stop reading the scanner and miss the one that mattered. Teams that cope have a triage step: most findings go to "patch on schedule", a few to "patch now", and energy goes where it counts.

## Triage on exploitation, reachability and blast radius

CVSS (now at version 4.0) measures how bad a flaw could be, not how likely it is to hurt you. A critical 9.8 in a function you never call matters less than a medium 6.4 in your auth path. We add three signals:

**Exploitation evidence.** CISA's Known Exploited Vulnerabilities (KEV) catalog lists CVEs confirmed exploited in the wild; it held 1,766 entries as of October 30, 2026. FIRST's EPSS publishes, every day, a 0 to 1 probability that a CVE will be exploited in the next 30 days. Both are free and scriptable.

**Reachability.** Is the vulnerable code actually called? Reachability analysis in modern SCA tools separates the transitive dependency that is never imported from the function your request path hits. More on that in our [dependency scanning (SCA) guide](/blog/dependency-scanning-sca-guide).

**Blast radius.** A public payments API outranks an internal nightly cron job with the same CVE.

This check takes a list of CVE IDs from a scan and flags the ones that need a human today:

```
$ curl -s https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json \
    | jq -r '.vulnerabilities[].cveID' > kev.txt
$ grep -Fxf kev.txt scan-cves.txt          # any output here is a fix-now candidate
CVE-2021-44228
$ curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2021-44228' | jq -r '.data[] | "\(.cve) epss=\(.epss) pct=\(.percentile)"'
CVE-2021-44228 epss=0.999990000 pct=1.000000000
```

Log4Shell still scores 0.99999 on EPSS as of October 30, 2026, five years after disclosure. Old does not mean safe.

## The four buckets

**Fix now (within 24 hours).** In KEV or with a public exploit, reachable, internet-exposed. Pager-worthy. Recent examples: Log4Shell, the polyfill.io supply-chain attack, and in October 2026 [PHP2Shell, the unauthenticated RCE in PHP core](/blog/php2shell-cve-2026-109618-109619), [GitLab's CVSS 10 regex RCE](/blog/gitlab-cve-2026-110264-regex-rce-patch-now) and [the exploited Linux TCP bug on Kubernetes nodes](/blog/linux-kernel-tcp-cve-2026-98432-mptcp-kubernetes-nodes).

**Fix soon (within 30 days).** High severity, reachable, no exploitation evidence yet. The bulk of patch work, scheduled into the next sprint with a hard date.

**Fix later (within 90 days).** Medium severity, or high but unreachable, or in a non-public component. Rides the routine patch cycle.

**Accept.** Low severity, no reachable path, or no upstream fix with a mitigation in place. Documented with an owner and an expiry. "Accepted" must never become "forgotten."

## Patching SLOs

The buckets become deadlines:

| Severity | Internet-exposed | Internal |
|---|---|---|
| Critical + known exploited | 24h | 72h |
| Critical | 7 days | 14 days |
| High | 30 days | 60 days |
| Medium | 90 days | 180 days |
| Low | best effort | best effort |

The clock starts at CVE publication or at first evidence of exploitation, whichever comes first. The owner is the team that owns the component; at-risk SLOs escalate.

## The routine that catches most of it

Most vulnerability work is not emergency work. It is a rhythm:

- **Automated dependency PRs on every repo.** We auto-merge patch versions for trusted packages and review minor and major bumps.
- **Scheduled base image rebuilds,** so new OS-level fixes land without anyone doing anything.
- **A scan gate in CI** that fails on fixable high and critical findings.
- **A monthly OS patch window** rolled out dev, then staging, then prod.
- **A quarterly human review** of the medium and low long tail and of every open exception.

The Dependabot config and CI gate we start from:

```yaml
# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: npm
    directory: /
    schedule:
      interval: weekly
  - package-ecosystem: docker
    directory: /
    schedule:
      interval: weekly
```

```
$ trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 registry.example.com/api:sha-3f2c1a
```

`--ignore-unfixed` hides findings with no fix available, which is what keeps the gate actionable; `--exit-code 1` makes the job fail when anything remains. The scanning and admission side is in [container vulnerability scanning](/blog/container-vulnerability-scanning).

## Exceptions, with an expiry

Sometimes the SLO can't be met: no upstream patch yet, the fix breaks something critical, the component retires in two weeks, or the fix is a major-version bump that needs testing. Then we file an exception with the CVE and component, the reason, the compensating control (WAF rule, network restriction, alert), the expiry date, and a named owner (a person, not a team). Trivy's ignore file supports this directly:

```
# .trivyignore
# musl in the base image; not reachable from our code, rebuild lands with the next base bump. Owner: kiril
CVE-2019-14697 exp:2026-12-15
```

After the `exp:` date the finding reappears and fails the gate again, which forces the re-review. Exceptions live in one file in the security repo and get walked monthly.

## The emergency playbook

A few times a year something drops that interrupts everything. The sequence:

1. **Confirm reach.** Which repos, images and versions use it? An SBOM per image turns this from days into minutes; our [SBOM and attestation guide](/blog/supply-chain-security-sbom-attestation) covers generating them.
2. **Assess exposure.** Internet-facing first, internal after.
3. **Mitigate.** WAF rule, feature flag off, network restriction. Buy time.
4. **Patch** with the normal CI pipeline. Panicked patches that skip tests break things worse.
5. **Verify** the fixed version is what is actually running, and rescan; transitive dependencies sometimes need a second rebuild.
6. **Communicate:** internal updates, status page if customer-facing, a post-incident doc once stable.

The hard part is always step 1.

## Failure modes we keep seeing

- **`TOOMANYREQUESTS` when Trivy downloads its database in CI.** Many pipelines pulling the vulnerability DB at once hit registry rate limits. Cache the DB between jobs or point `--db-repository` at a mirror.
- **The gate goes green but the CVE is still running.** The image was rebuilt but pods were not rolled, or the base image tag was cached. Verify the digest in production, not the one in CI.
- **Every row treated as urgent.** The team starts ignoring the scanner. Triage on KEV, EPSS and reachability first.
- **Accepted risks with no expiry.** The team rotates and the CVE stays open for years. Every exception gets an `exp:` date.
- **No inventory.** "Do we use it?" should take minutes. Without SBOMs it takes days.

## Where this breaks

KEV and EPSS only cover published CVEs, so they say nothing about a zero-day in your own code or a malicious package with no CVE yet. Reachability analysis is language-dependent and misses reflection and dynamic loading. The SLO table assumes you can deploy daily; a regulated or embedded product with quarterly releases needs a separate hotfix path. And if you sell software in the EU, the Cyber Resilience Act adds reporting duties on top of this, covered in [what EU CRA Article 14 means for DevOps teams](/blog/eu-cra-article-14-what-devops-teams-do-now).

## Questions people ask

### How fast should critical vulnerabilities be patched?

Our default is 24 hours for a critical CVE that is known to be exploited and sits on an internet-facing service, 72 hours if internal, and 7 days for criticals with no exploitation evidence. The exploitation signal matters more than the score, so check CISA's KEV catalog and the CVE's EPSS probability before setting the deadline.

### What is the difference between CVSS and EPSS?

CVSS rates how severe a vulnerability would be if exploited, on a 0 to 10 scale, and the current version is 4.0. EPSS, also from FIRST, estimates the probability that a CVE will be exploited in the wild in the next 30 days, refreshed daily. Use CVSS for impact and EPSS plus CISA's KEV catalog for urgency.

### Is it ever acceptable to not patch a vulnerability?

Yes, when the vulnerable code is unreachable, the component is about to be retired, or no fix exists and a compensating control is in place. Write it down as an exception with the reason, the control, a named owner and an expiry date, then review it before it expires. Undocumented acceptance is how CVEs stay open for years.

## The call we'd make

You will not patch everything instantly, and teams claiming zero open vulnerabilities are usually not looking. Aim instead for every CVE seen, triaged and bucketed: KEV-listed and exposed gets fixed today, the routine handles the rest on SLO, and exceptions expire. Make the routine boring and automated so the emergencies stay rare.

## Sources

- [CISA: Known Exploited Vulnerabilities Catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)
- [FIRST: Exploit Prediction Scoring System (EPSS)](https://www.first.org/epss/)
- [FIRST: CVSS v4.0 specification](https://www.first.org/cvss/v4.0/)
- [Trivy docs: Filtering findings](https://trivy.dev/docs/latest/configuration/filtering/)

