Handling Vulnerabilities in Production: What We Actually Do
You always have known vulnerabilities. The question is how you triage, patch, and respond. A KEV-and-EPSS triage discipline that holds up through incidents and routine work.
Key takeaways
- Triage every CVE on exploitation evidence, reachability and blast radius, not CVSS alone.
- Anything in CISA's Known Exploited Vulnerabilities catalog on an internet-facing service gets fixed within 24 hours; the rest follows patching SLOs by severity.
- Automate dependency updates, rebuild images on a schedule, and give every accepted risk an owner and an expiry date.
On this page
Triage every CVE on three things: is it exploited in the wild (CISA's KEV catalog, FIRST's EPSS score), is the vulnerable code reachable, and what sits behind it. Known-exploited plus internet-facing means fix within 24 hours. Everything else follows a severity-based patching SLO, and every exception gets a named owner and an expiry date.
Verified against
Trivy v0.76.1, CVSS v4.0, CISA KEV catalog version 2026.10.30 (1,766 entries), against first.org, cisa.gov and the trivy.dev filtering docs. Checked October 2026.
If you run non-trivial software, you have known vulnerabilities right now: open CVEs in OS packages, an unpatched advisory in some npm or Python dependency, a bundled library upstream abandoned. That is normal. The question is whether you have a system for handling them. This is the one we run after a few real incidents and a lot of routine maintenance.
Scanner output is a baseline, not a crisis
Scanners will report hundreds of findings. Teams that treat every red row as urgent burn out, stop reading the scanner and miss the one that mattered. Teams that cope have a triage step: most findings go to "patch on schedule", a few to "patch now", and energy goes where it counts.
Triage on exploitation, reachability and blast radius
CVSS (now at version 4.0) measures how bad a flaw could be, not how likely it is to hurt you. A critical 9.8 in a function you never call matters less than a medium 6.4 in your auth path. We add three signals:
Exploitation evidence. CISA's Known Exploited Vulnerabilities (KEV) catalog lists CVEs confirmed exploited in the wild; it held 1,766 entries as of October 30, 2026. FIRST's EPSS publishes, every day, a 0 to 1 probability that a CVE will be exploited in the next 30 days. Both are free and scriptable.
Reachability. Is the vulnerable code actually called? Reachability analysis in modern SCA tools separates the transitive dependency that is never imported from the function your request path hits. More on that in our dependency scanning (SCA) guide.
Blast radius. A public payments API outranks an internal nightly cron job with the same CVE.
This check takes a list of CVE IDs from a scan and flags the ones that need a human today:
curl -s https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json \
| jq -r '.vulnerabilities[].cveID' > kev.txt
grep -Fxf kev.txt scan-cves.txt # any output here is a fix-now candidate
CVE-2021-44228
curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2021-44228' | jq -r '.data[] | "\(.cve) epss=\(.epss) pct=\(.percentile)"'
CVE-2021-44228 epss=0.999990000 pct=1.000000000
Log4Shell still scores 0.99999 on EPSS as of October 30, 2026, five years after disclosure. Old does not mean safe.
The four buckets
Fix now (within 24 hours). In KEV or with a public exploit, reachable, internet-exposed. Pager-worthy. Recent examples: Log4Shell, the polyfill.io supply-chain attack, and in October 2026 PHP2Shell, the unauthenticated RCE in PHP core, GitLab's CVSS 10 regex RCE and the exploited Linux TCP bug on Kubernetes nodes.
Fix soon (within 30 days). High severity, reachable, no exploitation evidence yet. The bulk of patch work, scheduled into the next sprint with a hard date.
Fix later (within 90 days). Medium severity, or high but unreachable, or in a non-public component. Rides the routine patch cycle.
Accept. Low severity, no reachable path, or no upstream fix with a mitigation in place. Documented with an owner and an expiry. "Accepted" must never become "forgotten."
Patching SLOs
The buckets become deadlines:
| Severity | Internet-exposed | Internal |
|---|---|---|
| Critical + known exploited | 24h | 72h |
| Critical | 7 days | 14 days |
| High | 30 days | 60 days |
| Medium | 90 days | 180 days |
| Low | best effort | best effort |
The clock starts at CVE publication or at first evidence of exploitation, whichever comes first. The owner is the team that owns the component; at-risk SLOs escalate.
The routine that catches most of it
Most vulnerability work is not emergency work. It is a rhythm:
- Automated dependency PRs on every repo. We auto-merge patch versions for trusted packages and review minor and major bumps.
- Scheduled base image rebuilds, so new OS-level fixes land without anyone doing anything.
- A scan gate in CI that fails on fixable high and critical findings.
- A monthly OS patch window rolled out dev, then staging, then prod.
- A quarterly human review of the medium and low long tail and of every open exception.
The Dependabot config and CI gate we start from:
version: 2
updates:
- package-ecosystem: npm
directory: /
schedule:
interval: weekly
- package-ecosystem: docker
directory: /
schedule:
interval: weekly
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 registry.example.com/api:sha-3f2c1a
--ignore-unfixed hides findings with no fix available, which is what keeps the gate actionable; --exit-code 1 makes the job fail when anything remains. The scanning and admission side is in container vulnerability scanning.
Exceptions, with an expiry
Sometimes the SLO can't be met: no upstream patch yet, the fix breaks something critical, the component retires in two weeks, or the fix is a major-version bump that needs testing. Then we file an exception with the CVE and component, the reason, the compensating control (WAF rule, network restriction, alert), the expiry date, and a named owner (a person, not a team). Trivy's ignore file supports this directly:
# .trivyignore
# musl in the base image; not reachable from our code, rebuild lands with the next base bump. Owner: kiril
CVE-2019-14697 exp:2026-12-15
After the exp: date the finding reappears and fails the gate again, which forces the re-review. Exceptions live in one file in the security repo and get walked monthly.
The emergency playbook
A few times a year something drops that interrupts everything. The sequence:
- Confirm reach. Which repos, images and versions use it? An SBOM per image turns this from days into minutes; our SBOM and attestation guide covers generating them.
- Assess exposure. Internet-facing first, internal after.
- Mitigate. WAF rule, feature flag off, network restriction. Buy time.
- Patch with the normal CI pipeline. Panicked patches that skip tests break things worse.
- Verify the fixed version is what is actually running, and rescan; transitive dependencies sometimes need a second rebuild.
- Communicate: internal updates, status page if customer-facing, a post-incident doc once stable.
The hard part is always step 1.
Failure modes we keep seeing
TOOMANYREQUESTSwhen Trivy downloads its database in CI. Many pipelines pulling the vulnerability DB at once hit registry rate limits. Cache the DB between jobs or point--db-repositoryat a mirror.- The gate goes green but the CVE is still running. The image was rebuilt but pods were not rolled, or the base image tag was cached. Verify the digest in production, not the one in CI.
- Every row treated as urgent. The team starts ignoring the scanner. Triage on KEV, EPSS and reachability first.
- Accepted risks with no expiry. The team rotates and the CVE stays open for years. Every exception gets an
exp:date. - No inventory. "Do we use it?" should take minutes. Without SBOMs it takes days.
Where this breaks
KEV and EPSS only cover published CVEs, so they say nothing about a zero-day in your own code or a malicious package with no CVE yet. Reachability analysis is language-dependent and misses reflection and dynamic loading. The SLO table assumes you can deploy daily; a regulated or embedded product with quarterly releases needs a separate hotfix path. And if you sell software in the EU, the Cyber Resilience Act adds reporting duties on top of this, covered in what EU CRA Article 14 means for DevOps teams.
Questions people ask
How fast should critical vulnerabilities be patched?
Our default is 24 hours for a critical CVE that is known to be exploited and sits on an internet-facing service, 72 hours if internal, and 7 days for criticals with no exploitation evidence. The exploitation signal matters more than the score, so check CISA's KEV catalog and the CVE's EPSS probability before setting the deadline.
What is the difference between CVSS and EPSS?
CVSS rates how severe a vulnerability would be if exploited, on a 0 to 10 scale, and the current version is 4.0. EPSS, also from FIRST, estimates the probability that a CVE will be exploited in the wild in the next 30 days, refreshed daily. Use CVSS for impact and EPSS plus CISA's KEV catalog for urgency.
Is it ever acceptable to not patch a vulnerability?
Yes, when the vulnerable code is unreachable, the component is about to be retired, or no fix exists and a compensating control is in place. Write it down as an exception with the reason, the control, a named owner and an expiry date, then review it before it expires. Undocumented acceptance is how CVEs stay open for years.
The call we'd make
You will not patch everything instantly, and teams claiming zero open vulnerabilities are usually not looking. Aim instead for every CVE seen, triaged and bucketed: KEV-listed and exposed gets fixed today, the routine handles the rest on SLO, and exceptions expire. Make the routine boring and automated so the emergencies stay rare.
Sources
Cite this article
Kiril Urbonas (2026). Handling Vulnerabilities in Production: What We Actually Do. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/handling-vulnerabilities-in-production
[Handling Vulnerabilities in Production: What We Actually Do](https://www.devopsness.com/blog/handling-vulnerabilities-in-production) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from DevOps
Explore more articles in this category
The HTTP Security Headers Guide (with CSP)
A practical walkthrough of the HTTP response headers that harden a web app, with a real rollout plan for Content-Security-Policy.
Exactly-Once Delivery: Myth, Reality, and What to Do Instead
Exactly-once network delivery is impossible, but exactly-once processing is achievable through at-least-once delivery plus idempotent consumers.
Chaos Engineering: What We Actually Run as Game Days
How we'd run a quarterly chaos game day. The scenarios that tend to surface real problems, the ones that rarely do, and the operational discipline that makes the practice pay back.
You might have missed
Evergreen posts worth revisiting.