Linux Performance Troubleshooting: A Real Incident Walkthrough
Step-by-step debugging of a production Linux server hitting 100% CPU. From top to perf to the actual fix.
On this page
Last month our API servers started responding in 8 seconds instead of 200ms. Here's exactly how we diagnosed and fixed it using standard Linux tools.
Step 1: top - Get the Big Picture
top -bn1 | head -20
Output showed:
- CPU: 98% user, 1% system, 1% idle
- One process (
node) consuming 94% CPU - Load average: 12.4 on a 4-core machine
Step 2: Identify the Process
ps aux --sort=-%cpu | head -5
The culprit was our Node.js API process, PID 28431.
Step 3: strace - What's It Doing?
strace -c -p 28431 -e trace=read,write,futex
Results: 89% of syscalls were futex (lock contention) and read from a file descriptor.
Step 4: Check File Descriptors
ls -la /proc/28431/fd | wc -l
# Result: 4,847
lsof -p 28431 | grep -c "TCP"
# Result: 4,201 TCP connections
We had 4,201 open TCP connections. Our connection pool had no limit, and a downstream service was responding slowly, causing connections to pile up.
Step 5: Confirm with ss
ss -tnp | grep 28431 | awk '{print $4}' | sort | uniq -c | sort -rn | head
Result: 3,800 connections to port 5432 (PostgreSQL) in ESTABLISHED state. The database wasn't the bottleneck—our pool was creating connections faster than queries completed.
The Fix
- Added connection pool limits in our database client:
const pool = new Pool({
max: 20, // was unlimited
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 5000,
});
- Added a circuit breaker for the slow downstream service
- Set up alerts for open file descriptor count:
- alert: HighFileDescriptors
expr: process_open_fds > 1000
for: 5m
labels:
severity: warning
Best Practices for Linux Performance
- Start with
top, then drill down withps,strace,lsof - Check file descriptors when CPU is high—connection leaks are a common cause
- Set limits on everything: connection pools, file descriptors (
ulimit), memory - Monitor proactively: track open FDs, TCP connections, and load average
- Reproduce in staging before making production changes when possible
The issue wasn't a code bug in the traditional sense—it was a missing configuration. Default "unlimited" settings are the cause of more outages than most teams realize.
Cite this article
Kiril Urbonas (2026). Linux Performance Troubleshooting: A Real Incident Walkthrough. DevOpsNess. https://www.devopsness.com/blog/linux-performance-troubleshooting-a-real-incident-walkthrough
[Linux Performance Troubleshooting: A Real Incident Walkthrough](https://www.devopsness.com/blog/linux-performance-troubleshooting-a-real-incident-walkthrough) (Kiril Urbonas, DevOpsNess, 2026-03-31)
More from Linux
Explore more articles in this category
Systemd Service Management: Creating and Managing Services
A practical guide to writing and managing systemd services for production. The unit file features that earn their place, plus the operational workflows.
Practical Guide: Linux Performance Baseline Methodology
When everything seems "slow," a baseline gives you something to measure against. The capture-and-compare workflow we use on every Linux host.
Shell Scripting Best Practices: Writing Maintainable Scripts
We have a few hundred shell scripts in production. The patterns that make them survive contact with reality, and the ones we've stopped writing.
You might have missed
Evergreen posts worth revisiting.