Step-by-step debugging of a production Linux server hitting 100% CPU. From top to perf to the actual fix.
Linux Performance Troubleshooting: A Real Incident Walkthrough
Last month our API servers started responding in 8 seconds instead of 200ms. Here's exactly how we diagnosed and fixed it using standard Linux tools.
Output showed:
- CPU: 98% user, 1% system, 1% idle
- One process (
node) consuming 94% CPU
- Load average: 12.4 on a 4-core machine
ps aux --sort=-%cpu | head -5
The culprit was our Node.js API process, PID 28431.
strace -c -p 28431 -e trace=read,write,futex
Results: 89% of syscalls were futex (lock contention) and read from a file descriptor.
ls -la /proc/28431/fd | wc -l
# Result: 4,847
lsof -p 28431 | grep -c "TCP"
# Result: 4,201 TCP connections
We had 4,201 open TCP connections. Our connection pool had no limit, and a downstream service was responding slowly, causing connections to pile up.
ss -tnp | grep 28431 | awk '{print $4}' | sort | uniq -c | sort -rn | head
Result: 3,800 connections to port 5432 (PostgreSQL) in ESTABLISHED state. The database wasn't the bottleneck—our pool was creating connections faster than queries completed.
- Added connection pool limits in our database client:
const pool = new Pool({
max: 20, // was unlimited
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 5000,
});
- Added a circuit breaker for the slow downstream service
- Set up alerts for open file descriptor count:
# /etc/prometheus/alerts/fd_alert.yml
- alert: HighFileDescriptors
expr: process_open_fds > 1000
for: 5m
labels:
severity: warning
- Start with
top, then drill down with ps, strace, lsof
- Check file descriptors when CPU is high—connection leaks are a common cause
- Set limits on everything: connection pools, file descriptors (
ulimit), memory
- Monitor proactively: track open FDs, TCP connections, and load average
- Reproduce in staging before making production changes when possible
The issue wasn't a code bug in the traditional sense—it was a missing configuration. Default "unlimited" settings are the cause of more outages than most teams realize.
