Linux Performance Troubleshooting: A Real Incident Walkthrough

Linux Performance Troubleshooting: A Real Incident Walkthrough#

Last month our API servers started responding in 8 seconds instead of 200ms. Here's exactly how we diagnosed and fixed it using standard Linux tools.

Step 1: top - Get the Big Picture#

top -bn1 | head -20

Output showed:

• CPU: 98% user, 1% system, 1% idle
• One process (node) consuming 94% CPU
• Load average: 12.4 on a 4-core machine

Step 2: Identify the Process#

ps aux --sort=-%cpu | head -5

The culprit was our Node.js API process, PID 28431.

Step 3: strace - What's It Doing?#

strace -c -p 28431 -e trace=read,write,futex

Results: 89% of syscalls were futex (lock contention) and read from a file descriptor.

Step 4: Check File Descriptors#

ls -la /proc/28431/fd | wc -l
# Result: 4,847

lsof -p 28431 | grep -c "TCP"
# Result: 4,201 TCP connections

We had 4,201 open TCP connections. Our connection pool had no limit, and a downstream service was responding slowly, causing connections to pile up.

Step 5: Confirm with ss#

ss -tnp | grep 28431 | awk '{print $4}' | sort | uniq -c | sort -rn | head

Result: 3,800 connections to port 5432 (PostgreSQL) in ESTABLISHED state. The database wasn't the bottleneck—our pool was creating connections faster than queries completed.

The Fix#

1. Added connection pool limits in our database client:

const pool = new Pool({
  max: 20,              // was unlimited
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
});

2. Added a circuit breaker for the slow downstream service
3. Set up alerts for open file descriptor count:

# /etc/prometheus/alerts/fd_alert.yml
- alert: HighFileDescriptors
  expr: process_open_fds > 1000
  for: 5m
  labels:
    severity: warning

Best Practices for Linux Performance#

• Start with top, then drill down with ps, strace, lsof
• Check file descriptors when CPU is high—connection leaks are a common cause
• Set limits on everything: connection pools, file descriptors (ulimit), memory
• Monitor proactively: track open FDs, TCP connections, and load average
• Reproduce in staging before making production changes when possible

The issue wasn't a code bug in the traditional sense—it was a missing configuration. Default "unlimited" settings are the cause of more outages than most teams realize.