Skip to main content
Linux Performance Troubleshooting: A Real Incident Walkthrough

Linux Performance Troubleshooting: A Real Incident Walkthrough

KU
· Founder & Chief Editor
• 2 min read•43 views

Step-by-step debugging of a production Linux server hitting 100% CPU. From top to perf to the actual fix.

Last month our API servers started responding in 8 seconds instead of 200ms. Here's exactly how we diagnosed and fixed it using standard Linux tools.

Step 1: top - Get the Big Picture

Bash
top -bn1 | head -20

Output showed:

  • CPU: 98% user, 1% system, 1% idle
  • One process (node) consuming 94% CPU
  • Load average: 12.4 on a 4-core machine

Step 2: Identify the Process

Bash
ps aux --sort=-%cpu | head -5

The culprit was our Node.js API process, PID 28431.

Step 3: strace - What's It Doing?

Bash
strace -c -p 28431 -e trace=read,write,futex

Results: 89% of syscalls were futex (lock contention) and read from a file descriptor.

Step 4: Check File Descriptors

Bash
ls -la /proc/28431/fd | wc -l
# Result: 4,847

lsof -p 28431 | grep -c "TCP"
# Result: 4,201 TCP connections

We had 4,201 open TCP connections. Our connection pool had no limit, and a downstream service was responding slowly, causing connections to pile up.

Step 5: Confirm with ss

Bash
ss -tnp | grep 28431 | awk '{print $4}' | sort | uniq -c | sort -rn | head

Result: 3,800 connections to port 5432 (PostgreSQL) in ESTABLISHED state. The database wasn't the bottleneck—our pool was creating connections faster than queries completed.

The Fix

  1. Added connection pool limits in our database client:
JavaScript
const pool = new Pool({
  max: 20,              // was unlimited
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
});
  1. Added a circuit breaker for the slow downstream service
  2. Set up alerts for open file descriptor count:
/etc/prometheus/alerts/fd_alert.yml
- alert: HighFileDescriptors
  expr: process_open_fds > 1000
  for: 5m
  labels:
    severity: warning

Best Practices for Linux Performance

  • Start with top, then drill down with ps, strace, lsof
  • Check file descriptors when CPU is high—connection leaks are a common cause
  • Set limits on everything: connection pools, file descriptors (ulimit), memory
  • Monitor proactively: track open FDs, TCP connections, and load average
  • Reproduce in staging before making production changes when possible

The issue wasn't a code bug in the traditional sense—it was a missing configuration. Default "unlimited" settings are the cause of more outages than most teams realize.

Explore topics:Linux
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). Linux Performance Troubleshooting: A Real Incident Walkthrough. DevOpsNess. https://www.devopsness.com/blog/linux-performance-troubleshooting-a-real-incident-walkthrough

Markdown

[Linux Performance Troubleshooting: A Real Incident Walkthrough](https://www.devopsness.com/blog/linux-performance-troubleshooting-a-real-incident-walkthrough) (Kiril Urbonas, DevOpsNess, 2026-03-31)

KU

Kiril Urbonas

Founder & Chief Editor

596 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.