# Linux System Monitoring with Prometheus and Grafana

URL: https://www.devopsness.com/blog/linux-system-monitoring-prometheus-grafana
Author: Kiril Urbonas
Published: 2024-02-07
Updated: 2024-02-07
Categories: Linux
Tags: Prometheus, Grafana, Monitoring, Observability, Linux

Summary: Set up comprehensive Linux system monitoring using Prometheus and Grafana. Monitor CPU, memory, disk, network, and application metrics with beautiful dashboards.

---
# Linux System Monitoring with Prometheus and Grafana

Comprehensive system monitoring is essential for maintaining healthy Linux servers. This guide shows you how to set up Prometheus and Grafana for production-grade monitoring.

## Why Prometheus and Grafana?

- **Prometheus**: Time-series database optimized for metrics
- **Grafana**: Powerful visualization and alerting
- **Open Source**: Free and community-driven
- **Scalable**: Handles millions of metrics

## Architecture Overview

```
┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│   Node      │────▶│  Prometheus  │────▶│   Grafana   │
│  Exporter   │     │   (Metrics)  │     │ (Dashboards)│
└─────────────┘     └──────────────┘     └─────────────┘
```

## Installation

### 1. Install Prometheus

```bash
# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*

# Create systemd service
sudo tee /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
```

### 2. Install Node Exporter

```bash
# Download Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo cp node_exporter-*/node_exporter /usr/local/bin/

# Create systemd service
sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
Type=simple
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
```

### 3. Configure Prometheus

```yaml
# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
        labels:
          instance: 'server-01'
          environment: 'production'
```

### 4. Install Grafana

```bash
# Add Grafana repository
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

# Install Grafana
sudo apt-get update
sudo apt-get install grafana

# Start Grafana
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
```

## Key Metrics to Monitor

### System Metrics

- **CPU Usage**: `node_cpu_seconds_total`
- **Memory**: `node_memory_MemTotal_bytes`, `node_memory_MemAvailable_bytes`
- **Disk I/O**: `node_disk_io_time_seconds_total`
- **Network**: `node_network_receive_bytes_total`, `node_network_transmit_bytes_total`
- **Load Average**: `node_load1`, `node_load5`, `node_load15`

### Application Metrics

- **Request Rate**: `http_requests_total`
- **Error Rate**: `http_requests_total{status=~"5.."}`
- **Response Time**: `http_request_duration_seconds`
- **Active Connections**: `http_connections_active`

## PromQL Queries

### CPU Usage

```
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
```

### Memory Usage

```
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100
```

### Disk Usage

```
100 - ((node_filesystem_avail_bytes{mountpoint="/"} * 100) / node_filesystem_size_bytes{mountpoint="/"})
```

## Grafana Dashboard Setup

### 1. Add Prometheus Data Source

1. Go to Configuration → Data Sources
2. Click "Add data source"
3. Select Prometheus
4. Enter URL: `http://localhost:9090`
5. Click "Save & Test"

### 2. Import Dashboard

1. Go to Dashboards → Import
2. Enter dashboard ID: `1860` (Node Exporter Full)
3. Select Prometheus data source
4. Click "Import"

## Alerting Rules

```yaml
# /etc/prometheus/alerts.yml
groups:
  - name: system_alerts
    interval: 30s
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for 5 minutes"

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 90%"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is below 10%"
```

## Docker Compose Setup

```yaml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus

  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'

volumes:
  prometheus-data:
  grafana-data:
```

## Best Practices

1. **Retention Policy**: Set appropriate retention for metrics
2. **Labeling**: Use consistent labels for filtering
3. **Cardinality**: Avoid high-cardinality labels
4. **Alerts**: Set up meaningful alert thresholds
5. **Dashboards**: Create focused dashboards per team/service
6. **Backup**: Regularly backup Grafana dashboards

## Conclusion

Prometheus and Grafana provide a powerful, open-source solution for Linux system monitoring. With proper configuration, you can gain deep insights into your infrastructure and applications, enabling proactive problem resolution.

