Prometheus vs Datadog — When Open Source Wins

We've run both, sometimes in the same quarter. A team inherits a Datadog bill that crossed six figures, panics, and asks if Prometheus can replace it. Another team is drowning in a homegrown Prometheus setup that pages nobody because Alertmanager was never wired up, and they want to buy their way out. Neither switch is free. The honest answer depends on what you actually monitor and who's around to keep it running.

So let's do this the way we'd do it at a whiteboard, not the way a vendor deck does it.

What each one actually is#

Prometheus is a pull-based metrics database. It scrapes HTTP endpoints your services expose, stores time series locally, and answers PromQL queries. Around it you bolt Alertmanager for routing pages and Grafana for dashboards. Three separate tools, all open source, all yours to run. The software costs nothing. The EC2 instances, the object storage, and the engineer who gets paged when the TSDB fills its disk are not nothing.

Datadog is a managed SaaS agent that pushes everything to their cloud: metrics, traces, logs, real user monitoring, synthetics, the works. You install one agent, click some integrations, and dashboards appear. You pay per host, per million custom metrics, per gigabyte of logs ingested, per APM host, and per a dozen other line items that each looked small when you signed up.

That framing already tells you the trade. One is a metrics tool you assemble and operate. The other is an everything platform you rent.

Cost, honestly#

The "Prometheus is free" line is true and misleading. The binary is free. Running it at any real scale is not. You'll pay for compute, for persistent volumes, and eventually for a long-term storage layer. But that cost scales with your infrastructure and your competence, not with a price sheet. A team that knows what it's doing runs metrics for a mid-size fleet on a few thousand dollars a month of infrastructure.

Datadog bills per host and per volume. That model is fine at ten hosts and brutal at a thousand, especially once custom metrics and log ingestion climb. The bills that shock people are almost never the base host fee. They're custom metrics cardinality and log volume, the two things that grow silently until finance forwards you a chart.

Rough shape of it:

Scaling Prometheus past one box#

A single Prometheus server is a single point of failure with local disk retention measured in weeks. That's fine for a homelab and unacceptable for a platform team. This is where the open source story gets more involved.

The usual fixes are Thanos, Mimir, or Cortex. Thanos ships your TSDB blocks to object storage (S3, GCS) and gives you a global query view plus deduplication across replica pairs. Mimir, from the Grafana folks, is the more opinionated horizontally scalable option and tends to be what larger teams land on now. Cortex is the older ancestor of both. All three solve the same two problems: long-term retention in cheap object storage, and high availability so one dead node doesn't blind you.

The catch is that you're now operating a distributed system to hold your monitoring for your other distributed system. That is real work. Budget for it or don't start.

Cardinality and retention, the tax nobody quotes#

Both systems punish high cardinality, just differently. In Prometheus, a label with millions of unique values (user IDs, request IDs, full URLs) blows up memory and can OOM the server. You feel it as an outage. In Datadog, that same explosion arrives as a custom-metrics overage on the invoice. You feel it as a finance conversation. Same root cause, different failure surface.

Retention is the mirror image. Prometheus keeps data cheaply only if you've built out object storage. Datadog keeps it for you but charges for longer windows, and the default retention is shorter than most people assume until they go looking for last quarter's data and it's gone.

Alerting and lock-in#

Alertmanager is genuinely good once configured: dedup, grouping, silences, routing to PagerDuty or Slack. It's also a config file that a human has to get right, and a badly configured Alertmanager is worse than none because it breeds alert fatigue. Datadog's alerting is friendlier out of the box and ties directly to the same tags you monitor on, which is a real convenience.

Lock-in is the quiet decider. Prometheus speaks open formats. OpenMetrics, remote-write, and PromQL are portable, and Grafana points at anything. If you want to leave, you can. Datadog's value is the integrated whole, and that whole is proprietary. The agent, the tags, the dashboards, and the stored data don't come with you. The switching cost is exactly as high as the convenience was pleasant.

For the broader picture on where these sit among other tooling, our roundup of observability tools covers the adjacent options.

A config we'd actually ship#

Here's a scrape job plus a recording rule, the boring stuff that makes PromQL fast in production:

# prometheus.yml
scrape_configs:
  - job_name: 'api'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ['api-1:9090', 'api-2:9090']
    metric_relabel_configs:
      # drop a high-cardinality label before it hits the TSDB
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        target_label: keep
        replacement: 'yes'

# rules.yml
groups:
  - name: api-latency
    rules:
      - record: job:http_request_p99:5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))

That metric_relabel_configs block is the discipline Datadog charges you to skip. You decide what not to keep before it costs you.

The call we'd make#

If you're metrics-centric, cost-sensitive, and have at least one engineer who can own the stack, run Prometheus. Add Mimir or Thanos the day retention and HA stop being optional, and wire Alertmanager properly on day one, not day ninety. You'll spend engineering time and save real money, and you'll own your data.

If you want traces, logs, RUM, and dashboards on the same day with zero ops headcount to spare, buy Datadog and accept the bill as the price of not staffing an observability team. Just cap your custom metrics and log ingestion early, before the invoice teaches you to. The tool that wins is the one your team can actually operate.