Monitoring as Code: Observability in IaC

About two years ago our monitoring lived in a mix of Datadog UI clicks, Grafana dashboards drifted from anyone's memory, and PagerDuty configs nobody could explain. We migrated everything to "monitoring as code" — dashboards, alerts, and SLOs in Git. This post is the working version after the dust settled.

Why we migrated#

Specific pain points that drove the change:

Drift. Two engineers click-fixed a dashboard slightly differently. Now there are two slightly-different copies. Nobody is sure which is canonical.

No code review. A new alert was created by clicking around. It was misconfigured. It paged at 3 AM for nothing. Nobody had reviewed it before it went live.

No history. "Why is this alert configured this way?" "Who knows; whoever set it up is no longer on the team."

Replicating across environments. A new staging environment needs the same dashboards as prod. "Just click through and re-create them" is slow and error-prone.

Restoring after corruption. A botched Datadog API call deleted a bunch of monitors. Restoring them was hours of work.

All of these go away when monitoring lives in Git.

What we put in Git#

Categories:

Dashboards (Grafana, Datadog): JSON definitions in repo. Synced via operator (Grafana operator) or CD pipeline (Datadog API on merge).

Alerts / Monitors: YAML or JSON definitions. Same sync pattern.

SLOs: separate definition files. Each SLO has a target, a SLI, an alerting policy, and an error budget.

Notification policies: who gets paged for what. Routes, escalations, oncall schedules (where the tool allows).

Recording rules (Prometheus): definitions in YAML, applied to Prometheus.

Custom queries / saved views: useful queries we've built up live in repo as runbook content.

What we did NOT put in Git:

• Ad-hoc dashboards engineers create for one-off investigations (these stay in the UI; we delete them periodically)
• Per-incident temporary monitors (auto-delete after the incident)
• User preferences (theme, default time range, etc.)

Tooling#

For our stack:

Grafana: dashboards as JSON, applied via the Grafana operator (reads ConfigMaps). Alerts via grafana-operator's CRDs.

Datadog: dashboards and monitors managed via Terraform (Datadog provider). Could use Datadog's CLI or API directly; Terraform fits our existing IaC tooling.

PagerDuty: schedules, escalation policies, services managed via Terraform.

Prometheus: alert rules as YAML, applied via the kube-prometheus-stack chart. Recording rules same way.

The pattern: each tool has some way to apply config from files. We use whatever fits — Terraform for SaaS APIs, K8s operators for in-cluster tools, plain CD pipelines for the rest.

Repo structure#

Our monitoring repo:

monitoring/
  dashboards/
    grafana/
      service-overview.json
      database-health.json
      ...
    datadog/
      api-performance.json
      ...
  alerts/
    grafana/
      ...
    datadog/
      ...
  slos/
    api-availability.yaml
    checkout-latency.yaml
  pagerduty/
    services.tf
    schedules.tf

Each file is reviewable on its own. PRs touching monitoring go through the same code review as anything else.

Workflow#

A typical monitoring change:

1. Engineer opens a PR adding/editing a YAML or JSON file.
2. CI validates: schema checks, simulates apply (dry-run via the tool's API).
3. Reviewer reviews the change. Catches obvious issues (typos, wrong thresholds, bad routing).
4. Merge → CD pipeline applies.
5. Change is live.

The biggest cultural change: monitoring changes go through code review. Engineers got used to clicking through monitoring tools; now they edit YAML and wait for a review. Friction is real but the quality benefit is clear.

Standardizing dashboards#

Across ~80 services, we wanted dashboards to look similar so engineers could orient quickly. Standard dashboard structure:

• Top row: golden signals (request rate, error rate, p50/p95/p99 latency)
• Middle row: dependencies (database calls, downstream service calls, cache hit rate)
• Bottom row: resource usage (CPU, memory, network)

We have a Grafana dashboard template. Adding a new service: copy the template, change the data source filter to the new service. ~10 minutes vs an afternoon for hand-built dashboards.

For Datadog, similar — a dashboard template that team owners copy and customize.

Alert hygiene#

Alerts as code makes alert hygiene visible. We have rules:

Every alert has an owner. A team or person; not "platform." Documented in the alert metadata.

Every alert has a runbook. A URL pointing to documentation. The runbook explains what the alert means, what to investigate, and what to do.

Every alert has a severity. Page (wakes someone up), ticket (work item, daytime response), info (FYI in chat).

Every alert has a "for" duration. Don't fire on a single bad data point.

We have a CI check that rejects alert PRs missing any of these. Without enforcement, the discipline drifts.

Alert review: kill the noise#

Quarterly review:

• Alerts that fired and were resolved without action: investigate, raise threshold or delete
• Alerts that haven't fired in 6 months: are they still useful? Often yes (catastrophic conditions); sometimes they're outdated
• Alerts that pulse-fire and self-recover: bad alert (or bad threshold)

We've gotten our alert volume down to a sustainable level via these reviews. Without them, alerts accumulate and become noise.

SLO definitions#

For services with defined SLOs, we have YAML definitions:

name: api-availability
service: api
sli:
  type: availability
  query: 'sum(rate(http_requests_total{service="api",status=~"5.."}[5m])) / sum(rate(http_requests_total{service="api"}[5m]))'
target: 0.999  # 99.9% availability
window: 30d
alerting:
  burn_rate_fast: { window: 1h, threshold: 14.4 }   # 2% burn in 1h
  burn_rate_slow: { window: 6h, threshold: 6.0 }    # 5% burn in 6h

A small operator reads these and creates the corresponding Prometheus rules and Grafana dashboards. The YAML is the source; everything else is generated.

When the SLO target changes (e.g., we tighten from 99.5% to 99.9%), edit the YAML. The downstream artifacts regenerate.

Migration: how we did it#

The migration took ~3 months. Approach:

1. Audit. List every dashboard, monitor, and alert. ~400 things across various tools.
2. Categorize. "Keep, delete, merge with another." About 30% got deleted (unused, redundant). The rest got migrated.
3. Migrate by team. Each team's monitoring moved from UI to code over a sprint.
4. Lock the UI. Once migrated, we removed UI write access for the migrated tools (still readable; not editable).
5. Final cleanup. Stragglers and edge cases.

The "lock UI" step was important. Without it, engineers continued to click-edit the migrated artifacts; drift returned. With UI write access removed (or at least restricted to admins for emergencies), the discipline held.

What we got wrong#

Specific issues during the migration:

Dashboard exports had volatile fields. Datadog dashboard JSON includes generated fields (timestamps, internal IDs). Exporting and committing these caused git diffs on every save even when nothing meaningful changed. We wrote a normalizer that strips volatile fields before commit.

Auto-generated names conflicting. Tools generate IDs from names; renaming a dashboard would create a duplicate. We added pre-merge checks for ID conflicts.

Apply pipelines failed silently. Initial CD pipeline reported "applied" even when the API rejected the change. We added explicit verification that the apply took effect.

Backward-incompatible Datadog API changes. Once or twice, Datadog changed the API and our committed configs broke. Pinned to specific Terraform provider versions and updated deliberately.

What we still don't do#

A few corners we haven't tackled:

On-call schedule rotations as code. PagerDuty schedules are partially in code, but actual rotation (who's on call this week) is still managed in the UI. The complexity of "fairness across timezones, vacation, etc." doesn't fit cleanly in static config.

Notebook-style operational analyses. When someone investigates an incident in a Datadog notebook, that's not in Git. We're OK with this — it's exploratory work.

Auto-generated dashboards. Some teams have asked for "spin up a dashboard automatically when a new service deploys." We haven't built this; teams write their own dashboards using the template.

Cost#

Operational cost:

• Monitoring tools themselves (Datadog, Grafana managed, etc.): primary monitoring spend, unchanged by IaC migration
• The CI/CD infrastructure to apply monitoring config: minor, shared with other CI
• Engineer time on migration: ~3 months of one engineer's effort
• Engineer time ongoing: maybe an hour/week to address operator issues

Compared to the operational cost of drift, broken dashboards, and noisy alerts: large positive ROI.

What I'd tell a team starting#

Migrate one tool at a time. Don't try to convert everything at once.

Lock UI write access after migration. Without this, drift returns.

Standardize templates. Dashboards that look similar across services are easier to navigate.

Every alert has owner, runbook, severity, duration. Enforce in CI.

Review alerts quarterly. Without active pruning, noise accumulates.

SLOs as YAML; downstream artifacts generated. One source of truth.

Don't put ad-hoc work in Git. Investigation dashboards stay in the UI; production monitoring goes in Git.

Monitoring as code isn't glamorous infrastructure work, but the day-to-day quality-of-life improvements are real. Engineers stop fighting drift; new services get good dashboards instantly; alerts are reviewed before firing. Like most discipline-driven improvements, the value compounds — six months in, the system is much better than the click-driven version, and you don't really notice anymore. That's the goal.