SLI Design: Picking Metrics That Actually Correlate With User Experience

The most expensive SLI mistake is the one where the dashboard stays green while users churn. We once had 214 alerts wired into PagerDuty and a checkout flow that still felt slow. During an incident review a support lead played a recording of a user rage-clicking "Place order" while every dashboard in the room glowed green: CPU fine, memory fine, no pod restarts. Pick the wrong indicator (average latency instead of p99, raw 5xx rate instead of errors users saw) and you get a reliability program that is correct about things nobody cares about. This is the discipline we now run for picking SLIs that move when user experience moves.

The point of an SLI, restated

A Service Level Indicator is a measurement that should correlate with whether users are having a good time. The SLO is the threshold; the SLI is the measurement. Get the SLI wrong and the SLO is meaningless.

The test: when users report problems, the SLI should be moving the wrong way, and when users are happy it should be in target. That sounds tautological, and most inherited SLIs still fail it.

The quickest filter we know: for every candidate, finish this sentence out loud. "If this gets worse, a user will notice because ___." If you can't finish it, it's a diagnostic, not an SLI. Diagnostics are useful during an incident; they don't belong on the reliability scorecard. CPU at 80% doesn't finish the sentence. Checkout p99 crossing 2 seconds does: they feel the spinner. That one filter cut our candidate list from roughly forty metrics to four categories.

The five-step audit

For every customer-facing service, we periodically run this check:

1. List what users actually care about. For a checkout API: did my payment succeed? Was it fast?
2. Map each one to a measurable metric. Payment success becomes the success rate of POST /checkout; speed becomes its response time.
3. Backtest against past incidents. During the last bad day, did this metric move?
4. Tune the aggregation. Averages hide tail latency; medians hide the 5% bad day. Use p95 or p99 for latency and rate-of-bad-events for errors.
5. Check the inverse. When the metric was bad, were users actually unhappy, or was it noise?

Most services have 1-3 SLIs that survive. Six per service is a sign of confusion, not rigor.

Common bad SLIs

Average latency. A p50 of 80ms looks great while p99 sits at 8 seconds. Users who hit the tail churn.

Server-side latency only. The gap between what the app measures and what the user waits for is queueing, and queueing is exactly what spikes under load. We now measure at the load balancer, not inside the app.

Raw 5xx rate. A service can be 100% 200s while returning wrong data, and a 500 that a client retried successfully hurt nobody. Pair status codes with content correctness where you can, and count only errors the user saw.

Uptime from an external pinger. A synthetic /health check returns 200 while a specific user's request fails on lock contention specific to their data. Sample real user traffic.

CPU and memory usage. Causes, not experience. Users don't care whether CPU is at 50% or 90% if requests succeed and are fast.

Aggregate error rate across all endpoints. If /search is at 0.1% errors and /checkout at 10%, the aggregate looks fine because search dominates volume.

The four signals that survive

Everything that passed the audit fell into four categories.

Latency on critical operations, as percentiles. Request time on POST /checkout, POST /login, GET /me, each its own SLI. We track p95 for "this is getting slow" trends and p99 for "the tail is exploding" issues that show up before most users notice. Match the percentile to the journey: p99 for an interactive search box, p50 is enough for a nightly export. Our checkout query:

# p99 latency, checkout submit, 5m window
histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{
      route="/api/checkout/submit", method="POST"
    }[5m])
  )
)

Target: p99 under 800ms. Internal testing showed abandonment climbing sharply past a full second, and we wanted headroom.

Errors that reached the user. Success rate per critical operation, counting only non-retried failures:

sum(rate(http_requests_total{route=~"/api/.*", code=~"5..", retried="false"}[5m]))
/
sum(rate(http_requests_total{route=~"/api/.*"}[5m]))

The retried="false" label dropped our reported error rate from 0.9% to 0.11%, because most of the 0.9% was retried and invisible to customers. Chasing the inflated number would have burned a sprint on nothing. For B2B, we add a per-customer error rate for the top 50 accounts: the aggregate looks fine while one large customer sits at 30% errors, which otherwise surfaces only when the account team gets an angry email. And for async work, background job completion rate: a 200 doesn't mean the email sent or the webhook fired.

Saturation of the one resource that gates users. Not every queue depth, just the resource that makes users wait when it fills. For us that was the database connection pool:

max_over_time(pgbouncer_pools_cl_waiting[1m]) > 5

Waiting clients on PgBouncer meant requests were parked. That gauge predicted latency incidents about ten minutes ahead of the latency SLI, which made it our best leading indicator. We treat it as an early warning, not a scorecard number.

Freshness, the signal people forget. For anything data-driven (an analytics feed, a recommendation model, a replicated read store), "is it correct" really means "is it recent." Stale data returns HTTP 200 and looks healthy:

SELECT EXTRACT(EPOCH FROM (now() - max(event_time))) AS lag_seconds
FROM analytics.orders_replica;

We alert past 300 seconds. Green pipelines serving three-hour-old data caused two of our worst customer complaints last year, and no latency or error metric would ever have caught them.

How we test correlation

Backtest against incidents. For each significant incident in the last 6 months, look at the SLI during the window. If it was steady during a user-reported outage, it's the wrong SLI.

Forward correlation against user feedback. Plot support ticket volume against the SLI. If tickets spike while the SLI is green, something is missing.

We've reworked our SLIs twice this way. Both times we found we were tracking technical health (server CPU, queue depth) and missing user-facing outcomes (a specific feature failing, latency in one region).

Targets, scope and alerting

Once the SLI is right, the target is easier:

• Critical user-facing operations: 99.9% success, p99 under 1s (typical for our shape).
• Background and async: 99.5% completion within 5 minutes.
• Internal tools: 99%; internal users are more tolerant.

Don't pick targets above what the team can maintain. 99.99% sounds great until the fourth 2am page of the month.

Not everything gets an SLI. We focus on the handful of critical customer-facing services, internal services whose failure would break one of those, and a few high-volume background processors. Admin tools, batch jobs and exploratory data work get basic health monitoring without SLOs.

Alert on the SLO by burn rate, not by "the SLI crossed a line for 30 seconds": a fast-burn page when you'd exhaust a month's budget in about two days, a slower rule for gradual bleed. The rules and window pairs are in burn-rate alerting: the SLO discipline that prevents alert fatigue; the broader pattern is in our deep dive on SLO-based monitoring for APIs, and the dashboards on top in monitoring that actually helps on-call.

Questions people ask

What are the four golden signals for SLIs?

Latency, errors, traffic and saturation are the classic four. For user-facing SLIs we swap traffic for freshness: latency as percentiles, errors the user actually saw, saturation of the single resource that makes users wait, and data freshness for anything that serves stored or replicated data. Traffic is useful context but rarely tells you a user is hurting.

How many SLIs should a service have?

Two to four. One success-rate and one latency SLI per critical operation covers most services, plus freshness or job completion where data or async work is involved. Past that you're tracking diagnostics, and the scorecard stops meaning anything.

Should I measure latency at the server or the load balancer?

The load balancer, or as close to the user as you can get. Server-side timing misses queueing in front of the app, which is exactly what grows under load.

The decision, concretely

• Candidate metric that fails "a user will notice because ___"? Demote it to a diagnostic dashboard.
• Latency SLI on an average or measured inside the app? Switch to p95/p99 at the load balancer.
• Error SLI counting retried requests? Filter them out, or you'll chase phantom errors.
• Data-serving system with no freshness SLI? Add a lag query and alert past your staleness tolerance.

The call we'd make

Start over rather than keep the old dashboards "just in case." Pick latency, user-visible errors, the one gating resource and freshness; backtest each against your last few incidents; wire burn-rate alerts on those and page on nothing else. Keep the old wall of graphs as diagnostics you open during an incident. Our page volume went from 214 alerts to 9, and the nine meant something. The caveat: SLIs drift as the product changes, so rerun the audit at least once a year, which takes about an afternoon per service.

Sources

• Google SRE Book: Monitoring Distributed Systems (the four golden signals)
• Google SRE Workbook: Alerting on SLOs