Most SLI dashboards track things nobody notices. Here's how we picked the handful of signals that map to real user pain, and dropped the vanity metrics.
Four Signals That Matter: Choosing SLIs Users Actually Feel
We had 214 alerts wired into PagerDuty and a checkout flow that still felt slow to customers. During one incident review a support lead pulled up a recording of a user rage-clicking the "Place order" button while every dashboard in the room glowed green. CPU was fine, memory was fine, the pods weren't restarting. The thing users touched was broken and nothing we measured said so.
That mismatch is the whole problem. You can watch a hundred host-level metrics and still miss the one number a person experiences. So we tore the dashboards down and rebuilt around signals a user would actually feel if they degraded.
The trick that changed things for us: for every SLI, finish this sentence out loud. "If this gets worse, a user will notice because ___." If you can't finish it, it's not an SLI, it's a diagnostic. Diagnostics are useful during an incident. They don't belong on the reliability scorecard.
CPU at 80% doesn't finish the sentence. A user cannot feel CPU. Checkout p99 crossing 2 seconds does finish it, they feel the spinner. That single filter cut our candidate list from roughly forty metrics to four categories.
Latency is the obvious one and the most commonly botched. Two mistakes we made and fixed:
We averaged. Averages hide the tail where users leave. Switch to percentiles and pick the one that matches the journey. For a search box, p99 matters because a slow tail feels janky. For a nightly export, p50 is enough.
We measured server time, not the time the user waits. The gap between them is queueing, and queueing is exactly what spikes under load. We now measure at the load balancer, not inside the app.
Here's the histogram query we run for the checkout submit endpoint:
histogram_quantile(
0.99,
sum by (le) (
rate(http_request_duration_seconds_bucket{
route="/api/checkout/submit", method="POST"
}[5m])
)
)
Our target: p99 under 800ms, measured over a 5-minute window. We picked 800ms because internal testing showed abandonment climbing sharply past a full second, and we wanted headroom.
Not every 500 in your logs hurt someone. A background retry that failed and then succeeded is noise. What counts is the error the user saw. We split error rate by whether the request was user-facing and whether it was retried.
sum(rate(http_requests_total{route=~"/api/.*", code=~"5..", retried="false"}[5m]))
/
sum(rate(http_requests_total{route=~"/api/.*"}[5m]))
The retried="false" label is the important part. Adding it dropped our reported error rate from 0.9% to 0.11%, because most of that 0.9% was retried and invisible to customers. Chasing the inflated number would have burned a sprint on nothing.
Saturation is where teams over-instrument. You do not need every queue depth on the wall. You need the one resource that, when full, makes users wait. For us that was the database connection pool. When it saturated, requests queued, and latency spiked, and the chain of pain started there.
max_over_time(pgbouncer_pools_cl_waiting[1m]) > 5
Any waiting clients on PgBouncer meant requests were parked. That single gauge predicted latency incidents about ten minutes ahead of the latency SLI itself, which made it our best leading indicator.
For anything data-driven, an analytics feed, a recommendation model, a replicated read store, "is it correct" is really "is it recent." Stale data returns HTTP 200 and looks perfectly healthy. Users notice when yesterday's inventory tells them something is in stock and it isn't.
We track lag directly:
SELECT EXTRACT(EPOCH FROM (now() - max(event_time))) AS lag_seconds
FROM analytics.orders_replica;
Alert if lag_seconds goes past 300. Green pipelines with three-hour-old data caused two of our worst customer complaints last year, and no latency or error metric would ever have caught them.
Four signals, one target each, and a burn-rate alert instead of a static threshold. Static thresholds page you at 3am for a blip. Burn rate pages you when you're actually spending your error budget fast enough to run out.
- alert: CheckoutLatencyBurn
expr: |
slo:latency_error_ratio:rate1h{service="checkout"} > (14.4 * 0.001)
and
slo:latency_error_ratio:rate5m{service="checkout"} > (14.4 * 0.001)
for: 2m
labels:
severity: page
The 14.4 multiplier against a 0.1% budget is the standard fast-burn window, it fires when you'd exhaust a month's budget in about two days.
If you're starting over, resist the urge to keep the old dashboards "just in case." Pick latency, user-visible errors, the one gating resource, and freshness. Wire burn-rate alerts on those four and page on nothing else. Everything the old wall-of-graphs gave us, we kept as diagnostics you open during an incident, not as things that wake someone. Our page volume dropped from 214 alerts to 9, and the nine actually meant something.
