Horizontal and vertical autoscalers solve different problems and break in different ways. The thresholds, cooldowns, and conflicts we learned the hard way.
Kubernetes HPA and VPA — Tuning From Production Pain
The HorizontalPodAutoscaler and VerticalPodAutoscaler are advertised as "set it and forget it" — Kubernetes scales your workload when load arrives, you go to sleep. In practice neither defaults are right for any production workload we've run, and the two of them conflict in ways the docs don't make obvious until your pods are flapping at 3am.
This is what we've learned tuning HPA/VPA across ~40 services over two years.
HPA scales replica count based on observed load. CPU is the default metric; you can also use memory, custom metrics from Prometheus, or external metrics (queue depth). Add pods when load is up, remove when down.
VPA changes the resource requests/limits on a pod based on observed usage. The pod stays at one replica conceptually; its CPU/memory allocation changes.
The mental model: HPA = "how many copies?"; VPA = "how big should each copy be?"
You use HPA for workloads that can shard horizontally (most stateless services). You use VPA for workloads where replicas don't help — a single coordinator process, a worker doing serial CPU-heavy work, anything with hot state that's hard to split.
The default HPA config waits 5 minutes after scale-up before scaling down. It uses a target CPU utilization (e.g., 70%) and scales until average across pods hits the target. Sounds reasonable. Two problems:
Thundering herd on scale-up. A traffic spike hits, CPU jumps to 95%, HPA decides to add 3 pods. Those pods take 30-90 seconds to start (image pull, app initialization). During that window the existing pods are still at 95%. By the time the new ones come up, the spike is over and HPA sees CPU at 40%. Five minutes later it scales back down. Repeat on the next spike.
The fix that worked: scale up aggressively, scale down conservatively. Set behavior.scaleUp.stabilizationWindowSeconds: 0 and policies allowing rapid scale-up (e.g., double replicas every 30s). Set behavior.scaleDown.stabilizationWindowSeconds: 300 and conservative scale-down policies (max 10% per minute).
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 30
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60
This costs more during sustained idle (the autoscaler is slow to shrink) but prevents flap. We accept the cost.
CPU is often the wrong metric. A worker process consuming a queue isn't bottlenecked on CPU — it's bottlenecked on the rate of work. CPU might be 30% while the queue depth is 50,000 messages. HPA on CPU never scales up.
The fix: scale on the metric that correlates with user-visible latency. For queue consumers, that's queue depth. For HTTP services, often it's p95 latency or in-flight requests. For an LLM gateway, it's tokens-in-flight or batch utilization.
We've used the Prometheus Adapter to expose custom metrics. Setting it up is a 4-hour task; the payoff is HPA that scales when load is up, not when CPU happens to be up.
VPA has three modes:
- Off — recommends only; doesn't apply.
- Initial — applies recommendations at pod creation; doesn't change running pods.
- Auto — actively evicts and recreates pods to apply new resource recommendations.
Auto mode evicts pods when it wants to resize them. For most production workloads, that's disruptive. We run Initial on everything and Off (recommendation-only) on stateful workloads.
The pattern that works:
- Run VPA in
Off mode for 2 weeks on a workload.
- Review the recommended request/limit values weekly.
- Update the deployment manifest with the recommended values manually.
- Repeat quarterly or when the workload changes.
This is more work than Auto but avoids the "VPA decided to evict your pod during peak traffic" problem.
HPA scales by replica count based on CPU. VPA changes the CPU request. If you run both on the same workload and the same metric, they fight: VPA reduces CPU requests because CPU usage is low; HPA sees CPU utilization (as a fraction of request) jump and scales out; VPA sees average CPU drop again and lowers requests further. Death spiral.
The Kubernetes docs say "don't run HPA and VPA on the same metric." That's correct but not actionable until you've felt it.
Patterns that work:
- HPA on a custom metric (queue depth, RPS), VPA on CPU/memory in
Off mode. Most common in our setup. HPA handles "how many"; VPA tells us how to size each pod manually.
- HPA on CPU, no VPA. Set requests/limits from observation; let HPA handle scaling.
- VPA
Auto on a single-replica workload, no HPA. Vertical scaling only.
What we don't do: run HPA on CPU and VPA in Auto mode on the same workload. The few times we tried it (because the docs said it was supported with care), we got flap within hours.
Once you've picked a metric, what target value? The target is usually wrong out of the box.
For CPU-based HPA: target 50-70% utilization. Higher = more efficient but slower to react. Lower = more headroom but wasteful. We default to 60%; some latency-sensitive services run at 40%.
For latency-based HPA: target p95 of half your SLO. If your SLO is 200ms p95, scale to keep p95 below 100ms — gives headroom before you breach.
For queue depth: depends on the consumer's processing rate. We target "queue depth = 30 seconds of work" — if a worker processes 100 msg/sec, target 3000 messages. Above that, scale up.
The single most impactful tuning we did: split scale-up and scale-down cooldowns.
- scale-up: fast (0-30s stabilization). Real traffic doesn't wait.
- scale-down: slow (5-15 minutes). Avoid removing pods that you'll need again in a minute.
Most flap problems are scale-down problems. Be patient on scaling down and most flap goes away.
Treating "average CPU" as truthful. A workload with 1 hot pod (95% CPU) and 9 cold pods (5% CPU) averages 14%. HPA scales down; the hot pod gets hotter; latency suffers. Solution: use topk of CPU across pods if your routing is uneven, or fix the routing.
Forgetting init containers. A pod with a heavy init container takes 60+ seconds to be Ready. HPA's stabilization windows need to account for that, or you'll over-scale.
Setting minReplicas too low. A 1-pod minimum means the first traffic hit goes to one cold pod. We set minReplicas to the count needed to handle baseline + 30% headroom. Cold starts cost more than the extra pod.
Not testing scaling under load. Read the HPA config, deploy it, hope. Doesn't work. Run load tests that exercise scale-up and scale-down before trusting any config. We use k6 with ramping profiles.
- Replica count over time. Flap = repeated up/down within minutes.
- Scale events. Why did HPA make a decision?
kubectl describe hpa shows recent reasons.
- Pod startup time. Long start = HPA can't keep up with sudden spikes regardless of policy.
- VPA recommendations vs. current requests. Big delta = workload has drifted and you should review.
Autoscaling on Kubernetes is one of those features that's easy to enable and hard to operate well. The defaults are starting points, not endpoints. Tune for the load shape you actually see; measure flap; accept that perfect autoscaling is the wrong goal — good enough, predictable autoscaling is.
