cgroup v2 Limits: What Actually Constrains Your Containers

cgroup v2 Limits: What Actually Constrains Your Containers#

A latency-sensitive service was set to --cpus=2 and its dashboard showed CPU at "100%", but requests that normally took 8ms were spiking to 90ms in bursts. The team assumed they needed more cores. They didn't. The container was being throttled by the CFS quota, sitting idle for milliseconds at a time while it waited for the next scheduling period. The metric that mattered wasn't CPU usage, it was nr_throttled in the cgroup's cpu.stat. That's the whole reason it pays to understand what cgroup v2 actually enforces.

Where the knobs live#

On a cgroup v2 system (default on most current distros: systemd mounts it at /sys/fs/cgroup) every container gets a directory of control files. For a container run by Docker under systemd, it's roughly:

$ cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.max
200000 100000

Those two numbers are quota and period in microseconds. 200000 100000 means "200ms of CPU time per 100ms period", i.e. 2 full cores. That's what --cpus=2 sets. Crucially, this is a hard ceiling enforced per 100ms window. If your service does a burst of work needing 3 cores for 60ms, it exhausts its 200ms budget partway through the window and gets parked until the window rolls over. That park is the latency spike.

Reading the throttling that CPU% hides#

CPU utilization graphs average over seconds and completely hide sub-second throttling. The truth is here:

$ cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.stat
usage_usec 48120000
nr_periods 61000
nr_throttled 9200
throttled_usec 1380000

nr_throttled 9200 out of nr_periods 61000 means the container hit its ceiling in roughly 15% of scheduling windows. That's the smoking gun. The fix was not more cores on average, it was raising the quota to absorb bursts (--cpus=3), or better, widening the period so short bursts don't trip the ceiling. After bumping to 3 cores, nr_throttled fell to near zero and p99 dropped back to single-digit milliseconds even though average CPU barely moved. Throttling, not saturation, was the problem the entire time.

Memory is a cliff, not a slope#

memory.max is the hard limit. Cross it and the kernel OOM-kills a process inside the cgroup, immediately, no grace.

$ cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.max
536870912          # 512 MiB
$ cat .../memory.current
502341632
$ cat .../memory.events
low 0
high 0
max 14
oom 1
oom_kill 1

memory.events is the file people forget. oom_kill 1 means the kernel already killed something here, which is why your container "mysteriously restarted". The subtler field is high. If you set memory.high (below memory.max), the kernel throttles allocations and reclaims aggressively as you approach it, giving the app a chance to shed memory before the hard kill. Kubernetes doesn't set memory.high by default, so you get the cliff, not the slope. Setting a memory.high roughly 10% under the limit turned hard OOM kills into gradual reclaim pressure for one of our JVM services, and the pod stopped getting killed outright.

One more trap: memory.max counts page cache too. A process doing heavy file I/O can push memory.current up with cache pages and trigger reclaim even though its actual heap is small. Watch memory.stat for file vs anon before you assume a leak.

IO limits nobody sets until they wish they had#

io.max throttles block I/O per device, keyed by major:minor. It's rarely set because it needs the device number, but it's the only thing standing between a noisy batch job and your latency-sensitive neighbor on the same disk.

$ echo "259:0 rbps=104857600 wbps=52428800" \
    > /sys/fs/cgroup/.../io.max

That caps the cgroup to 100 MB/s read and 50 MB/s write on device 259:0. We reached for this exactly once, when a nightly analytics container was starving the database's cgroup on a shared NVMe. Capping the batch job's write bandwidth fixed database write latency without touching the database at all.

The call we'd make#

Stop reading CPU percentage as the health signal for constrained containers. Alert on nr_throttled / nr_periods from cpu.stat and on oom_kill from memory.events; those two files explain most "it's slow but CPU looks fine" and "it keeps restarting" mysteries. Set memory.high below memory.max so memory pressure is a ramp instead of a cliff. And keep io.max in your back pocket for the day a batch job tries to eat the disk.