cgroup v2 Limits: What Actually Constrains Your Containers
A container with a 2-core limit was pegged at 100% CPU yet running slow. The throttling counter told the real story, and it wasn't the number we set.
Key takeaways
- Under cgroup v2, a container is constrained by cpu.max (a CPU quota per 100ms period), memory.max (a hard limit that triggers the OOM killer), memory.high (a throttle point) and io.max (per-device bandwidth).
- CPU percentage hides quota throttling, so read nr_throttled in cpu.stat and oom_kill in memory.events instead.
On this page
Under cgroup v2, three files constrain a container: cpu.max caps CPU time per 100ms period, memory.max is a hard ceiling that invokes the OOM killer, and io.max throttles disk bandwidth per device. When a limited container is slow but CPU looks fine, read nr_throttled in cpu.stat. When it keeps restarting, read oom_kill in memory.events.
Verified against
Linux kernel cgroup v2 admin guide, Docker Engine resource-constraints docs, Kubernetes 1.37 (cgroup v1 deprecated since 1.35), against docs.kernel.org, docs.docker.com and kubernetes.io. Checked October 2026.
Here is the failure pattern this post is about. A latency-sensitive service runs with --cpus=2, its dashboard shows CPU at "100%", and requests that normally take 8ms spike to 90ms in bursts. The obvious conclusion is "it needs more cores." Usually it doesn't. The container is being throttled by its CPU quota, sitting idle for milliseconds at a time while it waits for the next scheduling period. The metric that matters is not CPU usage but nr_throttled in the cgroup's cpu.stat.
Where the knobs live
On a cgroup v2 system (the default on current distros, mounted by systemd at /sys/fs/cgroup) every container gets a directory of control files. Kubernetes now expects this: the kubelet refuses to start on a cgroup v1 node by default, and cgroup v1 has been deprecated since Kubernetes 1.35, as covered in what the kubelet's cgroup v1 change means. For a container run by Docker under systemd, the path looks roughly like this:
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.max
200000 100000
The format is $MAX $PERIOD in microseconds, and the kernel default is max 100000. 200000 100000 means "200ms of CPU time per 100ms period", which is 2 full cores. Docker's docs describe --cpus as shorthand for --cpu-period=100000 plus a matching --cpu-quota, so --cpus=2 writes exactly this. Crucially, it is a hard ceiling enforced per 100ms window. If your service does a burst of work needing 3 cores for 60ms, it exhausts its 200ms budget partway through the window and gets parked until the window rolls over. That park is the latency spike.
Reading the throttling that CPU% hides
CPU utilization graphs average over seconds and hide sub-second throttling completely. The truth is here:
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.stat
usage_usec 48120000
nr_periods 61000
nr_throttled 9200
throttled_usec 1380000
In this example output, nr_throttled 9200 out of nr_periods 61000 means the container hit its ceiling in roughly 15% of scheduling windows. That's the smoking gun. The fix is not more cores on average. It is raising the quota enough to absorb bursts (--cpus=3), or reducing how bursty the work is. When the quota covers the burst, nr_throttled falls toward zero and tail latency recovers even though average CPU barely moves. The kernel docs add one caveat: these counters only cover throttling caused by this cgroup's own limit, not throttling inherited from a parent cgroup, so check the parent slice too when the numbers don't add up.
Memory is a cliff, not a slope
memory.max is the hard limit. When usage hits it and the kernel cannot reclaim enough, the OOM killer runs inside the cgroup, with no grace period.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.max
536870912
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.events
low 0
high 0
max 14
oom 1
oom_kill 1
That memory.max value is 512 MiB. memory.events is the file people forget. oom_kill 1 means the kernel already killed a process here, which is why your container "mysteriously restarted" (exit code 137; see Kubernetes exit codes for the mapping). max 14 counts the times usage was about to cross the limit.
The subtler field is high. If you set memory.high below memory.max, the kernel throttles the cgroup and puts it under heavy reclaim pressure as it approaches that boundary, and it never invokes the OOM killer at that point. That gives the app a chance to shed memory before the hard kill. Kubernetes only writes memory.high when the MemoryQoS feature gate is on, so a default cluster gets the cliff, not the slope. For runtimes like the JVM that grow into whatever headroom they see, a memory.high around 10% under the limit is a reasonable starting point to test.
One more trap: the memory controller charges page cache as well as anonymous memory. A process doing heavy file I/O can push memory.current up with cache pages and trigger reclaim even though its actual heap is small. Compare file and anon in memory.stat before you assume a leak.
IO limits nobody sets until they wish they had
io.max throttles block I/O per device, keyed by major:minor. It's rarely set because it needs the device number, but it's the only thing standing between a noisy batch job and the latency-sensitive neighbor on the same disk. The keys are rbps, wbps, riops and wiops:
lsblk -o NAME,MAJ:MIN | grep nvme0n1
nvme0n1 259:0
echo "259:0 rbps=104857600 wbps=52428800" \
| sudo tee /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max
That caps the cgroup to 100 MiB/s read and 50 MiB/s write on device 259:0. The textbook use case is a nightly analytics container starving a database on a shared NVMe drive. Capping the batch job's write bandwidth protects database write latency without touching the database at all. Docker exposes the same control as --device-write-bps.
Questions people ask
How do I check if a container is CPU throttled?
Read cpu.stat in the container's cgroup directory under /sys/fs/cgroup. Divide nr_throttled by nr_periods to get the share of 100ms windows where the container hit its quota, and watch throttled_usec grow over time. In Kubernetes, cAdvisor exports the same data as container_cpu_cfs_throttled_periods_total. Anything consistently above a few percent on a latency-sensitive service is worth a higher limit.
What is the difference between memory.high and memory.max?
memory.max is the hard limit: if the kernel can't reclaim below it, the OOM killer kills a process in the cgroup. memory.high is a throttle point: above it the cgroup is slowed down and pushed into aggressive reclaim, but nothing is killed. Setting memory.high a little under memory.max turns a sudden OOM kill into gradual pressure the app can survive.
Does Kubernetes still support cgroup v1?
Barely. cgroup v1 is deprecated as of Kubernetes 1.35, and the kubelet won't start on a cgroup v1 node unless an admin sets failCgroupV1: false. cgroup v2 needs Linux 5.8 or later, a runtime such as containerd 1.4+ or CRI-O 1.20+, and the systemd cgroup driver. Plan node images on cgroup v2 now.
The call we'd make
Stop reading CPU percentage as the health signal for constrained containers. Alert on nr_throttled / nr_periods from cpu.stat and on oom_kill from memory.events; those two files explain most "it's slow but CPU looks fine" and "it keeps restarting" mysteries. Set memory.high below memory.max where your runtime allows it, so memory pressure is a ramp instead of a cliff. And keep io.max in your back pocket for the day a batch job tries to eat the disk.
Sources
Cite this article
Kiril Urbonas (2026). cgroup v2 Limits: What Actually Constrains Your Containers. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/cgroup-v2-limits-what-constrains-containers
[cgroup v2 Limits: What Actually Constrains Your Containers](https://www.devopsness.com/blog/cgroup-v2-limits-what-constrains-containers) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from Linux
Explore more articles in this category
Bash Scripting Tutorial: Write Your First Useful Script
Build a real disk-cleanup script step by step. Learn variables, conditionals, loops, error handling, and the safety preamble that prevents foot-guns.
DNS Troubleshooting on Linux: A Systematic Approach
\"It's always DNS\" is a joke because the failure modes are so scattered: resolver config, caching, search domains, split DNS. Here is where to actually look.
Shell Scripting Best Practices: Writing Maintainable Scripts
We have a few hundred shell scripts in production. The patterns that make them survive contact with reality, and the ones we've stopped writing.
You might have missed
Evergreen posts worth revisiting.