# cgroup v2 Limits: What Actually Constrains Your Containers

URL: https://www.devopsness.com/blog/cgroup-v2-limits-what-constrains-containers
Author: Kiril Urbonas
Published: 2026-06-29
Updated: 2026-10-31
Categories: Linux
Tags: Linux, cgroups, Containers, Performance, Linux Kernel

Summary: A container with a 2-core limit was pegged at 100% CPU yet running slow. The throttling counter told the real story, and it wasn't the number we set.

---
# cgroup v2 Limits: What Actually Constrains Your Containers

Under cgroup v2, three files constrain a container: `cpu.max` caps CPU time per 100ms period, `memory.max` is a hard ceiling that invokes the OOM killer, and `io.max` throttles disk bandwidth per device. When a limited container is slow but CPU looks fine, read `nr_throttled` in `cpu.stat`. When it keeps restarting, read `oom_kill` in `memory.events`.

> [!VERIFIED]
> Linux kernel cgroup v2 admin guide, Docker Engine resource-constraints docs, Kubernetes 1.37 (cgroup v1 deprecated since 1.35), against docs.kernel.org, docs.docker.com and kubernetes.io. Checked October 2026.

Here is the failure pattern this post is about. A latency-sensitive service runs with `--cpus=2`, its dashboard shows CPU at "100%", and requests that normally take 8ms spike to 90ms in bursts. The obvious conclusion is "it needs more cores." Usually it doesn't. The container is being throttled by its CPU quota, sitting idle for milliseconds at a time while it waits for the next scheduling period. The metric that matters is not CPU usage but `nr_throttled` in the cgroup's `cpu.stat`.

## Where the knobs live

On a cgroup v2 system (the default on current distros, mounted by systemd at `/sys/fs/cgroup`) every container gets a directory of control files. Kubernetes now expects this: the kubelet refuses to start on a cgroup v1 node by default, and cgroup v1 has been deprecated since Kubernetes 1.35, as covered in [what the kubelet's cgroup v1 change means](/blog/kubelet-cgroup-v1-kubernetes-1-35). For a container run by Docker under systemd, the path looks roughly like this:

```bash
$ cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.max
200000 100000
```

The format is `$MAX $PERIOD` in microseconds, and the kernel default is `max 100000`. `200000 100000` means "200ms of CPU time per 100ms period", which is 2 full cores. Docker's docs describe `--cpus` as shorthand for `--cpu-period=100000` plus a matching `--cpu-quota`, so `--cpus=2` writes exactly this. Crucially, it is a hard ceiling enforced per 100ms window. If your service does a burst of work needing 3 cores for 60ms, it exhausts its 200ms budget partway through the window and gets parked until the window rolls over. That park is the latency spike.

## Reading the throttling that CPU% hides

CPU utilization graphs average over seconds and hide sub-second throttling completely. The truth is here:

```bash
$ cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.stat
usage_usec 48120000
nr_periods 61000
nr_throttled 9200
throttled_usec 1380000
```

In this example output, `nr_throttled 9200` out of `nr_periods 61000` means the container hit its ceiling in roughly 15% of scheduling windows. That's the smoking gun. The fix is not more cores on average. It is raising the quota enough to absorb bursts (`--cpus=3`), or reducing how bursty the work is. When the quota covers the burst, `nr_throttled` falls toward zero and tail latency recovers even though average CPU barely moves. The kernel docs add one caveat: these counters only cover throttling caused by this cgroup's own limit, not throttling inherited from a parent cgroup, so check the parent slice too when the numbers don't add up.

## Memory is a cliff, not a slope

`memory.max` is the hard limit. When usage hits it and the kernel cannot reclaim enough, the OOM killer runs inside the cgroup, with no grace period.

```bash
$ cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.max
536870912
$ cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.events
low 0
high 0
max 14
oom 1
oom_kill 1
```

That `memory.max` value is 512 MiB. `memory.events` is the file people forget. `oom_kill 1` means the kernel already killed a process here, which is why your container "mysteriously restarted" (exit code 137; see [Kubernetes exit codes](/blog/kubernetes-exit-codes) for the mapping). `max 14` counts the times usage was about to cross the limit.

The subtler field is `high`. If you set `memory.high` below `memory.max`, the kernel throttles the cgroup and puts it under heavy reclaim pressure as it approaches that boundary, and it never invokes the OOM killer at that point. That gives the app a chance to shed memory before the hard kill. Kubernetes only writes `memory.high` when the MemoryQoS feature gate is on, so a default cluster gets the cliff, not the slope. For runtimes like the JVM that grow into whatever headroom they see, a `memory.high` around 10% under the limit is a reasonable starting point to test.

One more trap: the memory controller charges page cache as well as anonymous memory. A process doing heavy file I/O can push `memory.current` up with cache pages and trigger reclaim even though its actual heap is small. Compare `file` and `anon` in `memory.stat` before you assume a leak.

## IO limits nobody sets until they wish they had

`io.max` throttles block I/O per device, keyed by `major:minor`. It's rarely set because it needs the device number, but it's the only thing standing between a noisy batch job and the latency-sensitive neighbor on the same disk. The keys are `rbps`, `wbps`, `riops` and `wiops`:

```bash
$ lsblk -o NAME,MAJ:MIN | grep nvme0n1
nvme0n1     259:0
$ echo "259:0 rbps=104857600 wbps=52428800" \
    | sudo tee /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max
```

That caps the cgroup to 100 MiB/s read and 50 MiB/s write on device `259:0`. The textbook use case is a nightly analytics container starving a database on a shared NVMe drive. Capping the batch job's write bandwidth protects database write latency without touching the database at all. Docker exposes the same control as `--device-write-bps`.

## Questions people ask

### How do I check if a container is CPU throttled?

Read `cpu.stat` in the container's cgroup directory under `/sys/fs/cgroup`. Divide `nr_throttled` by `nr_periods` to get the share of 100ms windows where the container hit its quota, and watch `throttled_usec` grow over time. In Kubernetes, cAdvisor exports the same data as `container_cpu_cfs_throttled_periods_total`. Anything consistently above a few percent on a latency-sensitive service is worth a higher limit.

### What is the difference between memory.high and memory.max?

`memory.max` is the hard limit: if the kernel can't reclaim below it, the OOM killer kills a process in the cgroup. `memory.high` is a throttle point: above it the cgroup is slowed down and pushed into aggressive reclaim, but nothing is killed. Setting `memory.high` a little under `memory.max` turns a sudden OOM kill into gradual pressure the app can survive.

### Does Kubernetes still support cgroup v1?

Barely. cgroup v1 is deprecated as of Kubernetes 1.35, and the kubelet won't start on a cgroup v1 node unless an admin sets `failCgroupV1: false`. cgroup v2 needs Linux 5.8 or later, a runtime such as containerd 1.4+ or CRI-O 1.20+, and the systemd cgroup driver. Plan node images on cgroup v2 now.

## The call we'd make

Stop reading CPU percentage as the health signal for constrained containers. Alert on `nr_throttled / nr_periods` from `cpu.stat` and on `oom_kill` from `memory.events`; those two files explain most "it's slow but CPU looks fine" and "it keeps restarting" mysteries. Set `memory.high` below `memory.max` where your runtime allows it, so memory pressure is a ramp instead of a cliff. And keep `io.max` in your back pocket for the day a batch job tries to eat the disk.

## Sources

- [Linux kernel: Control Group v2 admin guide](https://docs.kernel.org/admin-guide/cgroup-v2.html)
- [Docker docs: Resource constraints](https://docs.docker.com/engine/containers/resource_constraints/)
- [Kubernetes docs: About cgroup v2](https://kubernetes.io/docs/concepts/architecture/cgroups/)
- [Kubernetes releases](https://kubernetes.io/releases/)

