Container performance problems usually live in the node kernel, not your app. Here is what we tune, why, and how we measure before touching anything.
Linux Performance Tuning for Containers and Kubernetes Nodes
Most "the app is slow" tickets that land on a platform team are not application bugs. They are the node kernel doing exactly what it was told to do, which happens to be the wrong thing for a container workload. The defaults on a fresh Ubuntu or Amazon Linux node are tuned for a general-purpose server, not for forty pods sharing cgroups and fighting over a conntrack table. This post is the checklist we actually run when a cluster starts misbehaving, in the order we run it.
One rule before any of it: measure first. We have watched teams double net.core.somaxconn on every node because a blog told them to, then spend a week confused about why nothing changed. Tune the thing the data points at, not the thing you read about last.
Pressure Stall Information (PSI) is the fastest way to tell whether a node is starved for CPU, memory, or I/O. It ships in the kernel and needs nothing installed.
# node-wide pressure, all three resources
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
# per-cgroup, e.g. a specific pod slice under cgroup v2
cat /sys/fs/cgroup/kubepods.slice/.../memory.pressure
The some avg10 number is the share of the last 10 seconds that at least one task was stalled waiting on that resource. Anything consistently above 5-10% on CPU or memory means the node is the bottleneck, and now you know which resource to chase. Pair PSI with node_exporter and kube-state-metrics so you get the same signal on a dashboard instead of by SSH.
The single most common hidden problem is CFS quota throttling. When a pod has a CPU limit, the kernel enforces it with a quota per 100ms period. If the pod burns its quota in 40ms, it gets frozen for the remaining 60ms of every period, even when the node is 90% idle. Latency-sensitive services fall off a cliff and the CPU graph looks fine.
Check it straight from the cgroup:
# cgroup v2
cat /sys/fs/cgroup/kubepods.slice/.../cpu.stat
# look at nr_throttled and throttled_usec climbing over time
Or in Prometheus:
rate(container_cpu_cfs_throttled_periods_total[5m])
/ rate(container_cpu_cfs_periods_total[5m])
If that ratio is meaningfully above zero for a service that feels slow, the limit is the problem. Options, roughly in order of how much we like them: raise the limit, remove the limit and rely on requests for scheduling, or for genuinely latency-sensitive workloads give the pod whole CPUs with the static CPU Manager policy so it stops sharing cores.
# kubelet config: pin guaranteed pods to exclusive cores
cpuManagerPolicy: static
kubeReserved:
cpu: "500m"
systemReserved:
cpu: "500m"
That only kicks in for Guaranteed-QoS pods with integer CPU requests. Requests, not limits, are what the scheduler packs against, so a node crammed to 100% of requests will throttle even without any single pod misbehaving.
Memory has no equivalent of throttling. A container that hits its memory limit gets OOM-killed, full stop. Under cgroup v2 the kernel reclaims page cache first, and memory.pressure will spike before the kill, which is your early warning.
Kubernetes historically demanded swap be off, and for years we just disabled it. Node-level swap support changed that, and for burstable workloads a little swap can absorb spikes that would otherwise kill a pod. We enable it cautiously and keep vm.swappiness low so the kernel treats it as a safety net, not a habit.
sysctl -w vm.swappiness=10
sysctl -w vm.overcommit_memory=1 # let the scheduler, not the kernel, gate memory
Watch container_memory_working_set_bytes against the limit, not RSS. Working set is what the OOM killer looks at.
Container image layers, logs, and emptyDir volumes all land on the node's ephemeral disk, and a single chatty pod can starve everyone else. On NVMe the none (or mq-deadline) scheduler usually beats the defaults; older cfq-style behavior just adds latency.
cat /sys/block/nvme0n1/queue/scheduler
echo none > /sys/block/nvme0n1/queue/scheduler
Set ephemeral-storage requests and limits so a runaway log file gets the pod evicted instead of filling the root filesystem and taking the kubelet down with it. For write-heavy workloads, mount container storage on its own volume with a filesystem you have actually tested (we default to ext4 or xfs and move on rather than chasing exotic options).
High-connection nodes hit two walls constantly. The first is the conntrack table. Every tracked connection consumes a slot, and when it fills you get silent packet drops and nf_conntrack: table full in dmesg.
# how full is it right now?
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
The second is the accept backlog and ephemeral port range for nodes that make lots of outbound connections. These are the sysctls we set on ingress and high-throughput nodes:
net.netfilter.nf_conntrack_max = 1048576
net.core.somaxconn = 32768
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
somaxconn only helps if the application actually passes a large backlog to listen(), so bumping it is necessary but not sufficient. If you run kube-proxy in iptables mode and see connection setup latency at scale, that is often the moment to move to IPVS mode, which scales service rules far better than a linear iptables chain.
A handful of kernel limits are per-node and bite container workloads specifically. vm.max_map_count is the classic: Elasticsearch, and anything with a large mmap footprint, will refuse to start below 262144. fs.inotify limits get exhausted by nodes running many pods that each watch files.
vm.max_map_count = 262144
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
fs.file-max = 2097152
Do not hand-edit /etc/sysctl.conf on live nodes. It drifts, and the next node the autoscaler brings up will not have it. Apply these through your provisioning path so they are reproducible. Ansible is our default for the base image:
- name: Container-node sysctls
ansible.posix.sysctl:
name: "{{ item.name }}"
value: "{{ item.value }}"
sysctl_set: true
state: present
reload: true
loop:
- { name: vm.max_map_count, value: "262144" }
- { name: fs.inotify.max_user_watches, value: "524288" }
- { name: net.core.somaxconn, value: "32768" }
For sysctls that a specific workload needs rather than the whole node, use pod-level securityContext.sysctls (for the namespaced-safe ones) or a node-tuning operator like the Node Tuning Operator / TuneD so the setting travels with the node pool and survives replacement. That keeps the "why is this node different" surprise out of your on-call.
For the workloads that genuinely care about tail latency (packet processing, some databases, in-memory caches), the last mile is topology. On a multi-socket node, memory access across NUMA nodes costs real nanoseconds. Turning on the kubelet Topology Manager with single-numa-node policy keeps a pod's CPUs and memory on the same socket. Hugepages cut TLB misses for large memory footprints:
# reserve 1024 x 2MB hugepages
sysctl -w vm.nr_hugepages=1024
Then request them like any other resource in the pod spec. This is not free complexity, so we only reach for it after PSI and throttling metrics say the ordinary tuning was not enough. If you are already deep in node internals, our note on Linux performance tuning fundamentals covers the provisioning side we lean on here.
Start every investigation at PSI and the CFS throttling ratio, because those two numbers explain the majority of container performance complaints without any guessing. Fix throttling and memory limits before you touch a single network sysctl. Bake the node-level sysctls into the image with Ansible or a tuning operator so they are boring and reproducible, and save NUMA plus hugepages for the handful of pods that measurably need them. Tuning you cannot point at a metric is just cargo cult with root access.
