Kubernetes cgroup OOM: When the Kernel Kills Before kubectl Knows

3:40 AM, a Tuesday I will not forget. A batch processor was dying every four minutes in production. kubectl top pod reported 38Mi average. The alert from our APM said peak RSS was 42Mi. The limit on the pod was 50Mi. By every number I could see, we had headroom. But the pod kept dying with exit code 137. I spent twenty minutes reading Go profiles before I remembered that kubectl top samples every thirty seconds and the kernel checks every single page fault. Somewhere in the gap between those two clocks, the container was touching 60Mi for half a second and the kernel was doing what kernels do. I had been debugging the wrong layer the whole time.

The scenario

Same repo, different folder. This one exists to make the gap between pod-level metrics and cgroup-level enforcement painfully visible.

issue.yaml runs stress with 100M of allocations against a 50Mi cgroup ceiling. You cannot win that fight.

Reproduce the issue

The restart count climbs while the metrics dashboards stay flat. That is the cgroup loop. Fast, clean, invisible to anything that samples on a poll.

Debug the hard way

One second of life. Born, killed, done.

If you can SSH to the node, the evidence is sharper:

Anon-rss 51200kB against a 50Mi limit. The kernel saw the overshoot at the exact microsecond it happened.

Why this happens

Kubernetes does not enforce memory limits. The Linux kernel does, through cgroups v1 or v2, depending on your distro. When you write limits.memory: 50Mi in a pod spec, the kubelet translates that into a cgroup file on the host, something like /sys/fs/cgroup/memory/.../memory.limit_in_bytes. From that moment on, every page fault inside the container goes through a kernel check. If the total charged memory exceeds the limit by even one page, the OOM killer fires. There is no poll, no sample, no averaging. It is instantaneous.

kubectl top pod and your Prometheus dashboards work very differently. They scrape cAdvisor on a schedule, usually every 15 to 30 seconds, and they report whatever they saw at those moments. Short spikes between samples are invisible to them. A pod that touches 120Mi for 200 milliseconds looks identical to a pod that stays at 38Mi forever, because the sample never landed during the spike.

Once you see the two layers clearly, cgroup OOMs stop being a mystery. Your graphs are not lying, they are just looking away at the wrong moment. The kernel never looks away.

The fix

Same workload numbers as Day 15, different point. Here we are not just giving the app more headroom, we are giving it enough headroom that short allocation spikes during GC or buffer resize cannot punch through the cgroup ceiling.

The lesson

1. kubectl top samples, the kernel does not. If the two disagree, trust the kernel.
2. Memory limits are enforced at the page-fault layer. Spikes shorter than your scrape interval can still kill you.
3. When an OOM happens that your graphs cannot explain, walk down one layer to dmesg or cgroup stats. The evidence is always there, you just need to open the right file.

Day 16 of 35. Tomorrow we jump from compute to storage, starting with a volumeMount that points at a volume that does not exist.