Kubernetes Resource Limits Must Be Greater Than Requests: Here's Why

2:33 AM. This one is not actually an incident, it is a CI job screaming. A developer pushed a change to a Helm chart and the deploy pipeline has gone red on the kubectl apply step. The error message, if you squint at the pipeline logs, is short and angry. I have seen it enough times to know what it is before I even scroll. Limit less than request. The API server refused the pod at submission time, which is actually the friendliest failure mode Kubernetes has, because the cluster state never gets dirty. The bad manifest never becomes a live object.

Most of the complaints I get about Kubernetes error messages are fair. This one, honestly, is not. This one the API server gets right.

The scenario

description.md, issue.yaml, fix.yaml, resource_mismatch.sh. The issue manifest sets CPU request to 500 millicores and CPU limit to 100 millicores. That is not a typo, that is the whole point of the scenario.

Reproduce the issue

Unlike everything else in this series so far, there is no Pending pod to stare at. No describe to read. No events to parse. The API server rejected the manifest and nothing was created.

Confirm it does not exist. That is the signature of a validation failure: no object, not even a bad one.

Debug the hard way

The "debug" here is literally reading the error message out loud. spec.containers[0].resources.requests is the offending path, 500m is the offending value, and the rule is must be less than or equal to cpu limit. Three pieces of information in one line. Look at the manifest:

Request 500m, limit 100m. The request is the floor the scheduler guarantees, the limit is the ceiling the kubelet enforces. A floor above a ceiling is nonsense. The API server runs this check during admission and refuses.

If you want to see the rule in the API docs, the field is documented on the ResourceRequirements type. The admission controller that enforces it is called LimitRanger when a LimitRange is in play, but even without one, the core API server still runs the floor-below-ceiling check for you.

Why this happens

Requests and limits are two different things and most people conflate them. The request is what the scheduler uses to decide where a pod fits. It is a reservation. The kubelet uses it to set the cgroup's cpu.shares and the container's memory guarantee. The limit is the cap that the kernel enforces. For CPU it is a throttle, for memory it is a hard ceiling that, when crossed, gets you OOMKilled. Tomorrow's post is about that exact death.

When the limit is smaller than the request, the scheduler and the kernel would be operating on contradictory instructions. You would be telling the scheduler "reserve 500 millicores for me" and telling the kernel "never give me more than 100 millicores." The scheduler would find a node with 500m free, place the pod, and then the kernel would immediately throttle it down to 100. Kubernetes refuses to let you set up that trap.

The reason it is a friendly failure is that the rejection happens at kubectl apply, before any scheduling, before any image pull, before any pod ever exists. Your CI pipeline dies cleanly. There is no half-running workload to clean up.

The fix

The diff that matters:

Setting request and limit to the same value is the Guaranteed QoS class, which is the right default for most serious workloads. If you want burstable behaviour, set the limit higher than the request, never lower.

The lesson

1. When kubectl apply fails and no object is created, read the error message literally. It is almost always a validation rule and it will tell you the field.
2. Requests are a floor, limits are a ceiling. The floor can never be above the ceiling.
3. Setting request equal to limit gives you the Guaranteed QoS class, which is the safest default for serious workloads.

Day 13 of 35 — tomorrow, the Linux kernel kills a container in cold blood, and we read the OOM score to find out why.