RunContainerError in Kubernetes: Why Your Pod Never Starts

2:07 AM. A Helm chart update just rolled to production and every new pod is stuck at RunContainerError. My manager is typing in Slack. The image pulled fine, the node has capacity, the previous revision is still serving traffic but the HPA is about to scale it and I am one event away from a real outage. I run kubectl logs and get nothing because the container never produced a single byte of stdout. The status column says four words. No stack trace. No hint. Just a pod that refuses to exist, yaar, and a clock that is not on my side.

The scenario

This one comes from my open-source repo troubleshoot-kubernetes-like-a-pro. You are going to reproduce the exact failure on your own cluster so the next time you see RunContainerError at 2 AM, your muscle memory takes over.

You will see three files: description.md with the background, issue.yaml with the broken pod, and fix.yaml with the working version. This assumes you already have a cluster running from Day 0, cloud or local, it does not matter.

Reproduce the issue

Apply the broken manifest.

There it is. RunContainerError, two restarts, the pod is burning cycles and telling you absolutely nothing about why.

Debug the hard way

First instinct, logs.

Empty. The container never reached a running state so there is no stdout to read. Juniors retry this command three or four times hoping it changes. It never will.

Try describe, scroll past volumes and conditions and tolerations until you hit the events block.

Read that carefully. Pulled is Normal, so the image is fine. Failed is where the truth lives. The last line is the whole story: executable file not found in $PATH. The kubelet did its job, containerd did its job, runc tried to exec the command and the binary was not there.

Why this happens

When you set command: in a pod spec, you override the image's ENTRYPOINT completely. The kubelet hands the pod to the container runtime, containerd creates a task, and runc does an execve with the first element of your command: array. If that path does not resolve inside the container's filesystem, execve returns ENOENT and the container never becomes a running process. There is no stdout because there is no process. There are no application logs because there is no application.

The pod is not broken at the Kubernetes level. It is broken at the Linux level, one layer below where most people look.

The fix

Apply the fixed manifest.

The key change is in the command: array. The broken version was pointing at a binary that does not exist in the busybox image:

sh exists. It is in /bin/sh. runc finds it, execs it, the container enters the running state.

The easiest way — with Kubilitics

Open the Pods view. The failing pod is already badged with its status and the failure counter is front and centre.

Click the pod and jump to the Events tab. Kubilitics groups the kubelet events by severity and collapses the noise, so the Warning line about the missing binary sits at the top of the list instead of buried in a wall of Normal Pulled events.

The Overview tab shows the full pod spec with Annotations and Tolerations expanded. The bad command: array is right there in the rendered YAML, one click away from confirming the root cause without leaving the app.

After kubectl apply -f fix.yaml, the list refreshes in real time. The fixed pod shows green, the broken one is still badged red, and you can see both outcomes at a glance while deciding whether to delete the old pod.

The lesson

1. When a pod status is RunContainerError, stop reading logs. Read the Failed event message and look for the runc line.
2. command: overrides ENTRYPOINT. You own the full exec path. Test it inside the image before you ship the YAML.
3. Empty logs are a diagnostic signal, not a dead end. If there is no stdout, the container never ran, which means the problem is in the runtime, not the app.

Day 1 of 35 — tomorrow the container does start, several times a minute, and still will not stay alive.