Kubernetes Disk I/O Errors: Pod Symptoms, Node Root Cause

4:02 AM. An ingest service was dying on exactly one node out of twenty-three. The pod spec was identical everywhere. The image was identical. The config was identical. Same Helm release, same values. On twenty-two nodes it ran fine. On worker-14 it crashed within three seconds every single time, with a cryptic no such file or directory coming out of a path that existed on every other host. Twenty minutes of reading Go stack traces and diffing node labels got me nowhere. The moment I ran kubectl describe pod and looked at the volume block, the pattern snapped into place: we had a hostPath volume pointing at a directory that had been wiped off worker-14 during a disk replacement two days earlier and never restored.

The scenario

The repo reproduces this cleanly with a hostPath that points at a path guaranteed not to exist.

issue.yaml mounts hostPath: /nonexistent-path with type: Directory. The kubelet will try to stat that directory on the node and fail.

Reproduce the issue

Stuck in ContainerCreating. No restarts, no crashloop, just waiting. That is the first tell: a hostPath problem does not crash the container, it prevents the container from ever starting.

Debug the hard way

The error names the path, names the check that failed, and tells you exactly which node component emitted it. The kubelet did a type: Directory validation, the directory did not exist, and the mount step aborted.

Now you know which node to look at. If you can, SSH there:

Confirmed. The pod-level error was a faithful report of a node-level reality. Nothing was wrong with the container, the image, or the cluster control plane.

Why this happens

Any pod that mounts node-local storage (hostPath, local PV, some CSI drivers with pre-provisioned volumes) couples the pod's health to the node's disk state. Kubernetes does not replicate hostPath directories between nodes. It does not create them on demand. If you write path: /data/logs in your spec, that directory has to exist on every node where the pod might land, or the mount fails, or you use a nodeSelector to pin the pod to a node where it does exist.

The reason these bugs feel mysterious is that kubectl presents everything from the pod's point of view, but the root cause lives one layer down on the node. The pod looks broken when actually the filesystem under it is missing. Storage hardware failures, disk replacements, rebuilds, and manual cleanups all leave nodes in slightly different states over time, and any workload that assumes filesystem uniformity across nodes is one bad reboot away from a FailedMount.

The lesson I took from the worker-14 outage: if your pod events mention mounts and your pod works on some nodes but not others, stop reading application logs. Walk to the node.

The fix

The repo's fix swaps the hostPath for an emptyDir, which removes the dependency on any specific node filesystem:

For a real workload that needs persistence, replace hostPath with a proper PVC backed by a CSI driver. HostPath is fine for system daemons and debugging. For application state, it is a trap.

The lesson

1. Mount errors live in pod events, not pod logs. kubectl describe pod is the first command, not the fourth.
2. HostPath and local-PV failures are node-local. Always resolve the pod's nodeName before you start guessing.
3. If you need persistence, use a PVC backed by a real CSI driver. HostPath is a debugging tool, not a storage strategy.

Day 20 of 35. Tomorrow, the eviction that hits your pod because a completely different pod filled the node's disk.