2:30 AM and the data pipeline team was staring at a PVC that had been Pending for 58 minutes. The Pod using it was stuck in ContainerCreating with zero events for the last twenty of those minutes. The on-call had already described the pod, described the PVC, described the node, and found nothing useful. "It just sits there yaar." He was right, it just sat there. PVCs do that. They sit there silently because nothing in the cluster is obligated to tell you why a claim has not been satisfied. The provisioner might be missing, the storageClass might be a typo, the topology might be wrong, the capacity might not match. Five different failures, one identical symptom.
The scenario
The PVC is Pending. No PV matches the storage class.
The PVC requests storageClass: slow but no StorageClass named slow is defined and no PV carries that class. The PVC sits in Pending indefinitely. The Pod that mounts it stays in ContainerCreating — it cannot start until the PVC binds.
The Pod waits on a volume that never arrives
The Pod spec mounts data-claim. Until that PVC reaches Bound the kubelet cannot start the container. Run kubectl describe pod data-pod and look for a FailedMount event referencing the unbound claim.
The PVC binding controller finds no candidate
The controller matches PVCs to PVs by storageClassName, accessModes, and capacity. Both existing PVs carry class: standard. The PVC requests slow — no match, no dynamic provisioner, no binding. The PVC stays Pending forever.
The storage class simply does not exist
Run kubectl get storageclass. You will see only standard. There is no slow class registered, so no provisioner can respond to the claim. The first kubectl describe pvc event will say exactly this.
This is one of my favorite scenarios in the repo because it teaches you to look in the right place instead of guessing.
git clone https://github.com/vellankikoti/troubleshoot-kubernetes-like-a-pro.git
cd troubleshoot-kubernetes-like-a-pro/scenarios/persistent-volume-claim-issues
lsissue.yaml declares a PVC with storageClassName: non-existent-storage-class and a Pod that mounts it. Nothing in the cluster can satisfy that claim.
Reproduce the issue
kubectl apply -f issue.yaml
kubectl get pvc,podNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/pvc-issue Pending non-existent-storage-class 40s
NAME READY STATUS RESTARTS AGE
pod/pvc-issue-pod 0/1 ContainerCreating 0 40sTwo objects. Neither of them is going anywhere.
Debug the hard way
kubectl describe pvc pvc-issueStatus: Pending
StorageClass: non-existent-storage-class
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 38s persistentvolume-controller storageclass.storage.k8s.io "non-existent-storage-class" not foundThere it is. One line. The storage class does not exist. The controller cannot dynamically provision anything because it has no provisioner to call.
kubectl get storageclassNAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE AGE
standard (default) rancher.io/local-path Delete WaitForFirstConsumer 6dOnly standard exists. The PVC asked for something that was never registered.
kubectl get pv
# No resources foundNo static PV waiting around either. No provisioner, no static PV, no binding. Ever.
Why this happens
A PVC can fail to bind for five distinct reasons, and the events only tell you one at a time. First, the storageClassName references a class that does not exist, which is what we have here. Second, the class exists but its provisioner is not running, so the dynamic request never gets serviced. Third, the topology constraint cannot be satisfied because the only nodes in the right zone are cordoned or full. Fourth, the requested accessModes do not match anything the underlying storage supports, for example a ReadWriteMany request on a backend that only offers ReadWriteOnce. Fifth, the requested capacity is larger than any available PV and the class is not dynamic, so nothing can grow to fit.
The mental model I use is a three-step funnel. Does the storageClass exist. Can its provisioner respond. Do the constraints match something real. If any of those three fail, the PVC sits in Pending forever. There is no timeout, no failure event beyond the first one, no auto-fallback. Silent infinity.
The trap is that the first event is often the only event. If you miss it, describe will just show you the Pending status with no explanation, and you will start guessing.
The fix
The PVC is Bound. Pod transitions to Running.
A static PV is created with matching capacity, accessModes, and storageClass. The binding controller pairs the PVC to pv-001 via volumeName and claimRef. The Pod mounts the volume and starts.
Pod mounts the volume and starts
Once the PVC reaches Bound the kubelet on the scheduled node attaches the volume and starts the container. The Pod status.phase flips to Running — the FailedMount events stop appearing.
PVC records the bound volume name
The binding controller writes spec.volumeName: pv-001 into the PVC. From this point the claim is exclusively paired with that PV — no other claim can bind to it. Check with kubectl get pvc data-claim -o yaml.
The PV records the back-reference
The controller also writes spec.claimRef into pv-001 pointing at data-claim. The two objects reference each other symmetrically. Verify with kubectl get pv pv-001 -o yaml.
The repo's fix takes a different path. Instead of creating the missing storage class, it creates a matching static PV and a PVC with no storage class at all.
kubectl delete -f issue.yaml
kubectl apply -f fix.yamlKey diff:
kind: PersistentVolume
metadata:
name: valid-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /tmp/datakubectl get pvc pvc-fixed
# pvc-fixed Bound valid-pv 1Gi RWO 15sBound. The binder matched the claim against the static PV because capacity, accessModes, and storageClass (empty on both) all lined up.
The lesson
- A Pending PVC always has a first event that explains it. If you missed it, describe the PVC again and look at the top of the events list.
- There are exactly five reasons a PVC does not bind: missing class, dead provisioner, bad topology, wrong accessMode, or capacity mismatch. Walk the funnel.
- Static PVs still work when dynamic provisioning does not. They are ugly, but they unblock you in minutes.
Day 18 of 35. Tomorrow, the volume binds but the app still cannot write to it, and the reason is three decades old.