Pod Deployment is Failing Due to Stale Mount Paths of Orphaned Pods On The Node

Problem

  • Pods are getting stuck in the ContianerCreating state post deployment and below mentioned error is recorded at the same time in kubelet logs.

E0422 18:03:55.666613 44260 kubelet_volumes.go:225] orphaned pod "ca7bcbd3-35f9-4497-a7f5-2e901728c3a3" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/ca7bcbd3-35f9-4497-a7f5-2e901728c3a3/volumes/kubernetes.io~portworx-volume/pvc-50a9d02c-57cd-4141-96d6-d6cdf0182625: operation not permitted : There were a total of 2 errors similar to this. Turn up verbosity to see them.

Environment

  • Platform9 Managed Kubernetes - All Versions

  • Kubelet

Answer

  • The logs explain that there is failure in executing the rmdir() kernel call on the stale mount paths left on the node by the orphaned pods.

  • This is a routine login in the kubelet.volume.go code to clean up the stale mount paths left on the node by orphaned pods.


  • Sometimes these volume paths have a left over file named vol_data.json and due to this the rmdir() kernel call on this mount path fails.

sudo ls -la /var/lib/kubelet/pods/1d4bfc07-3469-4eaa-992f-6d23c17f3aee/volumes/kubernetes.io~csi/pvc-13c81b28-4038-40d5-b6e8-4194e1d7be0e
total 12
drwxr-x--- 2 root root 4096 Oct 7 12:37 .
drwxr-x--- 3 root root 4096 Oct 2 19:37 ..
-rw-r--r-- 1 root root 270 Oct 7 12:37 vol_data.json


© 2025 Platform9. All Rights Reserved

On This Page
Pod Deployment is Failing Due to Stale Mount Paths of Orphaned Pods On The NodeProblemEnvironmentAnswer