Kubelet logs UnmountVolume.NewUnmounter Failure logs On vol_data.json Is Deleted

Problem

Kubelet continuously logs below logs on failing to unmount a volume from a pod and filling up on disk usage.

Kubelet logs
    
 
Error reconciler.go:193] "operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled true) for volume \"filestore\" (UniqueName: \"kubernetes.io/csi/csi.trident.netapp.io^pvc-<PVC_UUID>\") pod \"<POD_ID>\" (UID: \"<POD_ID>\") : UnmountVolume.NewUnmounter failed for volume \"filestore\" (UniqueName: \"kubernetes.io/csi/csi.trident.netapp.io^pvc-<PVC_ID>\") pod \"<POD_ID>\" (UID: \"<POD_ID>\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json]: open /var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json: no such file or directory" err="UnmountVolume.NewUnmounter failed for volume \"filestore\" (UniqueName: \"kubernetes.io/csi/csi.trident.netapp.io^pvc-<PVC_ID>\") pod \"<POD_ID>\" (UID: \"<POD_ID>\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json]: open /var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json: no such file or directory"
Copy

Environment

Platform9 Managed Kubernetes - v5.3 and Higher

Cause

This is a known upstream bug https://github.com/kubernetes/kubernetes/issues/101911 and https://github.com/kubernetes/kubernetes/pull/102576

Workaround

Find the path of the pod mentioned in logs

Example
    
xxxxxxxxxx
 
# ll /var/lib/kubelet/pods/<POD_ID>/drwxr-x---. 3 root root 37 May 20 14:53 pluginsdrwxr-x---. 5 root root 87 May 20 14:53 volumes
Copy

Check if the above pod running in the cluster

Command
    
xxxxxxxxxx
 
# kubectl get pods  -o custom-columns=PodName:.metadata.name,PodUID:.metadata.uid -A | egrep "<POD_ID>"#
Copy

Remove the content of those pod directories and restart pf9-kubelet

Command
    
xxxxxxxxxx
 
# rmd -Rf  /var/lib/kubelet/pods/<POD_ID># systemctl restart pf9-kubelet
Copy

Last updated on

Was this page helpful?