Pods Stuck In the Terminating State Due to Volume Unmount Error.

Problem

  • Pods are getting stuck in Terminating state after the deletion with the below mentioned error.

E0413 09:01:03.159172 8845 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/projected/$pod_id-kube-api-access-kxdxg podName:$pod_id nodeName:}" failed. No retries permitted until 2022-04-13 09:01:03.659142773 -0500 CDT m=+783674.310558707 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"kube-api-access-kxdxg\" (UniqueName: \"kubernetes.io/projected/$pod_id-kube-api-access-kxdxg\") pod \"$pod_id\" (UID: \"$pod_id\") : unlinkat /var/lib/kubelet/pods/$pod_id/volumes/kubernetes.io~projected/kube-api-access-kxdxg: device or resource busy"

Environment

  • Platform9 Managed Kubernetes - All Versions

  • Operating System: RHEL or CentOS v7.4 Onwards

Answer

  • This has been a known issue with RHEL and CentOS systems and starting with RHEL7.4 kernel there is a new sysctl parameter available to overcome this behaviour .

  • This parameter is may_detach_mounts and its value is set to0 by default.

  • It can be enabled by executing the below mentioned command on the appropriate kubernetes cluster nodes.

# echo 1 > /proc/sys/fs/may_detach_mounts
# cat /proc/sys/fs/may_detach_mounts
1

  • The kubelet will retry to unmount the projected volumes after enabling may_detach_mounts on the node.

  • This can take a few minutes and once done the terminating pod should get deleted from the node.


© 2025 Platform9. All Rights Reserved

On This Page
Pods Stuck In the Terminating State Due to Volume Unmount Error.ProblemEnvironmentAnswer