Pod Stuck in Terminating State Due to Inability to Clean Volume subPath Mount

Problem

A pod is stuck in a terminating state as the configmap mounted as a volume fails to unmount when trying to clean the subPath mount for it.

  • Pod Status

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE podname 0/1 Terminating 0 3h3m 10.200.42.191 10.3.9.109 [none]

  • Pod Description

Name: podnameVolumes:config:Type: ConfigMap (a volume populated by a ConfigMap)Name: podnameOptional: false

  • Kubelet Error

Error: "error cleaning subPath mounts for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b45800b2-1b3f-11ea-928b-005056bcb9b4-config\") pod \"b45800b2-1b3f-11ea-928b-005056bcb9b4\" (UID: \"b45800b2-1b3f-11ea-928b-005056bcb9b4\") : error processing /var/lib/kubelet/pods/b35800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem:error cleaning subpath mount /var/lib/kubelet/pods/b45800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7: remove /var/lib/kubelet/pods/b45800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7: device or resource busy"

Environment

  • Platform9 Managed Kubernetes - All Versions

Cause

In this case, the /var/lib/kubelet directory has a symlink associated to it.

$ ls -l /var/lib/kubelet lrwxrwxrwx. 1 root root XX XXX X YY:YY /var/lib/kubelet -> /mnt/docker-images/kubelet

Resolution

Temporary Quick Fix

  1. Manually unmount the volume.

# umount /var/lib/kubelet/pods/b35800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7

  1. Manually force delete the Pod.

$ kubectl delete pod --grace-period=0 --force [pod_name]

Long Term Solution

This involves making a change to the existing setup. On every host where the symlink persists, perform the following operations.

  1. Stop pf9-hostagent service.

$ systemctl stop pf9-hostagent

  1. Stop pf9-kube service.

$ systemctl stop pf9-kube

  1. Drop the symlink for /var/lib/kubelet.

$ rm -rf /var/lib/kubelet

  1. Start pf9-hostagent service.

$ systemctl start pf9-hostagent

Additional Information

Note: The bug Issue 65879 talks about setting the Kubelet flag --root-dir to the value referenced by symlink from /var/lib/kubelet to resolve the issue. If the --root-dir value has to be set, it would have to be hard coded as --root-dir isn't an option available for dynamic Kubelet config.

© 2025 Platform9. All Rights Reserved

On This Page
Pod Stuck in Terminating State Due to Inability to Clean Volume subPath MountProblemEnvironmentCauseResolutionAdditional Information