Unable To Add Node Due To "Device or resource busy"

Problem

Unable to add a new worker node with partition /var/lib/kubelet . Log shows "rm: cannot remove '/var/lib/kubelet': Device or resource busy"

Environment

• Platform9 Managed Kubernetes - All Versions

Answer

• Adding a worker node with the partition /var/lib/kubelet already mounted on it and see the "Device or resource busy" log. This is expected as the kubelet data directory i.e /var/lib/kubelet is cleaned up to ensure no pod state are stored outside the etcd database, for consistency. This operation fails if the directory is already mounted.
• To resolve this issue, unmount the/var/lib/kubelet filesystem from the nodes and let pf9 installer take care of creating this path during the node onboarding.