Pod Stuck in Pending State As Container Fails to Mount Persistent NFS Volumes

Problem

A pod is stuck in a pending state as the container fails to mount the NFS volume. We observe the below error while running kubectl describe command for the affected pod.

$ kubectl describe pod <pod-name>
...
Events: Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 26m (x15 over 60m) kubelet, XX.XX.XX.XX Unable to attach or mount volumes: unmounted volumes=[volume-name], unattached volumes=[volume-name [name]-svc-token-f7227]: timed out waiting for the condition Warning FailedMount 111s (x40 over 60m) kubelet, XX.XX.XX.XX (combined from similar events): MountVolume.SetUp failed for volume "pvc-uuid" : mount failed: exit status 32Mounting command: systemd-runMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/[id]/volumes/kubernetes.io~nfs/pvc-[id] --scope -- mount -t nfs XX.XX.XX.XX:/path/to/nfs/storage /var/lib/kubelet/pods/[id]/volumes/kubernetes.io~nfs/pvc-[id]Output: Running scope as unit: run-[id].scopemount: /var/lib/kubelet/pods/[id]/volumes/kubernetes.io~nfs/pvc-[id]: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.[type] helper program

Environment

  • Platform9 Managed Kubernetes - All Versions

  • Persistent Volumes

  • NFS

Cause

The package nfs-common is not available on the node which is required to mount the Persistent Volume shared using NFS.

Resolution

Install the package nfs-common on the node on which the affected pod is running. Once the package is installed the pod will be able to mount the volume on the next retry.