Kubernetes Master Node in NotReady State With Message "cni plugin not initialized"

Problem

  • A Kubernetes master node is showing as NotReady and the describe output for the node is showing "cni not initialized".

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 NotReady master 34d v1.21.3
master2 Ready master 34d v1.21.3
master3 Ready master 34d v1.21.3
  • A description of the node similarly reports KubeletNotReady due to the CNI plugin being uninitialized.

$ kubectl describe node master1
.....
Conditions:
Ready False Mon, 13 Jun 2022 21:42:05 +0000 Mon, 13 Jun 2022 21:32:01 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Environment

  • Platform9 Managed Kubernetes - v5.4 and Higher

  • Kubernetes - All v1.21

  • Runtime - Containerd

  • Container Network Interface - Calico

Cause

Due to an bug in the Platform9 Managed Kubernetes Stack the CNI config is not reloaded when a partial restart of the stack takes place.

Resolution

This issue is now fixed on the pf9-kube- 1.23.8-pmk.218 and above releases.

Workaround

  1. Verify that the CNI configuration directory referenced by containerd is not empty on the affected node.

For Calico based clusters the directory should contain the following files:

# ls -l /etc/cni/net.d/
total 8
-rw-r--r-- 1 root root 730 Jun 16 14:16 10-calico.conflist
-rw------- 1 root root 3094 Jun 16 14:16 calico-kubeconfig
  1. If the files mentioned above are missing, restart calico-node pod on the affected node.

# kubectl get pods -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-node-8brm4 1/1 Running 0 3d3h 10.128.142.117 master1 <none> <none>
calico-node-xwgn4 1/1 Running 0 3d3h 10.128.141.109 master2 <none> <none>
calico-node-bgmi3 1/1 Running 0 3d3h 10.128.141.109 master3 <none> <none>


# kubectl delete pod calico-node-8brm4 -n kube-system
Success

The Kubernetes nodes should now all show as Ready .

  1. Verify the status of the node after restarting calico-node pod.

# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 34d v1.21.3
master2 Ready master 34d v1.21.3
master3 Ready master 34d v1.21.3