Worker Node Intermittently Entering Into "Ready, SchedulingDisabled" State

Problem

One or more Kubernetes nodes are intermittently transitioning into a "Ready, SchedulingDisabled" state, where Kubelet is still heartbeating, but, workloads are no longer able to be scheduled to the node.

Environment

• Platform9 Managed Kubernetes - All Versions
• Kubelet

Cause

The node(s) which are impacted are likely facing resource contention/exhaustion - CPU and/or Memory – which prompts scheduling to be disabled for the node by kube-controller-manager, based on the node metrics reported by Kubelet.

Resolution

1. Ensure that appropriate pod limits are set.
2. Ensure that the pod limits set are not excessive, i.e. set too low, which would cause continual pod restarts/evictions, and drive up the resource usage of the node all the same.

Additional Information

• Kubernetes Documentation – Managing Resources for Containers