How to Enable Graceful Node Shutdown?

Problem

  • How to Enable Graceful Node Shutdown in a PMK cluster?

Environment

  • Platform9 Managed Kubernetes - v5.2 Onward

  • Kubernetes - v1.20 and Above

Procedure

  • For enabling Graceful Node Shutdown feature, there are changes that needs to be made to the node's Kubelet configuration.

  • For a cluster with Kubernetes version 1.20, it is required to explicitly enable the GracefulNodeShutdown feature under the featureGates (as it is a alpha feature). For Kubernetes v1.21 onward, it is enabled by default (as it is a beta feature).

  • To enable the GracefulNodeShutdown under the featureGates, edit the required master/worker configmap . Along with the feature gate, there is a requirement to set values ShutdownGracePeriod & ShutdownGracePeriodCriticalPods (by default are set to zero).

Warning

In order for this to happen safely without any intervention from other services which keeps track of the status of the pf9-kubelet service, first stop the pf9-hostagent & pf9-nodeletd services on ALL the worker/master nodes depending on which configmap is being edited.

sudo systemctl stop pf9-{hostagent,nodeletd}

  • There are 2 different default-kubelet-config configmaps under the namespace kube-system, one for master nodes master-default-kubelet-config and other for worker nodes worker-default-kubelet-config.

  • Edit appropriate configmap according to the requirement. e.g. in this case we are changing the worker configmap.

# kubectl edit configmap worker-default-kubelet-config -n kube-system
configmap/worker-default-kubelet-config edited
# kubectl get configmap worker-default-kubelet-config -n kube-system -o yaml
apiVersion: v1
data:
kubelet: |
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
...
featureGates:
DynamicKubeletConfig: true
GracefulNodeShutdown: true
maxPods: 200
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s

  • Start the pf9-hostagent service on all the worker/master nodes where the service was stopped initially. This will eventually start the pf9-nodeletd service.

sudo systemctl start pf9-hostagent

  • Verify if the GracefulNodeShutdown feature has been enabled or not from the worker node post editing the configmap.

# less /var/log/pf9/kubelet/kubelet.INFO | grep "feature gates"
I1208 22:01:52.214565 22305 feature_gate.go:243] feature gates: &{map[]}
I1208 22:01:52.217698 22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true]}
I1208 22:01:52.217779 22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true]}
I1208 22:01:52.220247 22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.220355 22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.243430 22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.243550 22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
[root@worker01 ~]# systemd-inhibit --list
Who: kubelet (UID 0/root, PID 22305/kubelet)
What: shutdown
Why: Kubelet needs time to handle node shutdown
Mode: delay

1 inhibitors listed.

Additional Information

Info

Graceful Node Shutdown

How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster

How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster


© 2025 Platform9. All Rights Reserved

On This Page
How to Enable Graceful Node Shutdown?ProblemEnvironmentProcedureAdditional Information