Graceful Node ShutDown is Not Working.

Problem

• Graceful NodeShutdown is not working for the PMK cluster nodes.
• Pods on the cluster node gets stuck in the Terminating state after unexpected shutdown even after graceful shutdown is configured.

Environment

• Platform9 Managed Kubernetes – All Versions
• Cluster Node OS: RHEL7, CentOS7
• Cluster Node systemd version earlier than systemd 251

Cause

• There is an upstream issue with the systemd where the systemd inhibitors are not honoured by the systemd.
• Due to this the cluster nodes never shutdown gracefully, instead they shutdown immediately as soon as the underlying system shutdown event has occurred.
• This further results in failure of pod/container graceful shutdown and the pods remain in the terminating state even after configuration of graceful node shutdown.

Resolution

• Upgrade the systemd version on the affected nodes to systemd 251 or beyond that.