After Cluster Upgrade from 1.20 to 1.21, The Worker Nodes Are Losing Custom Labels.

Problem

Worker nodes losing custom labels after the cluster upgrade from 1.20 to 1.21.

Environment

• Platform9 Managed Kubernetes - v5.5.
• Kubernetes version: 1.20.

Answer

• The custom labels are missing post the etcd corruption which happens due to missing etcd environment variables in the sunpike host objects and in the QbertDB nodes table associated with the master nodes.
• This is a known issue, Platform9 Engineering team is currently working to resolve this issue. The upcoming PMK-5.10 release will provide the solution for this issue.

Workaround

The custom labels needs to be manually added to all the worker nodes as a workaround until the fix is available in PMK-5.10 release.

Additional Information

An Internal bug PMK-5829 has been filed to track this issue. Please open a support ticket mentioning the JiraID- PMK-5829 to get more information about the progress.