AddOn Operator Pod Restarting due to Error "Use SANs or Temporarily Enable Common Name Matching with GODEBUG=x509ignoreCN=0"

Problem

AddOn Operator Pod Restarting due to Error "Use SANs or Temporarily Enable Common Name Matching with GODEBUG=x509ignoreCN=0"

Environment

  • Platform9 Edge Cloud - 5.3 LTS Patch #6 and Below

  • AddOn Operator

Cause

AddOn operator certificate generation is using CN instead of SAN.

$ kubectl logs pf9-addon-operator-5f5cd7649b-5dgvz -n pf9-addons

{"level":"error","msg":"Error in healthcheck: Error listing ClusterAddon objects from sunpike","time":"2022-07-07T07:38:49Z"} {"level":"error","msg":"List ClusterAddons error count: 5 of 10","time":"2022-07-07T07:38:49Z"} {"level":"error","msg":"Get \"https://airctl-pximsp02.pf9.localnet/qbert/v4/a1c4d5887ce34c81a2c8696bd9d67171/sunpike/apis/sunpike.platform9.com/v1alpha2/namespaces/default/clusteraddons?labelSelector=sunpike.pf9.io%2Fcluster%3Db4877409-fe75-421a-890f-4faa49b5434d\": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0Unable to list ClusterAddons","time":"2022-07-07T07:40:19Z"}

{"level":"error","msg":"Unable to get ClusterAddon objects for cluster: b4877409-fe75-421a-890f-4faa49b5434d Error listing ClusterAddon objects from sunpike","time":"2022-07-07T07:40:19Z"}

Resolution

  1. Scale the pf9-addon-operator deployment replica to 0.

kubectl scale deployment/pf9-addon-operator --replicas=0 -n pf9-addons
  1. Edit the pf9-addon-operator deployment and make the below changes in the pf9-addon-operator deployment.

env:
- name: GODEBUG
value: "x509ignoreCN=0"

Example:

  • Edit the pf9-addon-operator deployment:

# kubectl edit deployment/pf9-addon-operator -n pf9-addons
  • After modification, the changed spec content in spec.template should look like

spec:
containers:
- env:
- name: GODEBUG
value: x509ignoreCN=0
- name: LOGLEVEL
value: INFO
...
image: localhost:5100/platform9/pf9-addon-operator:3.2.3
  1. On all the master nodes part of the cluster, edit file /opt/pf9/pf9-kube/conf/addons/pf9-addon-operator/pf9-addon-operator-deployment.yaml and add the changes made similar to the deployment object above to this file. These changes made to pf9-addon-operator-deployment.yaml file will ensure that the env vars are persisted when the stack restarts.

    • Edit thepf9-addon-operator-deployment.yaml file.

$ sudo vi /opt/pf9/pf9-kube/conf/addons/pf9-addon-operator/pf9-addon-operator-deployment.yaml

- After modification, the changes should look like:

containers:
- name: pf9-addon-operator
...
env:
- name: GODEBUG
value: x509ignoreCN=0
- name: LOGLEVEL
value: "INFO"
  1. Scale the pf9-addon-operator deployment replica back to 1.

kubectl scale deployment/pf9-addon-operator --replicas=1 -n pf9-addons
  1. Check the status of the new pf9-addon-operator pod replica.

Additional Information

The issue is fixed starting 5.3 LTS Patch #10 & Patch #12 onwards.