AddOn Operator Pod Restarting due to Error "Use SANs or Temporarily Enable Common Name Matching with GODEBUG=x509ignoreCN=0"

Problem

AddOn Operator Pod Restarting due to Error "Use SANs or Temporarily Enable Common Name Matching with GODEBUG=x509ignoreCN=0"

Environment

• Platform9 Edge Cloud - 5.3 LTS Patch #6 and Below
• AddOn Operator

Cause

AddOn operator certificate generation is using CN instead of SAN.

$ kubectl logs pf9-addon-operator-5f5cd7649b-5dgvz -n pf9-addons

{"level":"error","msg":"Error in healthcheck: Error listing ClusterAddon objects from sunpike","time":"2022-07-07T07:38:49Z"} {"level":"error","msg":"List ClusterAddons error count: 5 of 10","time":"2022-07-07T07:38:49Z"} {"level":"error","msg":"Get \"https://airctl-pximsp02.pf9.localnet/qbert/v4/a1c4d5887ce34c81a2c8696bd9d67171/sunpike/apis/sunpike.platform9.com/v1alpha2/namespaces/default/clusteraddons?labelSelector=sunpike.pf9.io%2Fcluster%3Db4877409-fe75-421a-890f-4faa49b5434d\": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0Unable to list ClusterAddons","time":"2022-07-07T07:40:19Z"}

{"level":"error","msg":"Unable to get ClusterAddon objects for cluster: b4877409-fe75-421a-890f-4faa49b5434d Error listing ClusterAddon objects from sunpike","time":"2022-07-07T07:40:19Z"}

Resolution

1. Scale the pf9-addon-operator deployment replica to 0.

xxxxxxxxxx

kubectl scale deployment/pf9-addon-operator --replicas=0 -n pf9-addons

2. Edit the pf9-addon-operator deployment and make the below changes in the pf9-addon-operator deployment.

xxxxxxxxxx

env:

  - name: GODEBUG

    value: "x509ignoreCN=0"

Example:

• Edit the pf9-addon-operator deployment:

xxxxxxxxxx

# kubectl edit deployment/pf9-addon-operator -n pf9-addons

• After modification, the changed spec content in spec.template should look like

xxxxxxxxxx

spec:

  containers:

  - env:

    - name: GODEBUG

      value: x509ignoreCN=0

    - name: LOGLEVEL

      value: INFO

...

    image: localhost:5100/platform9/pf9-addon-operator:3.2.3

3. On all the master nodes part of the cluster, edit file /opt/pf9/pf9-kube/conf/addons/pf9-addon-operator/pf9-addon-operator-deployment.yaml and add the changes made similar to the deployment object above to this file. These changes made to pf9-addon-operator-deployment.yaml file will ensure that the env vars are persisted when the stack restarts.

   • Edit thepf9-addon-operator-deployment.yaml file.

xxxxxxxxxx

$ sudo vi /opt/pf9/pf9-kube/conf/addons/pf9-addon-operator/pf9-addon-operator-deployment.yaml

- After modification, the changes should look like:

xxxxxxxxxx

containers:

- name: pf9-addon-operator

...

  env:

  - name: GODEBUG

    value: x509ignoreCN=0

  - name: LOGLEVEL

    value: "INFO"

4. Scale the pf9-addon-operator deployment replica back to 1.

xxxxxxxxxx

kubectl scale deployment/pf9-addon-operator --replicas=1 -n pf9-addons

5. Check the status of the new pf9-addon-operator pod replica.

Additional Information

The issue is fixed starting 5.3 LTS Patch #10 & Patch #12 onwards.