How to Renew Percona and Minio Certificates

Problem

The Minio and Percona pods are seen in CrashLoopBackOff state with errors in the pod logs as:

{"log":"2024-10-02T09:04:27.612080Z 0 [ERROR] WSREP: handshake with remote endpoint ssl://10.20.3.13:4567 failed: asio.ssl:337047686: 'certificate verify failed' ( 337047686: 'error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed')\n","file":"/var/lib/mysql/mysqld-error.log"}
...
{"log":"2024-10-02T09:04:27.614816Z 0 [ERROR] WSREP: Provider/Node (gcomm://10.20.3.174,percona-db-pxc-db-pxc-1.percona-db-pxc-db-pxc) failed to establish connection with cluster (reason: 7)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"level":"FATAL","errKind":"","time":"2024-10-02T07:19:49.168830203Z","message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","error":{"message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","source":["cmd/server-main.go:169:cmd.serverHandleCmdArgs()"]}}

Environment

  • Platform9 Edge Cloud - v5.6 [LTS2].

Procedure

NOTE

Percona certs will get auto-renewed starting SMCP 5.11.

For previous versions, follow the steps mentioned below:

The steps to be followed for the Percona certificate rotation are:

  1. Retrieve the list of secrets in the percona namespace and verify the validity of the certificates.

$ kubectl get secrets -n percona

$ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

  1. Delete the Percona DB secret. (Note: No new secrets will be created at this step)

$ kubectl delete secrets percona-db-pxc-db-ssl -n percona
$ kubectl delete secrets percona-db-pxc-db-ssl-internal -n percona

  1. Get the list of Helm Deployments and make a note of the chart name for percona-db.

$ helm ls -n percona

  1. Perform the helm upgrade for Percona DB.

$ helm upgrade --install percona-db /opt/pf9/airctl/conf/helm_charts/pxc-db-1.11.6.tgz --namespace percona --reuse-values

  1. The deleted secrets will be recreated, and the certificates will also be renewed. Please verify this.

$ kubectl get secrets -n percona

$ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

The steps to be followed for the Minio certificate rotation are:

  1. Check the list of secrets in the Minio namespace and verify the certificate validity.

$ kubectl get secrets -n minio

$ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

  1. Restart the nodeletd stack in each master node one at a time within the Management cluster.

Info

The stack restart is mandatory as this will refresh the CA which signs the Minio certificates.

# systemctl stop pf9-nodeletd
# /opt/pf9/nodelet/nodeletd phases stop
# /opt/pf9/nodelet/nodeletd phases start
# systemctl start pf9-nodeletd

  1. Disable _requestAutoCert_in the minio tenant. (set requestAutoCert to false): $plugin[3qq4mela3y2]

$ kubectl -n minio edit tenant minio1
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
…
spec:
…
requestAutoCert: false <---------

  1. List the minio Secrets:

$ kubectl get secrets -n minio

  1. Delete the existing minio certs:

$ kubectl -n minio delete secret minio1-tls
$ kubectl -n minio delete secret operator-tls

  1. List the CSRs:

$ kubectl get csr -n minio

  1. Delete the minio CSR:

$ kubectl -n minio delete csr minio1-minio-csr

  1. Enable requestAuthCert in the minio tenant. (set requestAutoCert to true)

$ kubectl -n minio edit tenant minio1
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
…
spec:
…
requestAutoCert: true <---------

  1. The minio-tls and operator secret will be regenerated, which the minio pods use. Also check certificate validity.

$ kubectl -n minio get secrets

$ kubectl -n minio get csr

$ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

$ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates

Additional Information

Platform9 team has filed an internal jira AIR-1401 to track Minio cert issue, and jira AIR-1402 to track the Percona cert rotation issue. To know more details on the ETA for the fix and the release version, please reach out to support team mentioning the jira-ID.

© 2025 Platform9. All Rights Reserved

On This Page
How to Renew Percona and Minio CertificatesProblemEnvironmentProcedureThe steps to be followed for the Minio certificate rotation are:Additional Information