How to Renew Percona and Minio Certificates

Problem

The Minio and Percona pods are seen in CrashLoopBackOff state with errors in the pod logs as:

Percona pod logs
    
 
{"log":"2024-10-02T09:04:27.612080Z 0 [ERROR] WSREP: handshake with remote endpoint ssl://10.20.3.13:4567 failed: asio.ssl:337047686: 'certificate verify failed' ( 337047686: 'error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed')\n","file":"/var/lib/mysql/mysqld-error.log"}...{"log":"2024-10-02T09:04:27.614816Z 0 [ERROR] WSREP: Provider/Node (gcomm://10.20.3.174,percona-db-pxc-db-pxc-1.percona-db-pxc-db-pxc) failed to establish connection with cluster (reason: 7)\n","file":"/var/lib/mysql/mysqld-error.log"}
Copy

Minio pod logs
    
 
{"level":"FATAL","errKind":"","time":"2024-10-02T07:19:49.168830203Z","message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","error":{"message":"Get \"https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/minio1?key=MINIO_ARGS\": x509: certificate has expired or is not yet valid: current time 2024-10-02T07:19:49Z is after 2024-10-01T07:16:11Z","source":["cmd/server-main.go:169:cmd.serverHandleCmdArgs()"]}}
Copy

Procedure

Percona certs will get auto-renewed starting SMCP 5.11.

For previous versions, follow the steps mentioned below:

The steps to be followed for the Percona certificate rotation are:

Retrieve the list of secrets in the percona namespace and verify the validity of the certificates.

Management cluster
    
​x
 
$ kubectl get secrets -n percona $ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates $ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
Copy

Delete the Percona DB secret. (Note: No new secrets will be created at this step)

Management cluster
    
xxxxxxxxxx
 
$ kubectl delete secrets percona-db-pxc-db-ssl -n percona$ kubectl delete secrets percona-db-pxc-db-ssl-internal -n percona
Copy

Get the list of Helm Deployments and make a note of the chart name for percona-db.

Management cluster
    
xxxxxxxxxx
 
$ helm ls -n percona
Copy

Perform the helm upgrade for Percona DB.

Management cluster
    
 
$ helm upgrade --install percona-db /opt/pf9/airctl/conf/helm_charts/pxc-db-1.11.6.tgz --namespace percona --reuse-values
Copy

The deleted secrets will be recreated, and the certificates will also be renewed. Please verify this.

Management cluster
    
 
$ kubectl get secrets -n percona $ kubectl get secret/percona-db-pxc-db-ssl -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates $ kubectl get secret/percona-db-pxc-db-ssl-internal -n percona -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
Copy

The steps to be followed for the Minio certificate rotation are:

Check the list of secrets in the Minio namespace and verify the certificate validity.

Management cluster
    
 
$ kubectl get secrets -n minio $ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates $ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
Copy

Restart the nodeletd stack in each master node one at a time within the Management cluster.

The stack restart is mandatory as this will refresh the CA which signs the Minio certificates.

Each master node in Mgmt cluster
    
xxxxxxxxxx
 
# systemctl stop pf9-nodeletd # /opt/pf9/nodelet/nodeletd phases stop # /opt/pf9/nodelet/nodeletd phases start # systemctl start pf9-nodeletd
Copy

Disable _requestAutoCert_in the minio tenant. (set requestAutoCert to false):

Management cluster
    
xxxxxxxxxx
 
$ kubectl -n minio edit tenant minio1apiVersion: minio.min.io/v2kind: Tenantmetadata:…spec:…requestAutoCert: false           <---------
Copy

List the minio Secrets:

Management cluster
    
xxxxxxxxxx
 
$ kubectl get secrets -n minio
Copy

Delete the existing minio certs:

Management cluster
    
xxxxxxxxxx
 
$ kubectl -n minio delete secret minio1-tls$ kubectl -n minio delete secret operator-tls
Copy

List the CSRs:

Management cluster
    
xxxxxxxxxx
 
$ kubectl get csr -n minio
Copy

Delete the minio CSR:

Management cluster
    
xxxxxxxxxx
 
$ kubectl -n minio delete csr minio1-minio-csr
Copy

Enable requestAuthCert in the minio tenant. (set requestAutoCert to true)

Management cluster
    
xxxxxxxxxx
 
$ kubectl -n minio edit tenant minio1apiVersion: minio.min.io/v2kind: Tenantmetadata:…spec:…requestAutoCert: true           <---------
Copy

The minio-tls and operator secret will be regenerated, which the minio pods use. Also check certificate validity.

Management cluster
    
 
$ kubectl -n minio get secrets $ kubectl -n minio get csr $ kubectl get secret/operator-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates $ kubectl get secret/minio1-tls -n minio -o jsonpath='{.data.public\.crt}' | base64 --decode | openssl x509 -inform pem -noout -dates
Copy

Platform9 team has filed an internal jira AIR-1401 to track Minio cert issue, and jira AIR-1402 to track the Percona cert rotation issue. To know more details on the ETA for the fix and the release version, please reach out to support team mentioning the jira-ID.

Last updated on

Was this page helpful?

How to Renew Percona and Minio Certificates

Problem

Environment

Procedure

The steps to be followed for the Minio certificate rotation are:

Additional Information