Restore Certificates Manually for On-Prem Deployments [Internal Only]

Problem

On an On-Prem Platform9 Managed Kubernetes deployment, the management plane (DU) may be unavailable at times and a cluster node may need to be rebooted or have the PMK stack restarted on it. The reboot fails at the generating certificates step if the management plane is unavailable.

Environment

  • Platform9 Edge Cloud - v5.3 LTS Patch 10 and Lower

Procedure

Pre-conditions:

  1. The management plane VM is shut down or offline.

  2. The cluster node had been initializing while the above conditions were true and thus has failed to start the PMK stack (nodeletd phases) at the gen_certs step.

Steps:

  1. On the cluster node, stop the pf9-hostagent and pf9-nodeletd services.

# systemctl stop pf9-hostagent pf9-nodeletd

  1. Stop the PMK stack on the cluster node.

# /opt/pf9/nodelet/nodeletd phases stop

  1. Restore the certificates from backup by using the script provided below (in Additional Information).

  2. Start nodeletd phases on the cluster node.

# /opt/pf9/nodelet/nodeletd phases start

  1. Start pf9-hostagent service on the cluster node.

# systemctl start pf9-hostagent

Additional Information

Refer Zendesk Ticket: 1352090

Restore Certificate Script:

#!/usr/bin/env bash
CONFIG_DIR=${CONFIG_DIR:-/etc/pf9}
source $CONFIG_DIR/kube.env
if [ -e "$CONFIG_DIR/kube_override.env" ]; then
source $CONFIG_DIR/kube_override.env
fi
KUBE_DIR=$CONFIG_DIR/kube.d
cert_dir=$KUBE_DIR/certs
nodelet_running=`systemctl is-active pf9-nodeletd`
if [ "$nodelet_running" == "active" ]; then
echo "pf9-nodeletd is running. Please stop the service before running this script."
echo "systemctl stop pf9-hostagent pf9-nodeletd"
exit 1
fi
if [ -d $cert_dir ]; then
echo "Actual cert dir: $cert_dir already present. Exiting!"
exit 1
fiv
cert_dir_name=`ls -ltr $KUBE_DIR | grep 'certs.teardown' | head -1 | awk -F ' ' '{print $NF}' | xargs`
if [ "x$cert_dir_name" == "x" ]; then
echo "No certificate backup directories found"

Note: Existing certificates are backed up on the worker node automatically when the PMK stack is stopped.

Info

Starting 5.3 LTS Patch 11, the ability to skip gen_certs phase when the management plane is offline and the PMK stack is restarted on a node using nodelet phases is added. This resolves the issue when the management plane is offline and the PMK stack is restarted explicitly using nodeletd phases restart, where in on stop action previously the phase gen_certs was called which removed the certificates, and on start action, it failed to fetch them as the management plane was unavailable.

Note - Also, after a node reboot, the pf9-nodeletd service will skip running the stop and start functions of the gen_certs phase to avoid reaching out to the management plane.


© 2025 Platform9. All Rights Reserved

On This Page
Restore Certificates Manually for On-Prem Deployments [Internal Only]ProblemEnvironmentProcedureAdditional Information