How to Monitor Vault Token and Expiry Dates of Various certs?

Problem

How to check the status of vault token pro-actively

Environment

  • Platform9 Edge Cloud - v5.3 and Higher

Answer

Scenario-1: How to check the vault token status specific to a cluster?

Run the below commands from the DU Vm as the root user:

  1. Fetch the vault token:

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")
  1. Export the vault address:

export VAULT_ADDR=http://127.0.0.1:8200
  1. Export the cluster UUID:

mysql qbert -e “select name,id,uuid,status,lastOk,lastOp,taskStatus,kubeRoleVersion from clusters;”

CLUSTER_UUID=<cluster-UUID>
  1. Export the cluster vault token:

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")
  1. Execute the vault token lookup

/usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN

Example output:

# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

# export VAULT_ADDR=http://127.0.0.1:8200

# mysql qbert -e "select name,uuid from clusters;"
+--------------+--------------------------------------+
| name | uuid |
+--------------+--------------------------------------+
| test-cluster | [CLUSTER UUID] |
+--------------+--------------------------------------+
# CLUSTER_UUID=<CLUSTER UUID>

# CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")


# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Key Value
--- -----

Scenario-2: Monitoring Sunpike CA and Certs

Run the below commands from the DU Vm as the root user:

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

export VAULT_ADDR=http://127.0.0.1:8200

/usr/local/bin/vault read pki/cert/ca -format=json | jq -r '.data.certificate'| openssl x509 -noout -dates

usr/local/bin/vault read pki/roles/sunpike | grep ttl

If the Sunpike CA is expiring, please follow the manual steps mentioned in the following article to regenerate it: https://platform9.com/kb/PEC/refresh-sunpike-ca

To check the sunpike-kube-apiserver certificate expiry:

docker exec -it <sunpike-kube-apiserver-container-ID> sh -c "openssl s_client -connect localhost -port 6443 -showcerts 2>/dev/null </dev/null | openssl x509 -noout -dates"

Scenario-3: Monitoring Cluster CA and Certs

Run the below commands from the DU Vm as the root user:

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

export VAULT_ADDR=http://127.0.0.1:8200

export PATH=$PATH:/usr/local/bin

secrets_list=$(vault secrets list | awk '$2 == "pki" {print $1}' | grep "pmk-ca")
echo "$secrets_list" | while read secret; do echo -n "Certificate for Cluster: "; echo -n "$secret" | awk -F 'pmk-ca-' '{print $2}'; vault read "${secret}cert/ca" -format=json | jq -r '.data.certificate'| openssl x509 -noout -dates; done

usr/local/bin/vault read pki/roles/sunpike | grep ttl

This outputs the date on which the CA is created and the date on which it expires, per cluster. If the certs are expiring, please follow the manual steps mentioned in the following article to regenerate it: https://platform9.com/kb/PEC/refresh-cluster-ca

To verify the cluster certificate details from the node:

openssl x509 -text -noout -in /etc/pf9/kube.d/certs/etcd/client/ca.crt | grep -A2 -i validity
Or
openssl x509 -text -noout -in /tmp/authbs-certs.[LATEST-FILE-ID]/etcd/client/ca.crt | grep -A2 -i validity

Scanario-4 : Vault token renewal Steps:

Follow the steps mentioned in the KB Renew expired vault token from the DU VM.