How to Renew vault Token?

Problem

The vault token is expired.

Environment

  • Platform9 Managed Kubernetes v5.9.3

Validation

Steps to validate the token expiry:

  1. Exec into pf9-vault pod in Management Plane namespace.

$ kubectl exec -it -n <MANAGEMENT_PLANE_NAMESPACE> --kubeconfig <KUBECONFIG> $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash

  1. Export the required details.

# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

# export VAULT_ADDR=http://127.0.0.1:8200

# CLUSTER_UUID=<CLUSTER_UUID>

# OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

# ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

# CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

  1. Run the below command to know token expiry details:

# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN

Example:

SAMPLE:

# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Key Value
--- -----
accessor [ACCESSOR-ID]
creation_time [CREATION TIMESTAMP]
creation_ttl 26280h
display_name token
entity_id n/a
expire_time <>
explicit_max_ttl 0s
id [ID]
issue_time [ISSUE TIMESTAMP]
meta <nil>
num_uses 0
orphan false
path auth/token/create
policies [POLICIES]
renewable true
ttl 26215h49m50s
type service

Procedure

  1. Exec Into pf9-vault pod in customer namespace

export KUBECONFIG=kplane-enterprise-2.yaml
export NS=<customer_shortname>
kubectl exec -it -n $NS $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash

  1. Export required details (in pf9-vault pod)

CLUSTER_UUID=<cluster_UUID>
OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")
ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")
echo $OLD_VAULT_TOKEN
echo $ROOT_VAULT_TOKEN

  1. Generate New Token (in pf9-vault pod)

NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)
NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')
echo "New Vault-Token generated - $NEW_TOKEN"

  1. Update the new token in qbert Database (in pf9-vault pod)

mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"

  1. Open a new terminal window and Verify if the new token is updated at cluster and node level (Execute on a new terminal window)

export KUBECONFIG=kplane-enterprise-2.yaml
export NS=<customer_shortname>
kubectl -n $NS exec -it deploy/mysqld-exporter -- mysql qbert -e "select name,uuid,vaultToken from clusters where uuid=<cluster_uuid>"
kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts <HOST_UUID> -o yaml | grep -i vault

  1. If the token in Sunpike does not match the token in Qbert, run the following command to patch the Sunpike host object.(from the same terminal window as Step 5 )

export VAULT_TOKEN=<TOKEN_FROM_QBERT_DB>
export CLUSTER_UUID=<CLUSTER_UUID>
for i in $(kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts --no-headers | grep $CLUSTER_UUID | awk '{print $1}'); do kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl patch host $i -p '{"spec":{"pf9":{"vaultToken":"'${VAULT_TOKEN}'"}}}'; done

  1. restart the full stack restart on nodes that got stuck at the Cert Generation phase (if any).

  2. revoke the old Token (in pf9-vault pod) - once all nodes are working fine.

curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"token": "'$OLD_VAULT_TOKEN'"}' http://localhost:8200/v1/auth/token/revoke

Additional Information

To check if the vault token is expired follow these steps:

  1. From the problematic master node, perform this.

#grep -i vault /etc/pf9/kube.env

You will get an output like this:
export VAULT_TOKEN="<token_name>" <<--------

  1. Come to the DU VM and perform this.

# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")
# export VAULT_ADDR=http://127.0.0.1:8200
# vault token lookup <token_name_from_above>


© 2025 Platform9. All Rights Reserved

On This Page
How to Renew vault Token?ProblemEnvironmentProcedureAdditional Information