How to Renew vault Token?

Problem

The vault token is expired.

Environment

Platform9 Managed Kubernetes v5.9.3

Validation

Steps to validate the token expiry:

Exec into pf9-vault pod in Management Plane namespace.

Command
    
 
$ kubectl exec -it -n <MANAGEMENT_PLANE_NAMESPACE> --kubeconfig <KUBECONFIG> $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash
Copy

Export the required details.

Command
    
​x
 
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'") # export VAULT_ADDR=http://127.0.0.1:8200 # CLUSTER_UUID=<CLUSTER_UUID> # OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'") # ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'") # CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")
Copy

Run the below command to know token expiry details:

Command
    
xxxxxxxxxx
 
# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Copy

Example:

Command
    
xxxxxxxxxx
 
SAMPLE: #  /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKENKey                 Value---                 -----accessor            [ACCESSOR-ID]creation_time       [CREATION TIMESTAMP]creation_ttl        26280hdisplay_name        tokenentity_id           n/aexpire_time         <>explicit_max_ttl    0sid                  [ID]issue_time          [ISSUE TIMESTAMP]meta                <nil>num_uses            0orphan              falsepath                auth/token/createpolicies            [POLICIES]renewable           truettl                 26215h49m50stype                service
Copy

Procedure

Exec Into pf9-vault pod in customer namespace

Management cluster
    
 
export KUBECONFIG=kplane-enterprise-2.yamlexport NS=<customer_shortname>kubectl exec -it -n $NS $(kubectl get pods -n $NS -l du-app=pf9-vault -o jsonpath="{.items[0].metadata.name}") -- /bin/bash
Copy

Export required details (in pf9-vault pod)

Management Cluster
    
xxxxxxxxxx
 
CLUSTER_UUID=<cluster_UUID>OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")echo $OLD_VAULT_TOKENecho $ROOT_VAULT_TOKEN
Copy

Generate New Token (in pf9-vault pod)

Management Cluster
    
 
NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')echo "New Vault-Token generated - $NEW_TOKEN"
Copy

Update the new token in qbert Database (in pf9-vault pod)

Management Cluster
    
xxxxxxxxxx
 
mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"
Copy

Open a new terminal window and Verify if the new token is updated at cluster and node level (Execute on a new terminal window)

Management Cluster
    
 
export KUBECONFIG=kplane-enterprise-2.yamlexport NS=<customer_shortname>kubectl -n $NS exec -it deploy/mysqld-exporter -- mysql qbert -e "select name,uuid,vaultToken from clusters where uuid=<cluster_uuid>"kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts <HOST_UUID> -o yaml | grep -i vault
Copy

If the token in Sunpike does not match the token in Qbert, run the following command to patch the Sunpike host object.(from the same terminal window as Step 5 )

Management cluster
    
 
export VAULT_TOKEN=<TOKEN_FROM_QBERT_DB>export CLUSTER_UUID=<CLUSTER_UUID>for i in $(kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl get hosts --no-headers | grep $CLUSTER_UUID | awk '{print $1}'); do kubectl -n $NS exec -it deploy/sunpike-kube-apiserver -c sunpike-kube-apiserver -- kubectl patch host $i -p '{"spec":{"pf9":{"vaultToken":"'${VAULT_TOKEN}'"}}}'; done
Copy

restart the full stack restart on nodes that got stuck at the Cert Generation phase (if any).
revoke the old Token (in pf9-vault pod) - once all nodes are working fine.

Management Cluster
    
 
curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"token": "'$OLD_VAULT_TOKEN'"}' http://localhost:8200/v1/auth/token/revoke
Copy

Additional Information

To check if the vault token is expired follow these steps:

From the problematic master node, perform this.

Master node
    
xxxxxxxxxx
 
#grep -i vault /etc/pf9/kube.env You will get an output like this:   export VAULT_TOKEN="<token_name>" <<--------
Copy

Come to the DU VM and perform this.

DU VM
    
 
# export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")# export VAULT_ADDR=http://127.0.0.1:8200# vault token lookup <token_name_from_above>
Copy

Last updated on

Was this page helpful?