How To Renew Vault Token LTS1 Setup

Problem

  • Unable to run the kubectl commands as the vault token was expired on the master nodes.

  • Nodelet phases getting stuck on Gen-Certs Phase

File \"<string>\", line 1, in <module>"}
KeyError: 'data'"}
Certificate is not signed by CA"}
Error loading file /tmp/authbs-certs.abc/apiserver/
error sending status update to sunpike: rpc error: code = Unknown desc = apiserver storage error: an error on the server

"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate has expired or is not yet valid\""}

While checking the directory/tmp/authbs-certs.abc/apiserver/ mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.

[root@master0 apiserver]# cat /tmp/authbs-certs.abc/apiserver/request.json
{"errors":["permission denied"]}

Environment

  • Platform9 Edge Cloud v-5.3.

Solution

To recover from this issue, it is required to regenerate the vault token.

Validation

Steps to validate the token expiry:

  1. SSH into the DU VM as root user.

  2. Export the required details.

export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

export VAULT_ADDR=http://127.0.0.1:8200

CLUSTER_UUID=<CLUSTER_UUID>

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

CLUSTER_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

  1. Run the below command to know token expiry details:

/usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN

Example:

SAMPLE:

# /usr/local/bin/vault token lookup $CLUSTER_VAULT_TOKEN
Key Value
--- -----
accessor [ACCESSOR-ID]
creation_time [CREATION TIMESTAMP]
creation_ttl 26280h
display_name token
entity_id n/a
expire_time [EXPIRY TIMESTAMP]
explicit_max_ttl 0s
id [ID]
issue_time [ISSUE TIMESTAMP]
meta <nil>
num_uses 0
orphan false
path auth/token/create
policies [POLICIES]
renewable true
ttl 26215h49m50s
type service

Procedure

The steps to regenerate the vault token are:

Perform below 1 to 7 steps as a root user in the DU VM:

1. Retrieve the affected cluster's UUID

mysql qbert -e "select name,id,uuid,status,lastOk,lastOp,taskStatus,kubeRoleVersion from clusters;"

2. Check the current vault token for the affected cluster

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='$CLUSTER_UUID';"

3. Set the value for the following variables to regenerate the token:

CLUSTER_UUID=<Cluster-UUID>

OLD_VAULT_TOKEN=$(mysql qbert -Bse "SELECT vaultToken FROM clusters WHERE uuid='$CLUSTER_UUID'")

ROOT_VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

4. Verify the values of the variables $OLD_VAULT_TOKENand $ROOT_VAULT_TOKEN

echo $OLD_VAULT_TOKEN

echo $ROOT_VAULT_TOKEN

5. Generate new vault token using the below commands:

NEW_TOKEN_RESP=$(curl -X POST -H "X-Vault-Token: $ROOT_VAULT_TOKEN" --data '{"policies": ["'$CLUSTER_UUID'"], "ttl": "26280h"}' http://localhost:8200/v1/auth/token/create)

NEW_TOKEN=$(echo $NEW_TOKEN_RESP | jq -r '.auth.client_token')

echo "New Vault-Token generated - $NEW_TOKEN"

6. Update the new token in the qbert database:

mysql qbert -e "UPDATE clusters SET vaultToken='$NEW_TOKEN' WHERE uuid='$CLUSTER_UUID'"

7. Verify the newly generated token using:

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<uuid of the affected cluster>';"

  1. Restart the nodeletd phases on each master nodes/affected nodes one at a time.

# systemctl stop pf9-hostagent pf9-nodeletd
# /opt/pf9/nodelet/nodeletd phases restart
# systemctl start pf9-hostagent pf9-nodeletd

9. Check the new token in the file /etc/pf9/kube.env

# grep -i vault

Additional Information

Post pf9-vault token renewal, if the new token generated in the DU VM is not propagated to the nodes in file /etc/pf9/kube.env perform the below workaround:

  1. Manually copy the vault token from the DU VM:

mysql qbert -e "select name,uuid,vaultToken from clusters where uuid='<Cluster-ID>';"

  1. Replace the token in the node in file /etc/pf9/kube.env

export VAULT_TOKEN=<REPLACE_WITH_NEW_TOKEN>

  1. Restart the nodeletd phases in the nodes. This should pick up the new pf9-vault token.


© 2025 Platform9. All Rights Reserved

On This Page
How To Renew Vault Token LTS1 SetupProblemEnvironmentSolutionValidationProcedureAdditional Information