Kubectl command failing with certificate error

Problem

kubectl commands are failing and throwing the below certificate error in kubelet logs:

Errors in kubelet.log
    
xxxxxxxxxx
 
x509: certificate has expired or is not yet valid:
Copy

The nodelet phase failing on the first cert-gen phase with the below error:

nodelet logs
    
 
cat /tmp/authbs-certs.xxxx/kubelet/apiserver/request.jsonextract_vault_json certificate/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''certificate'\''])'Traceback (most recent call last):File "<string>", line 1, in <module>KeyError: 'data'cat /tmp/authbs-certs.fL0T/kubelet/apiserver/request.jsonextract_vault_json issuing_ca/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''issuing_ca'\''])'openssl verify -CAfile /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt /tmp/authbs-certs.xxxx/kubelet/apiserver/request.crtError loading file /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt+ echo 'Certificate is not signed by CA'Certificate is not signed by CA
Copy

While checking the directory/tmp/authbs-certs.xxxx/kubelet/apiserver mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.

Impacted node
    
xxxxxxxxxx
 
$ cat request.json{"errors":["permission denied"]}
Copy

Environment

Platform9 Edge Cloud - 5.3.0 or Higher

Cause

The vault token is expired.

Procedure

Retrieve the token using below command from the affected node:

Affected node
    
xxxxxxxxxx
 
$ grep -i vault /etc/pf9/kube.env
Copy

To check the validity of the vault token, run the below command:

Affected node
    
​x
 
$ export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'") $export VAULT_ADDR=http://127.0.0.1:8200 $vault token lookup <token id>
Copy

If the above command return like below instead of the token information, then its confirmed that the vault token is expired and need to renew the vault token:

Affected node
    
xxxxxxxxxx
 
Error looking up token: Error making API request. URL: POST http://127.0.0.1:8200/v1/auth/token/lookupCode: 403. Errors: * bad token
Copy

Resolution

To resolve the issue, renew the vault token as per the KB

Last updated on

Was this page helpful?