Kubectl command failing with certificate error

Problem

kubectl commands are failing and throwing the below certificate error in kubelet logs:

x509: certificate has expired or is not yet valid:

cat /tmp/authbs-certs.xxxx/kubelet/apiserver/request.json

extract_vault_json certificate

/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''certificate'\''])'

Traceback (most recent call last):

File "<string>", line 1, in <module>

KeyError: 'data'

cat /tmp/authbs-certs.fL0T/kubelet/apiserver/request.json

extract_vault_json issuing_ca

/opt/pf9/python/bin/python -c 'import sys, json; print(json.load(sys.stdin)['\''data'\'']['\''issuing_ca'\''])'

openssl verify -CAfile /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt /tmp/authbs-certs.xxxx/kubelet/apiserver/request.crt

Error loading file /tmp/authbs-certs.xxxx/kubelet/apiserver/ca.crt

+ echo 'Certificate is not signed by CA'

Certificate is not signed by CA

While checking the directory/tmp/authbs-certs.xxxx/kubelet/apiserver mentioned in the error, the file request.json had the entryPermission denied instead of the certificate information.

$ cat request.json

{"errors":["permission denied"]}

$ grep -i vault /etc/pf9/kube.env

$ export VAULT_TOKEN=$(mysql qbert -Bse "SELECT credential_value FROM qbert_secrets where credential_name='root_token'")

$export VAULT_ADDR=http://127.0.0.1:8200

$vault token lookup <token id>

If the above command return like below instead of the token information, then its confirmed that the vault token is expired and need to renew the vault token:

Error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup

Code: 403. Errors:

* bad token

Last updated on

Was this page helpful?