Kubectl Command Gives a Delayed Response

Problem

The kubectl commands takes time to show the outputs.
The below errors could be seen when run using --v=8 option.

Bash
    
​x
 
$ time kubectl get nodes --v=8----------Snip----------I0811 16:58:18.680294  143430 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the requestI0811 16:58:18.682150  143430 round_trippers.go:422] GET https://localhost:443/apis/metrics.k8s.io/v1beta1?timeout=32    <========================= NOTICE THE TIME GAP =========================> I0811 16:58:23.689536  143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5007 milliseconds. I0811 16:58:23.689584  143430 round_trippers.go:451] Response Headers:I0811 16:58:23.689597  143430 round_trippers.go:454]     Date: Thu, 11 Aug 2022 16:58:23 GMTI0811 16:58:23.689610  143430 round_trippers.go:454]     Audit-Id: 302ec193-57a5-47e0-8fca-9dc1a391222a.......I0811 16:58:23.689681  143430 round_trippers.go:454]     Content-Length: 57I0811 16:58:23.691529  143430 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the requestI0811 16:58:23.692455  143430 round_trippers.go:422] GET https://localhost:443/apis/metrics.k8s.io/v1beta1?timeout=32sI0811 16:58:23.692478  143430 round_trippers.go:429] Request Headers: <========================= NOTICE THE TIME GAP =========================> I0811 16:58:28.695759  143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5003 milliseconds.  I0811 16:58:28.695799  143430 round_trippers.go:451] Response Headers:I0811 16:58:28.695809  143430 round_trippers.go:454]     Audit-Id: 56774c23-e5c7-4437-912e-c95ed640df06I0811 16:58:28.695819  143430 round_trippers.go:454]     Cache-Control: no-cache, privateI0811 16:58:28.695827  143430 round_trippers.go:454]     Content-Type: text/plain; charset=utf-8----------Snip----------NAME                 STATUS   ROLES    AGE   VERSIONmy-cluster-master0   Ready    master   25h   v1.20.15my-cluster-master1   Ready    master   25h   v1.20.15my-cluster-master2   Ready    master   25h   v1.20.15my-cluster-worker0   Ready    worker   25h   v1.20.15 real    0m15.160suser    0m0.159ssys     0m0.077s
Copy

Environment

Platform9 Managed Kubernetes - All versions

Cause

While querying for the requested resource using kubectl commands, it also queries the metrics services to validate few things before giving the actual outputs.
The query to metrics service holds a timeout of 5003 milliseconds(5seconds) and retries 2 times during the kubectl call as seen above.

Bash
    
 
I0811 16:58:23.691529  143430 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request....I0811 16:58:28.695759  143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5003 milliseconds.
Copy

If the metrics service is unavailable due to some reason/issues, the total timeout adds up to 10 seconds, causing the actual output to return after ~15 seconds.

Resolution

Check the status of the resources associated with metrics apiservice.
The issue could be with any of the metrics resource. Below commands could be used to troubleshoot in respective direction.

Bash
    
xxxxxxxxxx
 
# kubectl top nodes# kubectl describe po <metric-server-pod> -n kube-system# kubectl logs <metric-server-pod> -n kube-system -c metrics-server-nanny# kubectl get --raw /apis/metrics.k8s.io/v1beta1 # kubectl get deployment <metrics-server>-n kube-system -oyaml# kubectl api-resources
Copy

Last updated on

Was this page helpful?