Kubectl Command Gives a Delayed Response

Problem

  • The kubectl commands takes time to show the outputs.

  • The below errors could be seen when run using --v=8 option.

$ time kubectl get nodes --v=8
----------Snip----------
I0811 16:58:18.680294 143430 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0811 16:58:18.682150 143430 round_trippers.go:422] GET https://localhost:443/apis/metrics.k8s.io/v1beta1?timeout=32

<========================= NOTICE THE TIME GAP =========================>

I0811 16:58:23.689536 143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5007 milliseconds.
I0811 16:58:23.689584 143430 round_trippers.go:451] Response Headers:
I0811 16:58:23.689597 143430 round_trippers.go:454] Date: Thu, 11 Aug 2022 16:58:23 GMT
I0811 16:58:23.689610 143430 round_trippers.go:454] Audit-Id: 302ec193-57a5-47e0-8fca-9dc1a391222a
.......
I0811 16:58:23.689681 143430 round_trippers.go:454] Content-Length: 57

Environment

  • Platform9 Managed Kubernetes - All versions

Cause

  • While querying for the requested resource using kubectl commands, it also queries the metrics services to validate few things before giving the actual outputs.

  • The query to metrics service holds a timeout of 5003 milliseconds(5seconds) and retries 2 times during the kubectl call as seen above.

I0811 16:58:23.691529 143430 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
....
I0811 16:58:28.695759 143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5003 milliseconds.

  • If the metrics service is unavailable due to some reason/issues, the total timeout adds up to 10 seconds, causing the actual output to return after ~15 seconds.

Resolution

  • Check the status of the resources associated with metrics apiservice.

  • The issue could be with any of the metrics resource. Below commands could be used to troubleshoot in respective direction.

# kubectl top nodes
# kubectl describe po <metric-server-pod> -n kube-system
# kubectl logs <metric-server-pod> -n kube-system -c metrics-server-nanny
# kubectl get --raw /apis/metrics.k8s.io/v1beta1
# kubectl get deployment <metrics-server>-n kube-system -oyaml
# kubectl api-resources


© 2025 Platform9. All Rights Reserved

On This Page
Kubectl Command Gives a Delayed ResponseProblemEnvironmentCauseResolution