Kubectl Exec Command is Failing Intermittently With i/o Timeout Error

Problem

• The kubectl exec command is failing intermittently with i/o timeout Error.

xxxxxxxxxx

$ kubect exec -it <POD_NAME> -n <NAMESPACE> -- bash

Error from server: error dialing backend: dial tcp [WORKER_NODE_IP]:10250: i/o timeout

• Corresponding API Server logs on the Master Node.

E0611 [TIME_STAMP] status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"error dialing backend: dial tcp [WORKER_NODE_IP]:10250: i/o timeout"}: error dialing backend: dial tcp [WORKER_NODE_IP]:10250: i/o timeout

Environment

• Platform9 Managed Kubenetes - v5.9 and Higher
• Platform9 Self Managed Cloud Platform - v-5.9.2-3199093 and Higher.

Cause

• Not all nodes in the Kubernetes cluster have the net.ipv4.ip_local_reserved_ports setting configured to reserve the NodePort range. This lead to ephemeral port conflicts and service binding failures.

Resolution

• Ensure consistent reservation of the NodePort range across all cluster nodes (both workers and master nodes) using the net.ipv4.ip_local_reserved_ports sysctl parameter.

Implementation Steps:

• Determine the NodePort range used in your cluster. Default is 30000–32767, unless customized in the Kubernetes API server configuration.
• Apply the reservation on all nodes.

xxxxxxxxxx

echo "net.ipv4.ip_local_reserved_ports = <NODEPORT_IP_RANGE>" >> /etc/sysctl.conf

sysctl -p

• Confirm the change is active.

xxxxxxxxxx

sysctl net.ipv4.ip_local_reserved_ports

• Use automation (e.g., Ansible, scripts, or DaemonSets) to enforce this setting across all current and future nodes.

Additional Information

• To learn more about using sysctls in a Kubernetes Cluster, refer to the Official Kubernetes Documentation.