Adding new node fails with error "Error registering network: failed to acquire lease: out of subnets"

Problem

• Adding new node fails & flannel shows below error:

{"log":"E0516 23:16:46.941582       1 main.go:293] Error registering network: failed to acquire lease: out of subnets\n","stream":"stderr","time":"2023-05-16T23:16:46.94167563Z"}

Environment

• Platform9 Managed Kubernetes- v5.4 and Higher.
• Cluster configured with flannel.
• The node is re-added to the cluster.

Cause

• Flanneld by default sets a TTL for each key in etcd. The default TTL is however 24 hours. So, the subnet is released after 24 hours and new nodes can be added then.

Resolution

• Check the current TTL value using the command below:

# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key get /coreos.com/network/subnets/10.244.72.1-24

• Default TTL value is set to 24 hours and re-adding/adding nodes should work after this TTL expiry after which the unused subnets would be released.