Degraded LoadBalancer Ingress Endpoint Performance (AWS)

Problem

A Kubernetes LoadBalancer Ingress endpoint for a particular service is slow to respond (± ~3 seconds).

❯ httpstat http://[SVC/LB Ingress FQDN]:8080/health

Connected to X.X.X.X:8080 from 127.0.0.1:62137

HTTP/1.1 200x-request-id: fae0bb7f-9a9b-402d-84a6-a1d1438820a7

Content-Type: application/vnd.spring-boot.actuator.v3+json

Content-Length: 15  

DNS Lookup TCP Connection Server Processing Content Transfer[ 6ms | 0ms | 20045ms | 1ms ] | | | | namelookup:6ms | | | connect:6ms | | starttransfer:20051ms | total:20052ms

Environment

• Platform9 Managed Kubernetes - All Versions
• AWS Cloud Provider
• Flannel

Cause

Flannel is unable to contact etcd. A firewall appliance may be present within the VPC which is blocking the traffic, especially on Port 4001 which is required for etcd communication as per the Networking Prerequisites.

Resolution

1. Whitelist the ports called out in the Networking Prerequisites, especially for etcd.
2. Restart Flannel on the worker node(s).

xxxxxxxxxx

$ docker stop flannel

$ docker start flannel