Calico Node Pods in CrashLoopBackOff: "Unable to get Typha service endpoints from Kubernetes"

Problem

  • The calico-node pod(s) are in CrashLoopBackOff state and/or constantly restarting.

NAME READY STATUS RESTARTS AGE
calico-kube-controllers-f6b98779-qfmhh 1/1 Running 1 101d
calico-node-69nk4 0/1 CrashLoopBackOff 131 7h37m
calico-node-g7pz7 0/1 CrashLoopBackOff 131 7h37m
calico-node-mngnc 0/1 Running 130 7h37m
calico-node-p4ktd 0/1 CrashLoopBackOff 129 7h37m
calico-node-wlv9t 0/1 CrashLoopBackOff 130 7h37m
calico-node-zl5ng 0/1 CrashLoopBackOff 129 7h37m
calico-typha-95b7898f5-g559h 1/1 Running 0 8h
calico-typha-95b7898f5-lmxbw 1/1 Running 0 8h
  • The pod log (via kubectl logs) for the calico-node pod(s) shows the following error related to calico-typha .

[ERROR][68] confd/discovery.go 126: Unable to get Typha service endpoints from Kubernetes. error=endpoints "calico-typha" not found
  • Similarly, the following error is recorded in the calico-typha pod(s).

[ERROR][7] lookup.go 65: Failed to get Typha endpoint from Kubernetes error=endpoints "calico-typha" not found
[WARNING][7] rebalance.go 55: Failed to get number of Typhas error=<nil> numTyphas=0 thread="k8s-poll"
  • No calico-typha service (separate from the DaemonSet and Deployment objects) exists in the kube-system namespace.

Environment

  • Platform9 Managed Kubernetes – v5.5

  • Calico

Cause

The calico-typha service is missing from the kube-system namespace.

Resolution

  1. Create a file calico-typha.yaml with the following content.

# Source: calico/templates/calico-typha.yaml
# This manifest creates a Service, which will be backed by Calico's Typha daemon.
# Typha sits in between Felix and the API server, reducing Calico's load on the API server.

apiVersion: v1
kind: Service
metadata:
name: calico-typha
namespace: kube-system
labels:
k8s-app: calico-typha
spec:
ports:
- port: 5473
protocol: TCP
targetPort: calico-typha
name: calico-typha
selector:
k8s-app: calico-typha
  1. Apply the YAML to the affected cluster.

kubectl apply -f calico-typha.yaml

Additional Information