Custom CertManager Pod in CrashLoopBackoff During Luigi Installation

Problem

The custom cert-manager pod is crashing due to permissions issue.

Cluster
    
xxxxxxxxxx
 
% kubectl get pods -A | grep cert-managercert-manager cert-manager-cainjector-646bf69b85-xhbxp 0/1 CrashLoopBackOff 64 (78s ago) 9h
Copy

Pod logs
    
 
% k logs cert-manager-cainjector-646bf69b85-z4ph9 -n cert-manager --tail 2E0404 20:55:14.115006 1 main.go:45] "cert-manager: error executing command" err="customresourcedefinitions.apiextensions.k8s.io \"certificates.cert-manager.io\" is forbidden: User \"system:serviceaccount:cert-manager:cert-manager-cainjector\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"
Copy

Environment

Platform9 Managed Kubernetes - v5.9.4
Kubernetes version 1.28.6

Answer

This is a known issue, and it is being tracked in the jira PMK-6659.

Workaround

To completely disable pf9 managed cert-manager and continue using custom cert-manager:

Patch the pf9-addon-operator image to the custom private image platform9/pf9-addon-operator:8.0.5-hf1 which doesn't install/uninstall pf9-managed cert-manager.
Apply the below script, which will point all the CRB from luigi-system to cert-manager system.

Master node
    
xxxxxxxxxx
 
#!/bin/bash # List of ClusterRoleBindings to updateCRBS=(  cert-manager-cainjector  cert-manager-controller-issuers  cert-manager-controller-clusterissuers  cert-manager-controller-certificates  cert-manager-controller-orders  cert-manager-controller-challenges  cert-manager-controller-ingress-shim  cert-manager-controller-approve:cert-manager-io  cert-manager-controller-certificatesigningrequests  cert-manager-webhook:subjectaccessreviews) # New namespace valueNEW_NAMESPACE="cert-manager" echo "Updating ClusterRoleBinding subjects to use namespace: $NEW_NAMESPACE" for crb in "${CRBS[@]}"; do  echo "Patching $crb..."  kubectl patch clusterrolebinding "$crb" \    --type=json \    -p='[{"op": "replace", "path": "/subjects/0/namespace", "value": "'"$NEW_NAMESPACE"'"}]'done echo "All ClusterRoleBindings updated successfully."
Copy

Edit the below webhooks to set the namespace as cert-manager instead of luigi-system namespace.

Master node
    
xxxxxxxxxx
 
kubectl edit ValidatingWebhookConfiguration cert-manager-webhookkubectl edit MutatingWebhookConfiguration cert-manager-webhook
Copy

` 4. And delete all the three cert-manager deployments from luigi-system.

Master node
    
xxxxxxxxxx
 
kubectl delete deploy cert-manager-webhook -n luigi-systemkubectl delete deploy cert-manager-cainjector -n luigi-systemkubectl delete deploy cert-manager -n luigi-system
Copy

Once this is done all the pf9-managed cert-manager will be completely cleaned and wont be applied again.

Additional Information

The fix release version is on PMK version 5.14

Last updated on

Was this page helpful?