Kubevirt Cluster Add-ons Failing To Uninstall With Error "kubevirt-operator-webhook not found"

Problem

pf9-addon pod logs:
    
 
reflector.go:436] storage/cacher.go:/sunpike.platform9.com/clusteraddon: watch of *sunpike.ClusterAddon ended with: Internal error occurred: Object 'Kind' is missing in '{"metadata":{"name":"<cluster-uuid>-kubevirt","namespace":"default","uid":"<cluster-uuid>","creationTimestamp":"2022-06-28T11:11:45Z","deletionTimestamp":"2022-06-30T15:43:01Z","deletionGracePeriodSeconds":0,"labels":{"sunpike.pf9.io/cluster":"<cluster-uuid>","type":"kubevirt"},"finalizers":["addons.pf9.io"],"managedFields":[{"manager":"unknown","operation":"Update","apiVersion":"sunpike.platform9.com/v1alpha2","time":"2022-06-28T11:11:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:sunpike.pf9.io/cluster":{},"f:type":{}}},"f:spec":{"f:clusterID":{},"f:type":{},"f:version":{},"f:watch":{}}}},{"manager":"sunpike-controller-manager","operation":"Update","apiVersion":"sunpike.platform9.com/v1alpha2","time":"2022-06-28T11:13:32Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:finalizers":{".":{},"v:\"addons.pf9.io\"":{}}}}},{"manager":"manager","operation":"Update","apiVersion":"sunpike.platform9.com/v1alpha2","time":"2022-07-01T04:11:40Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:healthy":{},"f:message":{},"f:phase":{}}}}]},"spec":{"clusterID":"<cluster-uuid>","version":"0.45.0","type":"kubevirt","override":{},"watch":true},"status":{"phase":"UninstallAddonError","message":"could not delete object (cdi.kubevirt.io/v1beta1, Kind=CDI) /cdi: Internal error occurred: failed calling webhook \"cdi-validate.cdi.kubevirt.io\": Post \"https://cdi-api.cdi.svc:443/cdi-validate?timeout=30s\": x509: certificate has expired or is not yet valid: current time 2022-06-30T15:44:34Z is after 2022-06-29T05:49:15Z","healthy":true,"lastChecked":null}}
Copy

kubevirt addon pod
    
 
message: 'could not delete object (kubevirt.io/v1, Kind=KubeVirt) kubevirt/kubevirt: Internal error occurred: failed calling webhook "kubevirt-validator.kubevirt.io": Post "https://kubevirt-operator-webhook.kubevirt.svc:443/kubevirt-validate-delete?timeout=10s": service "kubevirt-operator-webhook" not found'
Copy

Environment

Platform9 Managed Kubernetes - v5.4 and Higher
Kubevirt Cluster Add-ons

Cause

This is a known issue and Platform9 Engineering is working on the fix in KVIRT-131. Please reach out to Support to know the status of the JIRA

Workaround

Scale down the pf9-addon-operator pod:

command
    
xxxxxxxxxx
 
$ kubectl -n pf9-addons scale --replicas=0 deployment pf9-addon-operator
Copy

Remove finalizer from addon from pf9-addons namespace if kubevirt addons is stuck.

Edit pf9-addons deployment and set DISABLE_SUNPIKE_SYNC value as "true"

command
    
xxxxxxxxxx
 
$ kubectl edit deploy pf9-addon-operator -n pf9-addons
Copy

Make sure the all the resources are cleaned up from cdi and kubevirt namespaces

command
    
xxxxxxxxxx
 
$ kubectl get all -n cdi$ kubectl get all -n kubevirt
Copy

Once the resources are deleted scale the replica of pf9-addon-operator to 1 and set DISABLE_SUNPIKE_SYNC value as "false"

command
    
xxxxxxxxxx
 
$ kubectl edit deploy pf9-addon-operator -n pf9-addons$ kubectl scale --replicas=1 deployment pf9-addon-operator -n pf9-addons
Copy

Please reach out to Platform9 Support for any assistance or queries.

Last updated on

Was this page helpful?