Namespace stuck in "Terminating" state

Problem

• Namespace stuck in terminating state and unable to delete it.

Environment

• Platform9 Edge Cloud

• Platform9 Managed Kubernetes

Cause

• There is a finalizer defined for the affected namespace.

• After executing commands like, kubectl delete namespace <namespace>, Kubernetes checks for a finalizer in the metadata.finalizers field. A finalizer is a special metadata key that tells Kubernetes to wait until a specific condition is met before it fully deletes a resource. If the condition defined in the finalizer cannot be deleted for any reason, then the namespace is not deleted either. This puts the namespace into a terminating state awaiting the removal of the resource, which never occurs.

• Check for the kube-controller-manager log files present on the master nodes to identify what are the resources that is failed.

Resolution

• Identify the resource which is waited up on or any error using below command:

• Start by checking the condition field. If any error or condition pointing towards the api-resouces get the list of all api-resources.

• If the issue is due to the api-resource that is not able to handle the request then identify the issue with that resource:

• If the apiservices and its corresponding pods/deployment is no longer running on the node then this could be deleted using below command and later point reinstall (if required) the corresponding addons/application that provides this api-resource

• If there are no failed api-resources associated then, Dump the contents of the namespace in a temporary file, edit the temporary file to remove the kubernetes value from the finalizers field and save the file:

• Set a temporary proxy IP and port, using the following command. Be sure to keep your terminal window open until you delete the stuck namespace:

• From a new terminal window, make an API call with your temporary proxy IP and port :

• Now verify if the namespace is removed.