Revert a PV Stuck in a Deleting State

Problem

You accidentally deleted a PV but now it is stuck in a deleting state due to it being attached to a pod.

Environment

• Platform9 Managed Kubernetes - All Versions

Cause

The reason for PV not being deleted successfully is because the kubernetes.io/pv-protection finalizer protects it from being deleted while being attached to a pod.

Resolution

1. Login to the master node of the canary cluster and run the following command.

# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key get / --prefix --keys-only | grep -i 'persistentvolume'

2. In the output, you will find the name of the PV that is stuck in a Terminating state. Run the following command to get the PV data from ETCD:

# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key get /registry/persistentvolumes/<pv_name>

3. Copy the JSON output from the above command and use a text editor to remove the following 2 key value pairs:

xxxxxxxxxx

DeletionTimestamp

DeletionGracePeriodSeconds

4. Once you have the edited JSON, use the following command to update the ETCD using the following command.

# /opt/pf9/pf9-kube/bin/etcdctl --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key put /registry/persistentvolumes/<pv_name> '<edited_JSON>'