Nova-Compute (ostackhost) Fails to Create Resource Provider: "Conflicting resource provider name: <host> already exists."

Problem

• Deauthorizing and reauthorizing a host with the same hostname sometimes leaves a stale resource provider entry in the Nova __Placement DB.
• The following error is observed in the /var/log/pf9/ostackhost.log log.

ERROR nova.scheduler.client.report [...] Failed to create resource provider record in placement API for UUID 19b8176c-8228-43f2-ab7c-8bf1757b5d16. Got 409: {"errors": [{"status": 409, "request_id": "req-fe92d80b-23ac-48d4-96ef-8813e0d3adfe", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: host-kvm3 already exists. ", "title": "Conflict"}]}.

Environment

• Platform9 Managed OpenStack - v4.0 and Higher
• Nova-Compute

Cause

The Resource Manager service (running on and exposed from the Platform9 management plane) fails to clean up the resource provider entry in Nova for a host in the two situations below.

1. If the deauthorize operation did not succeed or if the host was forcefully deauthorized by performing a DB operation.
2. If there are any old instances still associated with this host or if there are any stale allocations.

Resolution

Issue 1:

1. Find the older host in the OpenStack Compute service list by its UUID.

$ openstack compute service list | grep c00563c1-372a-49f6-bb94-fddf7dfc74b2

| 273 | nova-compute   | c00563c1-372a-49f6-bb94-fddf7dfc74b2       | nova     | enabled  | down  | 2023-09-22T13:40:58.000000 |

2. Delete the stale service by ID.

xxxxxxxxxx

$ openstack compute service delete 273

3. Verify that a new Compute service entry is created with the new host UUID.

$  openstack compute service list

| 274 | nova-compute   | 19b8176c-8228-43f2-ab7c-8bf1757b5d16      | nova     | enabled  | up    | 2023-10-02T15:51:43.000000 |

4. Verify that the resource provider got created for the new host. Check the /var/log/pf9/ostackhost.log file to ensure that the resource provider was created successfully.

INFO nova.scheduler.client.report [...] Created resource provider record via placement API for resource provider with UUID 19b8176c-8228-43f2-ab7c-8bf1757b5d16 and name host-kvm3.

Issue 2:

1. Check for stale allocations.

xxxxxxxxxx

$ nova-manage placement audit --verbose

2. Get the Host name associated with the Resource Provider having stale allocations.

$ TOKEN=$(openstack token issue -f value -c id)

$ curl -X GET https://<FQDN>/placement/resource_providers/<Resource_Provider_ID> -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json"

{"uuid": "f39ef00a-ced6-4e72-8a10-94f8ec19342d", "name": "host-kvm3", "generation": 2, "links": [{"rel": "self", "href": "/resource_providers/f39ef00a-ced6-4e72-8a10-94f8ec19342d"}, {"rel": "inventories", "href": "/resource_providers/f39ef00a-ced6-4e72-8a10-94f8ec19342d/inventories"}, {"rel": "usages", "href": "/resource_providers/f39ef00a-ced6-4e72-8a10-94f8ec19342d/usages"}]}

3. Delete stale allocations.

xxxxxxxxxx

$ nova-manage placement audit --verbose --delete

4. Delete Resource Provider.

$ TOKEN=$(openstack token issue -f value -c id)

$ curl -X DELETE https://<FQDN>/placement/resource_providers/<Resource_Provider_ID> -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json"

5. Verify that the resource provider got created for the new host. Check the /var/log/pf9/ostackhost.log file to ensure that the resource provider was created successfully.

INFO nova.scheduler.client.report [...] Created resource provider record via placement API for resource provider with UUID 19b8176c-8228-43f2-ab7c-8bf1757b5d16 and name host-kvm3.

Additional Information

• Documentation