Nova-Compute (ostackhost) Fails to Create Resource Provider: "Conflicting resource provider name: <host> already exists."

Problem

  • Deauthorizing and reauthorizing a host with the same hostname sometimes leaves a stale resource provider entry in the Nova __Placement DB.

  • The following error is observed in the /var/log/pf9/ostackhost.log log.

ERROR nova.scheduler.client.report [...] Failed to create resource provider record in placement API for UUID 19b8176c-8228-43f2-ab7c-8bf1757b5d16. Got 409: {"errors": [{"status": 409, "request_id": "req-fe92d80b-23ac-48d4-96ef-8813e0d3adfe", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: host-kvm3 already exists. ", "title": "Conflict"}]}.

Environment

  • Platform9 Managed OpenStack - v4.0 and Higher

  • Nova-Compute

Cause

The Resource Manager service (running on and exposed from the Platform9 management plane) fails to clean up the resource provider entry in Nova for a host in the two situations below.

  1. If the deauthorize operation did not succeed or if the host was forcefully deauthorized by performing a DB operation.

  2. If there are any old instances still associated with this host or if there are any stale allocations.

Resolution

Issue 1:

  1. Find the older host in the OpenStack Compute service list by its UUID.

$ openstack compute service list | grep c00563c1-372a-49f6-bb94-fddf7dfc74b2
| 273 | nova-compute | c00563c1-372a-49f6-bb94-fddf7dfc74b2 | nova | enabled | down | 2023-09-22T13:40:58.000000 |
  1. Delete the stale service by ID.

$ openstack compute service delete 273
  1. Verify that a new Compute service entry is created with the new host UUID.

$ openstack compute service list
| 274 | nova-compute | 19b8176c-8228-43f2-ab7c-8bf1757b5d16 | nova | enabled | up | 2023-10-02T15:51:43.000000 |
  1. Verify that the resource provider got created for the new host. Check the /var/log/pf9/ostackhost.log file to ensure that the resource provider was created successfully.

INFO nova.scheduler.client.report [...] Created resource provider record via placement API for resource provider with UUID 19b8176c-8228-43f2-ab7c-8bf1757b5d16 and name host-kvm3.

Issue 2:

  1. Check for stale allocations.

$ nova-manage placement audit --verbose
  1. Get the Host name associated with the Resource Provider having stale allocations.

$ TOKEN=$(openstack token issue -f value -c id)
$ curl -X GET https://<FQDN>/placement/resource_providers/<Resource_Provider_ID> -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json"
{"uuid": "f39ef00a-ced6-4e72-8a10-94f8ec19342d", "name": "host-kvm3", "generation": 2, "links": [{"rel": "self", "href": "/resource_providers/f39ef00a-ced6-4e72-8a10-94f8ec19342d"}, {"rel": "inventories", "href": "/resource_providers/f39ef00a-ced6-4e72-8a10-94f8ec19342d/inventories"}, {"rel": "usages", "href": "/resource_providers/f39ef00a-ced6-4e72-8a10-94f8ec19342d/usages"}]}
  1. Delete stale allocations.

$ nova-manage placement audit --verbose --delete
  1. Delete Resource Provider.

$ TOKEN=$(openstack token issue -f value -c id)
$ curl -X DELETE https://<FQDN>/placement/resource_providers/<Resource_Provider_ID> -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json"
  1. Verify that the resource provider got created for the new host. Check the /var/log/pf9/ostackhost.log file to ensure that the resource provider was created successfully.

INFO nova.scheduler.client.report [...] Created resource provider record via placement API for resource provider with UUID 19b8176c-8228-43f2-ab7c-8bf1757b5d16 and name host-kvm3.

Additional Information