Instance Failed To Start When Multipath Volume is Attached

Problem

VM is failing to start when multipath volumes are attached to the VM and ostackhost shows below error logs:

ERROR oslo_messaging.rpc.server [req-ID] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: multipath -f <multipath_ID>
Exit code: 1
Stdout: 'Apr 09 04:25:29 | <multipath_ID>-part7: map in use\nApr 09 04:25:29 | failed to remove multipath map <multipath_ID>\n'
Stderr: ''
TRACE oslo_messaging.rpc.server Traceback (most recent call last):
TRACE oslo_messaging.rpc.server File "/opt/pf9/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
TRACE oslo_messaging.rpc.server File "/opt/pf9/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)

Environment

  • Platform9 Managed OpenStack - v5.1 and Higher

  • Multi-path Volumes

Cause

The hosts is failing to flush the multipath map and creating stale entries.

Resolution

  • Login to the host where the VM is running and list the paths to the LUN and volumes

# multipath -ll <multipath_ID>
# lsblk /dev/dm-<DM_ID>
  • Remove the volume entries

# dmsetup remove <volume-entries>
  • Remove the named multipath device.

# multipath -f /dev/disk/by-id/dm-uuid-mpath-<ID>
  • Now start the stop the server and start it again

# openstack server stop <UUID>
# openstack server start <UUID>