Instance Fails to Start/Power On: "blockdev: cannot open /dev/XXX: No such device or address"

Problem

Instance fails to start or be powered on and transitions into an ERROR state with the following fault/error message: "blockdev: cannot open /dev/XXX: No such device or address".

Environment

  • Platform9 Managed OpenStack - All Versions

  • Cinder (Block Storage)

  • LVM

  • iSCSI

Cause

A stale iSCSI target/connection – one that has not been logged out promptly following the shutdown of the VM – is present and the connection information is no longer valid.

Resolution

  1. Identify which iSCSI target corresponds to the unknown device (i.e. /dev/XXX).

Target: iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6 (non-flash)
Current Portal: 172.30.0.194:3260,1
Persistent Portal: 172.30.0.194:3260,1
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat:6552111637
Iface IPaddress: 172.30.0.152
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 477
iSCSI Connection State: IN LOGIN
iSCSI Session State: FREE
Internal iscsid Session State: REOPEN
...
************************
Attached SCSI devices:
  1. Logout the iSCSI target.

$ sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6 -p 172.30.0.194:3260 --logout
Logging out of session [sid: 477, target: iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6, portal: 172.30.0.194,3260]
Logout of [sid: 477, target: iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6, portal: 172.30.0.194,3260] successful.
  1. Restart or start the Instance/VM.

Tip

The 'attached scsi disk' will be followed by the device name without the preceding /dev path. For example, in the output above, the corresponding device name is sdcd.