Instance Failing to Boot, Entering Emergency Mode: "Timed out waiting for device"

Problem

• Instance is failing to boot.
• The instance console log shows that it timed out waiting for a block device to be mounted.

xxxxxxxxxx

Timed out waiting for device dev-vdc3.device.

• Subsequently, the instance can be seen transitioning into Emergency Mode.

xxxxxxxxxx

Welcome to emerg

Cannot open access to console, the root account is locked.

See sulogin(8) man page for more details.

Press Enter to continue.

Environment

• Platform9 Managed OpenStack - All Versions
• Nova
• Cinder
• CentOS – Guest Instance

Cause

The device name is improperly specified and/or the naming of the devices has changed during the init/boot process of the guest VM which is causing a conflict on attempted mount. The mounting failure results in entering into Emergency Mode.

Resolution

1. Rescue the instance – optionally specifying an image separate from the one used to boot initially.

xxxxxxxxxx

$ openstack server rescue [UUID] --image [IMAGE_UUID]

2. SSH or login to the VNC console (if password-authentication is enabled) to the instance which is now in a "Rescued" state.

xxxxxxxxxx

$ ssh -i [keypair] [rescue-user]@[IP/hostname]

Note: the same SSH keypair will be injected via cloud-init to the recovery image as was specified on instance creation. You may need to substitute the username dependent on the OS you're booting with for the rescue image, though. Additionally, the ECDSA fingerprint will change temporarily while booted into the rescue image, and you will need to either ignore 'StrictHostKeyChecking' or modify your .ssh/known_hosts file accordingly.

3. Mount the guest FS.
4. Edit /etc/fstab on the mounted guest FS and modify or remove any offending entry.
5. Unrescue the instance.

xxxxxxxxxx

$ openstack server unrescue [UUID]