Migrations Failing After Host Reprovisioned

Problem

Migrations of instances are failing to a specific host after the host was re-provisioned.

2022-02-14 07:40:05.963 ERROR oslo_messaging.rpc.server [req-8a10be95-3555-4c99-807e-e8ddbef6d01f Geraldo Macedo transition1] Exception during message handling: nova.exception.ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.

Command: ssh -o BatchMode=yes <Destination Host IP>  mkdir -p /pf9/instances/<Instance_uuid>

Exit code: 255

Stdout: ''

Stderr: 'No ECDSA host key is known for <Destination Host IP> and you have requested strict checking.\r\nHost key verification failed.\r\n'

Environment

• Platform9 Managed OpenStack - All Versions
• Nova

Cause

During a migration attempt, the pf9-ostackhost service is responsible to perform an ssh on the destination node and create a directory for the instance to store the disk. The Error message is observed when the source host has strict checking enabled in the sshd service configuration file and the key for the remote host is not correct.

Resolution

1. As a workaround, if the destination host is verified and correct then disable the strict key check policy in sshd service and restart the service on the source host.

xxxxxxxxxx

$ grep -v "^#" .ssh/config

Host *

  User pf9

  StrictHostKeyChecking yes <<< change this value to "no"

  IdentityFile /opt/pf9/home/.ssh/id_rsa

  PasswordAuthentication no

xxxxxxxxxx

# systemctl restart sshd

Additional Information

https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html