Title
Create new category
Edit page index title
Edit category
Edit link
Live Migration Fails With Error: "Migration pre-check error: CPU doesn't have compatibility. internal error: Unknown CPU feature ssbd.
Problem
Attempts to perform a live migration of an instance fails with the following error.
Migration pre-check error: CPU doesn't have compatibility. internal error: Unknown CPU feature ssbd Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResultEnvironment
- Platform9 Managed OpenStack - All Versions
- Red Hat Enterprise Linux
Cause
The instance being migrated may require a feature not available on the destination host. In this case, ssbd refers to "Speculative Store Bypass Disable", a Spectre vulnerability mitigation technique available in certain Red Hat kernels and presented as a CPU feature or capability on patched kernels.
When a virtual machine instance is created on a compute node with a kernel that is patched against a side-channel attack using speculative store bypass, subsequent migrations or resizes to other compute nodes may fail if that host has not been patched.
Required features for a virtual machine instance can be found using the virsh dumpxml [domain] command shown here.
[root@host1 ~]# virsh dumpxml 101 | grep require [feature policy='require' name='ss'/] [feature policy='require' name='stibp'/] [feature policy='require' name='ssbd'/] [feature policy='require' name='pdpe1gb'/] [feature policy='require' name='hypervisor'/]Resolution
- Verify that all hosts have the
ssbdCPU flag present.
If patched, the output from cat /proc/cpuinfo will reflect ssbd as a CPU flag.
[root@host1 ~]# cat /proc/cpuinfoprocessor : 0vendor_id : GenuineIntelcpu family : 6model : 79model name : Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHzstepping : 1microcode : 0xb00002ecpu MHz : 3200.012cache size : 35840 KBphysical id : 0siblings : 28core id : 0cpu cores : 14apicid : 0initial apicid : 0fpu : yesfpu_exception : yescpuid level : 20wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts spec_ctrl intel_stibp flush_l1dbogomips : 5187.86clflush size : 64cache_alignment : 64address sizes : 46 bits physical, 48 bits virtualpower management:...If patched, the output from virsh capabilities will also reflect the ssbd feature.
xxxxxxxxxx[root@host1 ~]# virsh capabilities | grep ssbd [feature name='ssbd'/]Unpatched hosts will not reflect the feature.
xxxxxxxxxx[root@host2 ~]# virsh capabilities | grep ssbd[root@host2 ~]#If patched, a file system path can be used to determine the state of Speculative Store Bypass:[root@host1 ~]# cat /sys/devices/system/cpu/vulnerabilities/spec_store_bypassMitigation: Speculative Store Bypass disabled via prctl and seccompUnpatched hosts may reflect two different states, depending on the kernel version.
[root@host2 ~]# cat /sys/devices/system/cpu/vulnerabilities/spec_store_bypasscat: /sys/devices/system/cpu/vulnerabilities/spec_store_bypass: No such file or directorynon-zero return code[root@host3 ~]# cat /sys/devices/system/cpu/vulnerabilities/spec_store_bypassVulnerable- If necessary, upgrade the kernel to a patched version.
The following kernels have been observed as providing Speculative Store Bypass mitigation.
xxxxxxxxxx3.10.0-862.14.4.el7.x86_643.10.0-862.11.6.el7.x86_64Unpatched kernels may include the following.
xxxxxxxxxx3.10.0-693.el7.x86_643.10.0-693.21.1.el7.x86_64Patched kernels may be provided during RHEL upgrades or installed manually. Upgrading from Red Hat Enterprise Linux 7.4 to Red Hat Enterprise Linux 7.5 should provide a patched kernel and allow migrations and resizes that previously failed due to this issue to complete properly.
Additional Information
For more information on how Red Hat addresses kernel side-channel attacks using Speculative Store Bypass, please refer to Kernel Side-Channel Attack using Speculative Store Bypass - CVE-2018-3639.
If a kernel upgrade is not possible, or if the error is unrelated to the ssbd CPU feature, an instance's XML file can be modified manually to allow for a successful migration between hosts with different capabilities. Please refer to Live Migration Fails With Error: "Unacceptable CPU Info: CPU Doesn't Have Compatibility."