Virtual Machines do not boot after moving from Windows Server 2012 R2 to Windows Server 2019

In the last couple of weeks, we have been hearing reports from customers who are encountering problems after migrating virtual machines directly from Windows Server 2012 R2 to Windows Server 2019. People are seeing error messages like the following:

Critical 03/01/2019 16:13:49 Hyper-V-Worker 18604 None
‘Test VM 1’ has encountered a fatal error but a memory dump could not be generated. Error 0x2. If the problem persists, contact Product Support for the guest operating system. (Virtual machine ID 90B45891-E0EB-4842-8070-F30FF25C663A)

Critical 03/01/2019 16:13:49 Hyper-V-Worker 18560 None
‘Test VM 1’ was reset because an unrecoverable error occurred on a virtual processor that caused a triple fault. If the problem persists, contact Product Support. (Virtual machine ID 90B45891-E0EB-4842-8070-F30FF25C663A)

We have been digging into these issues and have identified the root cause. In Windows Server 2019 we made several changes to the virtual machine firmware (a topic that I plan to blog about another day). In the process we unfortunately exposed a bug. The effect of the bug is that the firmware state on a version 5.0, Generation 2 virtual machine from Windows Server 2012 R2 cannot boot on Windows Server 2019.

Specifically – the bug is exposed by the IPv6 boot data that is stored in the firmware of a Generation 2 virtual machine. Note, this will not effect Generation 1 virtual machines.

We are actively working on a fix for this issue right now.

Workaround

In the meantime, it is possible to work around this. To get the virtual machine to boot you need to get Hyper-V to create new firmware entries for the IPv6 boot data. The easiest way to do this is to change the MAC addresses on any network adapters connected to the affected virtual machine. This process is different for virtual machines with dynamic and static MAC addresses.

Static MAC addresses

For virtual machines with network adapters that are set to use static MAC addresses – all you need to do is to open the virtual machine settings and change the MAC address to a new value:

I have also put together the following PowerShell snippet for people who like to automate things. This script will go through all network adapters on a virtual machine, find the ones with static MAC addresses, and increment them by 100.

# The name of the virtual machine that needs to be fixed
$VMname = "Broken VM"

# Iterate over all the network adapters in the virutal machine
Get-VMNetworkAdapter -VMName $VMname | % {
     # Skip any network adapters that are using dynamic MAC addresses
     if (!($_.DynamicMacAddressEnabled)) 
         {
             # Read the current MAC address, add 100, and set the new MAC address
             $newMac = ([int64]"0x$($_.MacAddress)"+100).ToString("X").PadLeft(12,"0")
             Set-VMNetworkAdapter -VMNetworkAdapter $_ -StaticMacAddress $newMac
         }
     } 

The reason why I chose to increment by 100 was incase people have consecutive MAC addresses.

Dynamic MAC addresses

If your virtual machine is using dynamic MAC addresses – it is possible that you will not hit this problem at all. There are a number of cases where Hyper-V will regenerate the MAC address automatically.

You can also force Hyper-V to regenerate dynamic MAC addresses by changing the dynamic MAC address pool range used by Hyper-V. Dimitris Tonias has written a great article on how to configure this that you should review.

Live Migration

One interesting note to make here – when you live migrate a virtual machine it does not boot through the firmware. This means that if you live migrate a virtual machine from Windows Server 2012 R2 to Windows Server 2019 it will continue to run. However it will not boot if you shut it down and try to start it again after the migration. In this situation the above work around will also address the problem.

My apologies to anyone who has been affected by this issue. hopefully we will have a fix out for this out soon!

Cheers,
Ben

Leave a Reply

XBOX One

XBOX one stats

Steam – Recently Played

No Steam games played recently

RSS

GiottoPress by Enrique Chavez