Nested virtualization in Azure

Fist of all*, I’d like to announce that e-book “Introduction to Windows Server 2016 (Russian Edition)” with my technical review and translation had been published and became available for download.

It was originally written for technical preview but this new edition has plenty fixes and additional notes with full adaptation to the current version of Windows Server. I believe we did the great work in a short time. If you find some typo or mistake, feel free to contact me. *I’m sorry..we’re way off topic..let’s back to Azure

Another important news that I’ve missed – Azure is going to support nested virtualization.  However, only new Ev3 and Dv3 VM series will support it. + Azure team has increased maximum disk size to 4096 Gb for both Premium and Standard disks. Hallelujah 🙂

These VMs come with enabled nested virtualization and completely ready for guest VMs or Hyper-V containers. There are many scenarios where it’d be helpful. For instance, you can create smaller number of large VMs using Ev3 or Dv3 VMs with a full control of resource allocation for any guest VMs. I tested nested VMs in WS2016 and now I’m looking forward to test them in Azure as well.

Azure Nested Virtualization| New VM Sizes Ev3 and Dv3

Update 07/14: Dv3 and Ev3 VMs became available for  certain Azure regions Read how to setup nested virtualization in Azure here

Azure v3 VMs , Nested Virtualization in Azure

 

Nested Virtualization in Windows Server 2016

Introduction

Hello guys,

If you read my post what’s new in Hyper-V Windows Server 2016 you may know about a long-awaited feature called as nested virtualization  .

In Windows Server 2012/2012 R2 we could install Hyper-V role inside a virtual machine, create VM (to test hyper-v replica, for example) but it was impossible to run VMs.

Since Technical Preview 4 and Windows 10 Build 10565 have been announced, Microsoft has added ability to run VMs inside virtualized environment

Nested Virtualization lets you run VMs inside the guest OS. In other words, It allows you to run Hyper-V Server inside a virtual machine.

When nested is not supported or not enabled, Hyper-V has privileged access to the hardware virtualization extensions (lower level) and does not expose them to the guest operating system. Once nested virtualization is enabled (ExposeVirtualizationExtensions = True) for VM Hyper-V exposes required extensions to guest VMs and then we have an ability to install Hyper-V and run guest’s own VMs.

You can set and check ExposeVirtualizationExtensions value by using PS:windows server 2016 enable nested virtialization_6

Nested virtualization is especially useful for development, demo labs, training labs and other test environments (It’s not recommended to use in production). MCTs should be happy.

There is no information about changes in Windows Server 2016 licensing program yet. I hope  we won’t have to buy additional licenses to cover nested Hyper-V VMs

VMware: Customers running nested VMware ESXi/ESX will need to obtain additional licenses for the nested ESXi/ESX.

Requirements

If you are interested in tasting nested virtualization you have to firstly check host requirements:

  • 4 GB RAM available minimum (we need to create a VM with 4 Gb static RAM. It will be our nested Hyper-V)
  • At least Windows Server 2016 TP4 or Windows 10 Build 10565 on both physical host and virtualized host. It’s highly recommended to have same builds in all environments.
  • This feature is currently Intel-only. Intel VT-x is required. There is no support for AMV-V yet

To enable nested virtualization:

1. Create virtual machine running the same build as host(TP4-Child, 2016 TP4, in my case)

  1. Run this script on your host or you use the following lines:
#Download and run
Invoke-WebRequest https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/hyperv-tools/Nested/Enable-NestedVm.ps1 -OutFile ~/Enable-NestedVm.ps1
~/Enable-NestedVm.ps1 -VmName "TP4-Child"

#If you don't have internet connection, download and copy script to host and run then
.\Enable-NestedVm.ps1 -VmName "TP4-Child"

Script does the following:

  • disables dynamic memory on VM
  • shutdowns VM
  • enables virtualization extensions
  • enables MAC Address Spoofing (for network connectivity inside the guests)
  • sets minimum required 4 Gb static memory to VM

tp4_enable_nested_virtualization_windows_server_2016

  1. Install Hyper-V on guest VM (do you know about PowerShell Direct?)
#Type credential for admin user on VM
$Credential=Get-Credential

#Invoke remote command
Invoke-Command -VMName TP4-Root { Enable-WindowsOptionalFeature -FeatureName Microsoft-Hyper-V -Online; Restart-Computer} -Credential $credential

windows server 2016 enable nested virtialization

  1. Install Hyper-V Management Tools
ICM -VMName TP4-Root {Add-WindowsFeature RSAT-Hyper-V-Tool -IncludeAllSubFeature} -Credential $credential

windows server 2016 enable nested virtialization_2

  1. I copied Windows Server 2016 TP4 ISO (read my post about guest services in Hyper-V)  to VM on which  I have just installed the Hyper-V role
Copy-VMFile -SourcePath "unc or local path to ISO" -DestinationPath c:\iso\nameofiso.iso -CreateFullPath -FileSource Host -VMName TP4-Root

windows server 2016 enable nested virtialization_36. Create VM (I named it as “TP4-Child”) inside VM (sounds weird Улыбка), attach ISO that we copied earlier and install OS as usual.windows server 2016 enable nested virtialization_47.  Homer – is my Hyper-V (virtualized) running on Hyper-V HV02 (physical host) , Bart – Homer’s child (I guess you knowУлыбка)

windows server 2016 enable nested virtialization_5

Please keep in mind some known issues and limitations:

  • Hosts with Device Guard enabled cannot expose virtualization extensions to guests.Hosts with Virtualization Based Security (VBS) enabled cannot expose virtualization extensions to guests. You must first disable VBS in order to preview nested virtualization.
  • Once nested virtualization is enabled in a virtual machine, the following features are no longer compatible with that VM (Homer)
    • Dynamic memory must be disabled
    • Runtime memory resize does not work
    • You cannot apply checkpoint to a running VM
    • VM which hosts other VMs cannot be live migrated.
    • You cannot save or restore VM
  • VM connection keeps being lost : if you are using blank password, change it and check connection again

I’d like to thank you for reading and have a nice virtualization! …and NESTED VIRTUALIZATION!