How to customize a VMware ESXi image and install it in a Hyper-V VM

I’ve been doing recently  VMware ESXi deployment in my lab environment and would like to share main steps required to make it work on nested Hyper-V. Needless to say, nested virtualization works great only for demo and labs, therefore, running ESXi under Hyper-V is a completely unsupported in production environments.

Anyway, carry out the following steps to install ESXi (6.0, in my case. although these steps should work for newer versions as well):

1. Download VMWare ESXi offline bundle available at product download page (e.g. ESXi6.0). You can also download a ESXi image customized by vendor. For example, here is a direct download link for Dell’s ESXi 6.0 image which includes Dell’s VIBs in addition to built-in installation bundles provided by VMware.

2. Download the driver  which allows running ESXi as a VM under Microsoft Hyper-V (net-tulip, it’s actually a network driver which should be added to ESXi image. Otherwise, ESXi installation will be blocked)

3. Copy the downloaded files to the same folder (e.g. ‘D:\Images\VMware ESXi 6\’). It’ll be used as a work folder.

4. Download and install VMware PowerCLI 6.3 or newer

5. Once PowerCLI is installed,  run it and set location to the folder containing the files, then add offline depot ZIP files to the current PowerCLI session as shown below:

cd 'D:\Images\VMware ESXi 6\'
Add-EsxSoftwareDepot .\esxi60_bundle.zip
Add-EsxSoftwareDepot .\net-tulip-1.1.15-1-offline_bundle.zip

6. Retrieve the name of the standard image profile and note it (it’ll will be used as a clone for a new profile):

Get-EsxImageProfile|ft Name

7. Create a new image profile by cloning existing profile which name you just noted, and then add the driver’s package to the profile:

#Create a new image profile
New-EsxImageProfile -CloneProfile ESXi-6.0.0-2494585  -Name rlevchenko.com -Vendor custom

#Add custom packages
Add-EsxSoftwarePackage -ImageProfile rlevchenko.com -SoftwarePackage net-tulip -Force

image

If AcceptanceLevel is set to PartnerSupported by default (as in the picture above) and custom packages which you are going to add to the image profile have Community acceptance level, you will receive an error during creating an ESXi ISO and it’s installation . To resolve this, set the acceptance level of the image profile to CommunitySupported by running the following command: Set-EsxImageProfile -AcceptanceLevel CommunitySupported –ImageProfile rlevchenko.com

image

8. Now it’s time to create an ISO from the customized ESXi image.To do this, run the following command:

Export-EsxImageProfile -ImageProfile rlevchenko.com -FilePath D:\Images\esxi60_custom.iso -ExportToIso -Force

Create a new VM with the following settings:

  • Generation 1
  • Static RAM (> 4Gb is recommended)
  • More than 1vCPU
  • Legacy network adapter connected to the switch

A sample of VM’s configuration is shown below:

image

9. Once you finished to configure a VM, enable virtualization extensions on the VM’s CPU. Optionally, you can download a script available at github to check VM’s configuration and  enable nested virtualization. Both options are allowed:

#Option 1
Set-VMProcessor -VMName "vHost-01" -ExposeVirtualizationExtensions $True

#Option2
 .\Enable-NestedVM.ps1 -vmName "vHost-01"

10. Turn on the VM, attach the created ESXi ISO and press TAB on the boot screen, then type ignoreHeadless=TRUE and press Enter. Otherwise, ESXi boot will hang while booting  (I assume it’s all because ESXi is running on non-HCL hardware. VM is a bit out of the HCL list..).

esxi on hyper-v_1

11. Complete ESXi installation process (as usual), reboot it and press SHIFT+O during the startup, and then enable ignoreHeadless option again as shown in the screenshot:

esxi on hyper-v_4

Once ESXi is successfully started, define settings for management network, enable a Shell, and then press Alt+F1 to enter to a console. We need to set a VMKernel boot-time parameter. Otherwise, you will always need to enable ignoreHeadless after every reboot.

Provide root credentials and  type esxcfg-advcfg -k TRUE ignoreHeadless

esxi on hyper-v_6

Close the console by pressing ALT+F2, reboot ESXi and verify that it starts up seamlessly.

That’s it. Now you have a ESXi host running on a Hyper-V VM.

Until then,

enjoy your day :)!

Deploying VMFleet to test S2D performance

VMfleet is a storage load generator to stress your S2D. Actually it burns your disks/cpu/storage network to check whether S2D is stable or not. VMFleet is also used to get performance results of S2D storage subsystem.

To get started, you need to do the following:

  1. Download VMFleet located at https://github.com/Microsoft/diskspd , click on Clone/Download and then extract archive to the folder named C:\VMFleet, for example.

  2. Download DiskSpd tool from http://aka.ms/diskspd and extract to C:\diskspd

  3. Prepare gold image of Windows Server 2016 Core. After OS installation, provide admin credentials and shutdown the VM. Sysprep is not required.

  4. Prepare VMFleet :

#Create new virtual disks for VMFleet by running this command:

Get-ClusterNode |% { New-Volume -StoragePoolFriendlyName S2D* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size 1Tb }

#Create new volume for VMFleet files with friendly name Collect:

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Collect -FileSystem CSVFS_ReFS -Size 500GB

#Install VMFleet by running the following one-liner:

.\install-vmfleet.ps1 -Source C:\VMFleet\diskspd-master\diskspd-master\Frameworks\VMFleet

#Copy the DISKSPD.exe to the Tools folder:

 Copy-Item c:\diskspd\diskspd.exe -Destination c:\clusterstorage\collect\control\tools

#Copy gold image (VHDX) with Windows Server 2016 Core (not sysprepped) to the Collect folder:

 Copy-Item C:\ClusterStorage\Volume03_test\VMFleet-Gold\VMFleet-Gold_disk_1.vhdx `
 -Destination C:\ClusterStorage\Collect

# Run update-csv.ps1 script to manage the placement of CSV volumes and VMs per a naming convention.
# This script also changes ownership of test CSV volumes if required.

.\update-csv.ps1 -renamecsvmounts $true

# Start creating VMs for VMfleet environment.
# In the following example, 15 VMs will be created on each host,
# adminpass represents my local administrator password used in the gold image
#  connectuser and connectpass are credentials with cluster management rights:

.\create-vmfleet.ps1 -basevhd 'C:\ClusterStorage\Collect\VMFleet-Gold_disk_1.vhdx'-vms 15 `
-adminpass 'Pass123' -connectuser 'rllab\rlevchenko' -connectpass '''RTM$nRdT@5J5'''

#Once VMs are created, change their configuration if required:

.\set-vmfleet.ps1 -ProcessorCount 2 -MemoryStartupBytes 4GB `
-MemoryMinimumBytes 4GB -MemoryMaximumBytes 4GB

At this step, you are ready to start VMFleet VMs and then check system’s status:

#To start VMs
.\start-vmfleet.ps1

#To check cluster health
.\test-clusterhealth.ps1

#To get the current status of cluster
.\watch-cluster.ps1

To run stress test, execute start-sweep.ps1 script that uses the following parameters for diskspd.exe:

  • b: list of buffer sizes (KiB)
  • t: list of thread counts
  • o: list of outstanding IO counts
  • w: list of write ratios
  • p: list of patterns (random: r, sequential: s, sequential interlocked: si)
  • warm: duration of pre-measurement warmup (seconds)
  • d: duration of measured interval (seconds)
  • cool: duration of post-measurement cooldown (seconds)

In the following example, I run 100% read test with 20 outstanding IOs and 8 threads, duration is set to 60 seconds excluding default 120 seconds for cooldown and warmup.

.\Start-Sweep.ps1 -b 4 -t 8 -o 20 -w 0 -d 60

To get the live test results, use then watch-cluster.ps1:

vmfleet_watchcluster

To stop VMFleet VMs after testing:

.\stop-vmfleet.ps1 -method Shutdown

To destroy VMFleet (CAUTION! This script stops and removes all clustered VMs) :

.\destroy-vmfleet.ps1