Deploying VMFleet to test S2D performance

VMfleet is a storage load generator to stress your S2D. Actually it burns your disks/cpu/storage network to check whether S2D is stable or not. VMFleet is also used to get performance results of S2D storage subsystem.

To get started, you need to do the following:

  1. Download VMFleet located at https://github.com/Microsoft/diskspd , click on Clone/Download and then extract archive to the folder named C:\VMFleet, for example.

  2. Download DiskSpd tool from http://aka.ms/diskspd and extract to C:\diskspd

  3. Prepare gold image of Windows Server 2016 Core. After OS installation, provide admin credentials and shutdown the VM. Sysprep is not required.

  4. Prepare VMFleet :

#Create new virtual disks for VMFleet by running this command:

Get-ClusterNode |% { New-Volume -StoragePoolFriendlyName S2D* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size 1Tb }

#Create new volume for VMFleet files with friendly name Collect:

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Collect -FileSystem CSVFS_ReFS -Size 500GB

#Install VMFleet by running the following one-liner:

.\install-vmfleet.ps1 -Source C:\VMFleet\diskspd-master\diskspd-master\Frameworks\VMFleet

#Copy the DISKSPD.exe to the Tools folder:

 Copy-Item c:\diskspd\diskspd.exe -Destination c:\clusterstorage\collect\control\tools

#Copy gold image (VHDX) with Windows Server 2016 Core (not sysprepped) to the Collect folder:

 Copy-Item C:\ClusterStorage\Volume03_test\VMFleet-Gold\VMFleet-Gold_disk_1.vhdx `
 -Destination C:\ClusterStorage\Collect

# Run update-csv.ps1 script to manage the placement of CSV volumes and VMs per a naming convention.
# This script also changes ownership of test CSV volumes if required.

.\update-csv.ps1 -renamecsvmounts $true

# Start creating VMs for VMfleet environment.
# In the following example, 15 VMs will be created on each host,
# adminpass represents my local administrator password used in the gold image
#  connectuser and connectpass are credentials with cluster management rights:

.\create-vmfleet.ps1 -basevhd 'C:\ClusterStorage\Collect\VMFleet-Gold_disk_1.vhdx'-vms 15 `
-adminpass 'Pass123' -connectuser 'rllab\rlevchenko' -connectpass '''RTM$nRdT@5J5'''

#Once VMs are created, change their configuration if required:

.\set-vmfleet.ps1 -ProcessorCount 2 -MemoryStartupBytes 4GB `
-MemoryMinimumBytes 4GB -MemoryMaximumBytes 4GB

At this step, you are ready to start VMFleet VMs and then check system’s status:

#To start VMs
.\start-vmfleet.ps1

#To check cluster health
.\test-clusterhealth.ps1

#To get the current status of cluster
.\watch-cluster.ps1

To run stress test, execute start-sweep.ps1 script that uses the following parameters for diskspd.exe:

  • b: list of buffer sizes (KiB)
  • t: list of thread counts
  • o: list of outstanding IO counts
  • w: list of write ratios
  • p: list of patterns (random: r, sequential: s, sequential interlocked: si)
  • warm: duration of pre-measurement warmup (seconds)
  • d: duration of measured interval (seconds)
  • cool: duration of post-measurement cooldown (seconds)

In the following example, I run 100% read test with 20 outstanding IOs and 8 threads, duration is set to 60 seconds excluding default 120 seconds for cooldown and warmup.

.\Start-Sweep.ps1 -b 4 -t 8 -o 20 -w 0 -d 60

To get the live test results, use then watch-cluster.ps1:

vmfleet_watchcluster

To stop VMFleet VMs after testing:

.\stop-vmfleet.ps1 -method Shutdown

To destroy VMFleet (CAUTION! This script stops and removes all clustered VMs) :

.\destroy-vmfleet.ps1

VMFleet: The specified network password is not correct

When you do setup of VMFleet in order to run S2D stress and performance tests, you are required to provide values for parameters connectuser and connectpass to describe user credentials for loopback host connection as shown in the example:

.\create-vmfleet.ps1 -basevhd 'C:\ClusterStorage\Collect\VMFleet-Gold_disk_1.vhdx' -vms 10 -adminpass 'Pass123' -connectuser 'rllab\rlevchenko' -connectpass 'RTY$nTyK@2y9'

These credentials are then injected to each VMFleet VM (162 and 163 strings in the create-vmfleet.ps1):

del -Force z:\users\administrator\launch.ps1 -ErrorAction SilentlyContinue
gc C:\ClusterStorage\collect\control\launch-template.ps1 |% { $_ -replace '__CONNECTUSER__',$using:connectuser -replace '__CONNECTPASS__',$using:connectpass } > z:\users\administrator\launch.ps1

Once VMFleet deployed required VMs, you usually start them by using start-vmfleet.ps1, and then VMFleet automatically tries running the injected launch.ps1 mentioned above :

$script = 'c:\run\master.ps1'

while ($true) {
    Write-Host -fore Green Launching $script `@ $(Get-Date)
    & $script -connectuser __CONNECTUSER__ -connectpass __CONNECTPASS__
    sleep -Seconds 1
}

As a result, master.ps1 will be executed on autologon by each VM using the provided connectuser and connectpass parameters. Master.ps1 establishes a mapping a location containing the Run Script (a standalone load script) and a problem that you might experience at this stage  – wrong credentials.

If you, like me, using special characters in the password (for example, $, @ and etc), you will need to type password using this format: ‘’’ password ‘’’. Otherwise, VMs won’t make a SMB mapping and master.ps1 script failed. The reason is that  connectpass and connectuser values are pasted to the launch.ps1  without any escaping (‘’, “ “ and so on) and PowerShell can cut off your password. So, the right command to create VMFleet VMs, in my example, looks as follows:

.\create-vmfleet.ps1 -basevhd 'C:\ClusterStorage\Collect\VMFleet-Gold_disk_1.vhdx' -vms 10 -adminpass 'Pass123' -connectuser 'rllab\rlevchenko' -connectpass '''RTY$nTyK@2y9'''

I’d also recommend to change ErrorAction to Stop in master.ps1 (45-49 strings) before creating VMs to simplify troubleshooting in case of any errors:

$null = net use l: /d

if ($(Get-SmbMapping) -eq $null) {
    New-SmbMapping -LocalPath l: -RemotePath \\169.254.1.1\c$\clusterstorage\collect\control -UserName $connectuser -Password $connectpass -ErrorAction Stop
}