Storage Spaces Direct in Windows Server 2019

Storage Spaces Direct (S2D) is the award-winning software-defined storage (SDS) tecnhology that was first introduced in Windows Server 2016. Since that moment, S2D has become my one of the favourite and frequently used role in Windows Server (on-premises and even in Azure).

Windows Server 2019 that is going to be generally available in the second half of this year (September or October – in accordance with WS 2012-2016 release dates) continues to develop S2D by adding new features. Why don’t we discuss them right now?

TIP: Windows Server 2019 preview build is available here

Scalability

More volumes, bigger capacity per server and cluster

Storage Spaces Direct Scalability in Windows Server 2019

USB Witness

Storage Spaces Direct can be configured with just two nodes. However, we will need to design witness placement to protect our cluster from unexpected failure (in other words, we should achieve quorum). In Windows Server 2016, you can place witness on a file share, cloud (details here) or disk witness (it’d be very strange if it was actually used in S2D clusters). What about customers who don’t have any AD infrastructure or Azure/Internet access? Here is a breakthrough – USB Witness.

usb witness windows server 2019

In short, you will be able to configure a true two-node S2D by using USB thumb drive connected to the router that is already used for VM/management traffic between nodes, for instance. Other two network interfaces (shown on the picture above), could be RDMA-adapters (recommended and supported) or Thunderbolt (POC, Project Kepler-47).

Simply insert the USB drive into the port on the router, set the share name and access information, configure witness in PowerShell: Set-ClusterQuorum -FileShareWitness \path\ -Credential  and you are ready to go.

TIP: the router should support SMB2+ and USB. And, given that a witness.log is a quite small file (just kilobytes), you can use any-sized USB drives. The list of the supported routers will be available later.

Data Deduplication

ReFS  is the recommended file system for S2D, improves VHDX creation/expansion speed (enables Accelerated VHDX operations), provides higher stability by detecting corruptions and allowing you to repair them with no volume downtime. However, some features such as ODX, Data Deduplication are not supported by ReFS in Windows Server 2016.

Starting with the Windows Server 2019 (1709 and later), Data Deduplication has been fully supported for ReFS. It means that you no longer need to choose between NTFS and ReFS file systems while planning S2D volumes. Create ReFS volume, enable Data Deduplication (PowerShell/Windows Admin Center), and then check the savings of storage space (use PowerShell and Get-DedupVolume cmdlet).

data deduplication s2d windows server 2019

Proactive outlier detection

It was quite challenging to investigate  performance issues in S2D in Windows Server 2016. We had to use PowerShell or performance counters (+VMFleet) to get a full picture of our setup’s behavior. Windows Server 2019 significantly simplifies that. S2D now records the outcome (success/failure) and latency (elapsed time) for every read/write IO to every drive without any performance impact. Therefore, drives with latency/outcome issues will be marked in PowerShell and Windows Admin Center as “Abnormal Latency” status. In addition, you can organize pooled drives into peer groups, and then compare latency against peers to quickly find any bottlenecks (new cmdlet: Get-PhysicalDiskIoReport).

This azure-inspired mechanism works on the lower level than performance counters and enabled by default for every SATA,SAS,NVMe drives.

latency outlier detection s2d 2019

Others

-Faster mirror-accelerated parity volumes (~x2)

-PMEM (Persistent Memory) drives support (Intel Optane/NVDIMM-N) for use as cache and capacity

-Deep integration with Windows Admin Center (a free HTML5-based management interface for entire Windows Server infrastructure. We’ll look at this a bit later)

-New networking recommendations for high performance, at scale, or deployments of 4+ nodes:  25 Gbps (or higher) NICs (two or more) that are remote-direct memory access (RDMA) capable, iWARP (recommended) or RoCE

Deploying VMFleet to test S2D performance

VMfleet is a storage load generator to stress your S2D. Actually it burns your disks/cpu/storage network to check whether S2D is stable or not. VMFleet is also used to get performance results of S2D storage subsystem.

To get started, you need to do the following:

  1. Download VMFleet located at https://github.com/Microsoft/diskspd , click on Clone/Download and then extract archive to the folder named C:\VMFleet, for example.

  2. Download DiskSpd tool from http://aka.ms/diskspd and extract to C:\diskspd

  3. Prepare gold image of Windows Server 2016 Core. After OS installation, provide admin credentials and shutdown the VM. Sysprep is not required.

  4. Prepare VMFleet :

#Create new virtual disks for VMFleet by running this command:

Get-ClusterNode |% { New-Volume -StoragePoolFriendlyName S2D* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size 1Tb }

#Create new volume for VMFleet files with friendly name Collect:

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Collect -FileSystem CSVFS_ReFS -Size 500GB

#Install VMFleet by running the following one-liner:

.\install-vmfleet.ps1 -Source C:\VMFleet\diskspd-master\diskspd-master\Frameworks\VMFleet

#Copy the DISKSPD.exe to the Tools folder:

 Copy-Item c:\diskspd\diskspd.exe -Destination c:\clusterstorage\collect\control\tools

#Copy gold image (VHDX) with Windows Server 2016 Core (not sysprepped) to the Collect folder:

 Copy-Item C:\ClusterStorage\Volume03_test\VMFleet-Gold\VMFleet-Gold_disk_1.vhdx `
 -Destination C:\ClusterStorage\Collect

# Run update-csv.ps1 script to manage the placement of CSV volumes and VMs per a naming convention.
# This script also changes ownership of test CSV volumes if required.

.\update-csv.ps1 -renamecsvmounts $true

# Start creating VMs for VMfleet environment.
# In the following example, 15 VMs will be created on each host,
# adminpass represents my local administrator password used in the gold image
#  connectuser and connectpass are credentials with cluster management rights:

.\create-vmfleet.ps1 -basevhd 'C:\ClusterStorage\Collect\VMFleet-Gold_disk_1.vhdx'-vms 15 `
-adminpass 'Pass123' -connectuser 'rllab\rlevchenko' -connectpass '''RTM$nRdT@5J5'''

#Once VMs are created, change their configuration if required:

.\set-vmfleet.ps1 -ProcessorCount 2 -MemoryStartupBytes 4GB `
-MemoryMinimumBytes 4GB -MemoryMaximumBytes 4GB

At this step, you are ready to start VMFleet VMs and then check system’s status:

#To start VMs
.\start-vmfleet.ps1

#To check cluster health
.\test-clusterhealth.ps1

#To get the current status of cluster
.\watch-cluster.ps1

To run stress test, execute start-sweep.ps1 script that uses the following parameters for diskspd.exe:

  • b: list of buffer sizes (KiB)
  • t: list of thread counts
  • o: list of outstanding IO counts
  • w: list of write ratios
  • p: list of patterns (random: r, sequential: s, sequential interlocked: si)
  • warm: duration of pre-measurement warmup (seconds)
  • d: duration of measured interval (seconds)
  • cool: duration of post-measurement cooldown (seconds)

In the following example, I run 100% read test with 20 outstanding IOs and 8 threads, duration is set to 60 seconds excluding default 120 seconds for cooldown and warmup.

.\Start-Sweep.ps1 -b 4 -t 8 -o 20 -w 0 -d 60

To get the live test results, use then watch-cluster.ps1:

vmfleet_watchcluster

To stop VMFleet VMs after testing:

.\stop-vmfleet.ps1 -method Shutdown

To destroy VMFleet (CAUTION! This script stops and removes all clustered VMs) :

.\destroy-vmfleet.ps1