Author: rlevchenko

How to make VMM highly available

Virtual Machine Manager is the core component of a medium- or large-sized virtualization platform. If you have a lot of hosts and virtual instances , private/hybrid clouds or network virtualization, having highly available VMM is strongly recommended. In this post we will discuss how satisfy all requirements for VMM High availability and set up 2 VMM HA instances.

Hardware and settings behind this post

  • 1 Hyper-V host (TP5, CU2)
  • 1 DC (domainlab.com, guest)
  • 2 VMM (VMM01, VMM 02 , guest)
  • 1 SQL Server (SQL01, guest)
  • internal vSwitches
  • all guests and hosts are up-to-date (Windows Update, no exotic  KBs)
  • user for setup: domain/ent administrator

Prepare your environment for VMM HA

1) VMM creates database to keep it’s configuration data, so remote SQL Server instance with Standard edition at least is required.

TIP:  to remove single point of failure I’d recommend to set up SQL Server HA. AlwaysOn FCI and AG are supported.

2) open 445 and 1433 ports on one or more SQL Servers (if your SQL is in HA)

For Windows Firewall (please note I use Domain Profile here):

New-NetFirewallRule -Name "SQL Server 445" -DisplayName "SQL Server 445" -Profile Domain -Direction Inbound -LocalPort 445 -Protocol TCP -Action Allow
New-NetFirewallRule -Name "SQL Server 1433" -DisplayName "SQL Server 1433" -Profile Domain -Direction Inbound -LocalPort 1433 -Protocol TCP -Action Allow

3) create service account for VMM Service and add one to the local Administrator group on each VMM host

#Use Powershell or dsa.msc
New-ADUser -Name "VMM Service Account" -GivenName "VMM SVC" -SamAccountName vmmsvc -UserPrincipalName vmmsvc@demolab.com -AccountPassword (Read-Host -AsSecureString "Type Password For Service Account")

#Use PS/CMD or lusrmgr.msc
ICM -ComputerName vmmservername {net localgroup administrators /add Demolab\VMMSVC}

4) as almost any clustered role in Windows Server, VMM HA is built on the top of the the well-known Failover-Clustering feature and some requirements are similar.

#Install Failover-Clustering feature on every VMM host
Install-WindowsFeature Failover-Clustering -IncludeManagementTools

TIP: it’s also recommended to add and prepare separate network for cluster heartbeats

5) Create a new cluster (cluadmins.msc or PS)

New-Cluster -Name VMMCL -Node VMM01, VMM02 -StaticAdress 192.168.10.40 -NoStorage -AdministrativeAccessPoint ActiveDirectoryAndDns -IgnoreNetwork 10.10.23.0/24

newclustervmm

6) Check that newly created cluster operates without any errors (cluadmins.msc/cluster core resources and cluster events)

image

7) VMM encrypts some data in the VMM database by using the Data Protection Application Programming Interface (DPAPI). For example, VMM encrypts Run As account credentials and passwords in guest operating system profiles. VMM also encrypts product key information in virtual hard disk properties for virtual machine role scenarios and configuration.

If you move VMM to another station, VMM will not retain the encrypted data. To be able to operate in HA mode, VMM requires Distributed Key Management which stores encryption data in AD DS.

To get it work you need to define the new container in ADDS (adsiedit.msc or PS)

TIP: The account with which you are installing VMM must have Full Control permissions to the VMMDKM container (use adsiedit.msc and security tab)

TIP: you need to create the container in the same domain as the user account with which you are installing VMM

#Get domain's distinguished name (example, "DC=contoso, DC=com")
$DN=(Get-AdDomain).DistinguishedName

#VMMDK container will be created at the root of domain context
New-ADObject -Name "VMMDKM" -Type Container -Path ($DN)

8) Download and install Windows ADK 8.1 with only Deployment Tools and Windows PE features on each VMM host

TIP:  you can download ADK files to a file share and use shared path to install ADK on the next VMM hosts. Timesaver.

adk81

VMM Installation

After you created a new cluster and added all of your VMM hosts to one, you are ready to run setup.exe from VMM media.

The steps are the same as for standalone installation. But there are some differences :

  • you cannot define library servers at the library configuration page. you must add them after setup from VMM console (HA File Server is recommended)
  • wizard automatically determinates (checks cluster membership) that you are going to create VMM HA (1st picture)
  • you need to define settings for VMM cluster role (name and IP address)
  • no option to store encryption keys on the local machine. DKM is selected by default.

Then you need to define

  • name, organization, product key
  • Windows Update settings (on/off)
  • SQL Connections settins (Name, Credentials and etc.)
  • Service account (vmmsvc , in my case) and password
  • DKM path (CN=VMMDKM, DC=demolab, DC=com)
  • cluster name (VMMHA) and it’s ip address

Repeat steps on each VMM host.

Download the latest UR (UR 10 for today) and install it then.

This slideshow requires JavaScript.

Open cluadmins.msc , go to Roles and verify that clustered role with the name VMMHA was created.

image

TIP: owner mode shows you current active VMM server for income connections. VMM HA uses active-passive model.

You can retrieve the active node from VMM console as well. Use FQDN of newly created cluster and default 8100 port as for connection string and check Fabric –Infrastructure – VMM Server section

TIP: it’s recommended to install VMM console on the remote station which is not a member of VMM HA cluster

image

Additional considerations

  • You can only have one implementation of a highly available VMM management server on a given failover cluster.
  • You can have VMM management servers installed on as many as sixteen nodes on a failover cluster, but there can only be one node active at any time.
  • You cannot perform a planned failover (for example, to install a security update or to do maintenance on a node of the cluster) by using the VMM console. To perform a planned failover, use Failover Cluster Manager.
  • During a planned failover, ensure that there are no tasks actively running on the VMM management server. Any running tasks will fail during a failover. Any failed jobs will not start automatically after a failover.
  • Any connections to a highly available VMM management server from the VMM console or the VMM Self-Service Portal (unavailable for new VMM versions (2012 SP1 and later)) will be lost during a failover. The VMM console will be able to reconnect automatically to the highly available VMM management server after a failover.

Thanks for reading.

Have a nice datacenter management!

P.S. don’t know how to upgrade old VMM even if it is installed on 2008 R2? Check my post.

FAQ: How to manually update ESXi and vCenter

In the previous post we discussed the tool that makes upgrading and updating process easier – vSphere Update Manager.

Now we are going to talk over manual process of updating core vSphere’s components – ESXi and vCenter. This type of updating is recommended when you have a small-sized infrastructure (whether it’s demo or production). Today’s FAQ does not cover complex architecture (with external SSO or etc.) and upgrading itself . To get some more information please use links in the references.

Note: updating is not a synonym for upgrading term. Jump from vSphere 6.0 to Update 1 is updating, vSphere 5.1 to 5.5 is upgrading

Hardware behind this post: 1 nested vCenter Appliance, 3 nested ESXi hosts, cluster (HA, DRS,FT) is configured

What’s the right update sequence for ESXi and vCenter ?

vCenter is always first, updating ESXi hosts are the second step. Additionally you need to check VMware Tools (3rd step) and VM Hardware Version (4th step) and upgrade them if applicable.

In vSphere 6 VMFS-3 is deprecated (there is no option to create VMFS-3 and support will no longer be available at the future vSphere releases). So, if you have VMFS-3 datastores you need to upgrade them to VMFS-5 (last step)

How to update vCenter Appliance?

Download ISO with updates @vmwarepatchportal

image

Note: patch-FP at the end of the filename means that is Full Patch , patch-TP – updates for the specific vCenter components and third-party

Mount ISO with updates (“…patch-FP.iso”, in my case) and run the following command:

software-packages install --iso --acceptEulas

vcenter update

Reboot vCenter:

shutdown reboot -r “vCenter Update”

OR

you can send updates to staging (without immediate installation) by using the following strings:

To stage updates:
software-packages stage --iso  

To list staged content:
software-packages list --staged   

To install (ISO must be attached):
software-packages install --staged

How to update Windows-based vCenter?

Go to VMware patch portal and download package for vCenter Windows

image

Attach ISO, run autorun.exe and click on “Patch All”

image

During the patch process, the Management Software undergoes a downtime as the patch restarts the services. So, be ready for vCenter unavailability for a short time.

How to update vCenter Appliance from the web client?

With the release vCenter Server Appliance 6.0 update 1b, you able to update vCenter Appliance from the web-based management interface (VAMI).

Open https://vcenterfqdn:5480, go to Update section and check for updates.

If your hosts are not up to date, click on “Install updates”.

and select “install all updates” or “install third-party updates”

image

How to update ESXi hosts?

Evacuate all critical workloads to the different host (place host to the maintenance mode), enable SSH and run (all updates can be downloaded in the same manner as for vCenter. the main difference – updates are in the zip-format):

escli software vib install -d /vmfs/volumes/deviceidhere/updates.zip

esxi_updates

or use DCUI with enabled bash

Reboot host : vSphere client, reboot cmd or press F12 –> F11

How to upgrade VMware tools and VM hardware?

VMware Tools is a suite of utilities that enhances the performance guest operating system and improves management of the VM. There are different packages for VMware Tools: built-in vCenter/ESXi,  OSPs (were specially created for some OS), OVM or Open VM Tools (newer version of OSPs and they are injected into some linux OS / new version can be installed from online repository). If VM has installed OVT, VMware tools status in vCenter is “guest managed” or “3rdParty/unmanaged”

VM Hardware version reflects the virtual machine’s supported virtual hardware features. These features correspond to the physical hardware available on the ESXi host on which you create the virtual machine. Virtual hardware features include BIOS and EFI, available virtual PCI slots, maximum number of CPUs, maximum memory configuration, and other characteristics typical to hardware

vSphere 6 supports the following versions :

  • VMware tools from ESXi 5.0 and later
  • VM Hardware 4.0 and later

VMware Tools and VM Hardware upgrading steps:

  1. Create backup or snapshot of VM
  2. Upgrade VM Tools by using web-client (install/upgrade VMware tools option , ESXi will mount ISO to the guest OS, run setup.exe on the mounted drive and follow instructions)
  3. If your guest OS is Linux-based VMware tools (Xubuntu, in my case): 
    sudo apt-get install open-vm-tools
  4. Power Off VM or Reboot VM (if you have the latest VM Hardware version)
  5. Right click on VM and select “Compatibility > Upgrade VM Compatibility” and then “Yes”
  6. Power On VM

References

Update sequence for vSphere 6.0 and its compatible VMware products

ESXi/ESX hosts and compatible virtual machine hardware versions list

vSphere 6.0 Upgrade Guide 

VMware Support For OVT

One-to-one mapping between VMware Tools for ESX/ESXi version-number codes