Virtual Server Agent Proxy Roles

VSA Proxy Roles Overview

Virtual Server Agent (VSA) proxies are defined at the instance level of the VSA pseudo client. The top listed VSA proxy is designated as the coordinator and all other proxies are designated as data movers. Note that the Coordinator proxy also acts as a data mover. The coordinator is responsible for communicating with the hypervisor to get information about VMs and distribute VM backups to data mover proxies. Data mover proxies communicate with the coordinator proxy and provide information on available resources and job status. In the event that the coordinator proxy is unavailable, the next proxy in the list assumes the role of coordinator. If a data mover proxy becomes unavailable, the coordinator proxy assigns jobs to other available proxies.

Virtual machine distribution and backup process

Virtual Machine Distribution Process

When a VSA subclient backup starts, the coordinator receives a list of all virtual machines listed in the subclient. Based on a defined set of rules, the coordinator creates a dynamic VM queue to determine the order in which virtual machines will be protected and which VSA proxies will back up each virtual machine.

Subclient Data Readers

The data readers setting in the advanced tab of the subclient defines the maximum number of streams used for the backup. When the job starts, if there are more VMs than available streams, each VM is allocated a single stream. If there are more streams than VMs, the coordinator automatically instructs the data mover proxy to use multiple streams for the VM backups. Depending on the number of available streams, each virtual disk in the VM is backed up as a single stream. This process is dynamic so as a job progresses, and more streams become available and less VMs require protection, multiple streams can be used to protect individual VMs.

Stream allocation when there are more VMs than data readers

Stream allocation when there are more data readers than VMs

Datastore Distribution

If VMs within a subclient exist across multiple Datastores, the coordinator assigns VMs to proxies, one VM per Datastore until the maximum stream count is reached. Each VM is assigned to a different data mover proxy, balancing stream loads across proxies based on proxy resources. This distributes the load across multiple Datastores, which improves backup performance and maintain a healthy Datastore state. In addition to the subclient Data Readers setting, a hard limit can be set for the maximum number of concurrent VMs that can be protected within a single Datastore using the nVolumeActivityLimit additional setting.

Datastore stream allocation

VSA Proxies

Commvault^® software uses VSA proxies to facilitate the movement of virtual machine data during backup and recovery operations. The VSA proxies are identified in the instance properties. For Microsoft Hyper-V, each VSA proxy is designated to protect virtual machines hosted on the physical Hyper-V server. For VMware, the VSA proxies is used as a pooled resource. This means that depending on resource availability different proxies may be used to backup VSA subclients each time a job runs. This method of backing up virtual machines provides for higher scalability and resiliency.

VSA Proxy placement for VMware^® and Hyper-V^®

VM and VSA Proxy Distribution Rules

Datastore distribution is the primary rule that determines the order in which VMs are backed up. Additional rules that determine VM backup order are:

1. Number of proxies available to back up a VM – The fewer proxies available, the higher in the queue the VM is. This also is dependent on transport mode. If the transport mode is set to Auto (default), SAN have highest priority, followed by HotAdd and then NDB mode. If a specific transport mode is defined in the subclient, only proxies that can protect the VM can be used – this could affect the available number of proxies which could result in a higher queue priority.
2. Number of virtual disks – VMs with more virtual disks are higher in the queue.
3. Size of virtual machine – Larger VMs are higher in the queue.

Stream Allocation and Proxy Throttling

During backup operations, the coordinator proxy gathers information on each data mover proxy to determine the default maximum stream count each proxy can handle. This is based on the following:

• 10 streams per CPU
• 1 stream per 100 MB available RAM

When the coordinator assigns jobs to the data mover proxies, it evenly distributes jobs until the default maximum number of streams on a proxy is reached. Once the threshold is reached it no longer assigns additional jobs to the proxy. If all proxies are handling the maximum number of streams and there are still streams available, the coordinator assigns additional jobs to proxies using a round-robin method.

Throttling can be hard set on a per proxy basis using the following registry keys:

• nStreamsPerCPU – limits the number of streams per CPU on the proxy
• nMemoryMBPerStream – sets the required memory on the proxy for each stream
• nStreamLimit – sets a limit on the total number of streams for a proxy
• bHardStreamLimit – sets a hard stream limit across all proxies within the VSA instance

Disable Dynamic Assignment

To disable dynamic VM assignment and force static assignment, configure the DynamicVMAssignmentAllowed additional setting on the coordinator (and failover candidates) and set the value to 'false.'  If this additional setting is configured with a value of 'true,' dynamic VM assignment is allowed, but is not forced. When dynamic VM assignment is disabled, virtual machines are assigned to proxies at the beginning of the job and assignments are not modified during the job.

Hyper-V Dynamic Distribution

There are several differences in the dynamic distribution of VMs in a Hyper-V environment:

• Cluster Shared Volume (CSV) owner – VMs are protected based on the VSA proxy that owns the cluster.
• Cluster – If CSV owner is not in the proxy list, VMs are dispatched to any node in the cluster.
• Host – When the hypervisor host is a VSA proxy and in the proxy list, the host VSA proxy is used.
• Any Proxy – If the hypervisor host is not a proxy or not in the list, VMs are distributed to any available proxy.

VSA Coordinator or Proxy Failover

If a VSA proxy protecting a virtual machine goes offline, VMs are returned to the priority queue. The next available proxy is assigned to the re-queued VM.

If the VSA coordinator proxy goes offline, VSA backup jobs managed by the coordinator are placed in a pending state. The next proxy in the list assumes the role of the active coordinator proxy and jobs will return to a running state. Any VMs that were in process of being protected are re-queued and restarted.

Stream Management and VSA

Using the VSA coordinator and data mover proxies, along with the intelligent load distribution, the number of streams that can be used to protect VMs can be set higher than in previous software versions. However, it is still important to understand possible negative consequences of misconfigurations.

The data readers setting in the subclient is the primary governor to determine the maximum number of VMs or virtual disks that can be protected at a given time. The load distribution attempts to balance VM backups across disk volumes. However, if the VMs requiring protection reside on only a few volumes and the data readers is set too high, problems can occur.

When a VM backup runs, a software snapshot is taken where block changes are cached, the frozen disk data is read from the volume, and normal I/O still occurs on the volume. With these three actions occurring simultaneously, if there are too many snap and backup operations occurring, significant performance degradation can occur. This can also cause major issues during snapshot cleanup operations.

As a general rule of thumb, each disk volume should have two concurrent snap and backup operations as a starting point. This number may vary greatly based on whether the disks are SAN or NAS, the size of the disks, and the performance characteristics. Consider the significant performance difference between spinning versus solid state disks. The two data readers per disk volume is a starting point. Adjust these numbers to a point where backup windows are being met. Also, consider mapping subclients to specific disk volumes and adjusting the data readers based on the performance characteristics of the disks.