Commvault

CommServe® Server LiveSync

Owned by Carl Brault (Unlicensed)
Last updated: Nov 30, 2021 by tdopko@commvault.com (Unlicensed)
13 min read

Quick Links to Topics

CommServe® Server LiveSync Overview

The CommServe® server coordinates all operations within a Commvault® environment. The server must be available at all times because it initiates every phase of backup and restore jobs.  However, if a disaster occurs or maintenance is executed on the server, it may prevent jobs from running for a long period of time. To alleviate this situation, use the CommServe® server LiveSync feature. This feature keeps the production server ready for disaster recovery and provides the ability to quickly failover to a designated standby server in the event of a disaster. 

The CommServe® server LiveSync feature leverages the SQL agent installed on both CommServe® servers. It protects the production database at a high frequency and restores it on the standby database. This significantly reduces the downtime in a disaster situation by keeping the target CommServe server in sync with the production one. By initiating a disaster failover, all clients and operations will automatically use the standby CommServe®server to resume as normal. If any maintenance is executed on the production server, a maintenance failover can be initiated to temporarily use the standby CommServe® server. Once completed, operations can fail back to the production server. 

Tip: Multiple servers can be defined as standby CommServe® servers. For instance, one can be used in the same datacenter as the production server for maintenance failovers, and the other can be used as a standby server at a remote site used for disaster failovers.

CommServe® server LiveSync operations


Communication Routing Methods 

To route communications to the proper CommServe® server, the LiveSync feature uses two approaches:

  • A proxy client server
  • Floating DNS

Proxy Client Server

By default, a client server is used as a proxy to route communication to the CommServe® server. The system automatically defines and configures the SQL agents installed on both CommServe® servers to act as a proxy. However, if another proxy client already exists in the environment, it can be used to replace the SQL agents. This method does not require DNS configuration and will automatically reroute traffic as needed.

Floating DNS

This method does not require any client proxy to route communications. Instead, a floating DNS entry is created that points to the active CommServe® server. If a failover occurs, that DNS entry is modified and clients are redirected to the new active server. 

Note: For existing environments, using the default proxy method is recommended. As for the floating DNS method, it requires installing the CommServe® server and client software with the floating DNS name, and therefore it is easier to implement in a newer installation.




CommServe® Server LiveSync Deployment

Deploying LiveSync for the CommServe® server requires several steps to properly implement and efficiently protect and replicate the production database to the target standby server. Whether it is a new Commvault® environment or an existing one, the steps must be followed in the exact order.

CommServe Server LiveSync Deployment steps:

Note: If LiveSync is deployed in an existing environment, skip steps 1-3.

  1. Install the CommServe component on the production server (if it is a new environment)

  2. Install a MediaAgent server (if it is a new environment)

  3. Configure a library (if it is a new environment)

  4. Install the CommServe Failover package on the production CommServe® server

  5. Install the CommServe® server package on the standby server

  6. Install the CommServe Failover package on the standby server

  7. Enable LiveSync using the two configured servers

Install the CommServe Failover Package on the Production CommServe® Server

If configuring LiveSync in a new environment, it is important to first install the CommServe® server, at least one MediaAgent, and configure at least one library. This is crucial as the LiveSync process backs up the CommServe® server database to a library. The backup is then restored in the standby server's database. This backup/restore operation is executed every 15 minutes by default. Once these components are installed and configured, the CommServe Failover package can be installed on the production CommServe® server. The installation configures all the required components, such as an unconfigured File System agent (used for communication), and the SQL agent (used to backup the CommServe® server database). These agents are installed in a different instance than the CommServe® server component software.

Commvault® Software Instances on a CommServe Server using LiveSync:  

  • Instance001 - used for the CommServe® server component and any other backup agents previously installed

  • Instance002 - used for the unconfigured File System agent and the SQL agent used by LiveSync

Note: During the failover package configuration, it is important to provide a descriptive client name that clearly identifies the instance used by the LiveSync feature.




To install the CommServe Failover package

1 - Right-click Setup.exe | Run as administrator.

2 - Choose the installation language.



3 - Accept the license agreement.

4 - Select to install packages on this computer.

5 - Select the CommServe Failover package.

6 - Provide the install path for the new instance.



7 - Skip this step since it is on the same server as the CommServe® server.

8 - Provide a client name for the SQL agent instance.

9 - Provide the host name to use.

10 - Provide the CommServe® server host name.



11 - If the environment is using a certificate, provide the certificate file by browsing and selecting it.

12 - Provide a username with permissions to install components in the Commvault® environment.

13 - Skip this window.

14 - Click to complete the installation.


Install the CommServe® Server Component on the Standby Server

Once the installation on the production CommServe® server is complete, the configuration of the standby server can begin. The first software component to install on the server is the CommServe® server package. This installs the CommServe® server and SQL instances. It also creates a blank database in which the production database will be replicated. As with the production server, the CommServe® server software will be installed and configured in Instance001.




To install the CommServe® Server package

1 - Right-click Setup.exe | Run as administrator.

2 - Choose the installation language.



3 - Accept the license agreement.

4 - Select to install packages on this computer.

5 - Select the CommServe® server package which automatically selects all required packages.



6 - Define the install path for the CommServe® server software.

7 - Define the index directory path for the MediaAgent installed along with the CommServe server component.

8 - Define the install path for the SQL instance.

9 - Define the path for the CommServe® database.



10 - Provide a path for the CommServe server DR backups.

11 - Skip this if you are not using OnePass for Exchange.

12 - This window sets the CommServe® server client name and hostname to use.



13 - Windows firewall can be disabled automatically or select No to configure firewall exceptions for Commvault software.

14 - Create a new empty CommServe® database.

15 - Provide a user account to register the Commvault environment.

16 - Click to complete the installation.


Install CommServe Failover Package on the Standby Server

The next step is to install the CommServe® Failover package on the standby CommServe® server. The installation follows the exact same steps as the production CommServe® server. The only difference is that although we install on the standby CommServe, the CommServe server name provided in the installation screen must be the production CommServe® server name.




CommServe Information screen

1 - On the CommServe Information screen, ensure to enter the name of the production CommServe® Server.


Enable CommServe® Server Live Sync

Once all required components are installed, it is now time to enable LiveSync between the two CommServe® servers. This automatically enables protection for the production database every 15 minutes. It also configures the recovery of that data on the destination database following the same frequency. Both operations are controlled by built-in schedules and schedule policies.

When enabling LiveSync, a storage policy must be defined to store the LiveSync SQL agent backups. By default, the CommServeDR storage policy, which is automatically created when configuring the first library, is used as a target for the backups. But a different storage policy can be used if needed. However, the selected storage policy MUST follow these rules:

  • It must be a dedicated policy (not mixed with other backup data)

  • It must be non-deduplicated

Any storage policy meeting these rules can be selected from the drop-down list.




To enable CommServe® Server LiveSync

1 - From the Home tab | Control Panel.

2 - Click the CommServe LiveSync applet.

3 - Check to enable LiveSync.

4 - Validate the CommServe® server list and the status.

5 - Define the storage policy to use to store the LiveSync SQL agent backups.



6 - From the Connectivity tab, select the communication routing method to use during a failover.

7 - From the Synchronization schedule tab, define the replication frequency.

8 - Click OK to apply the configuration settings.



Validating the LiveSync Configuration

It is important to validate the configuration of the CommServe® LiveSync feature. First, ensure that the built-in schedules are created. Then, execute the schedules and validate that the two nodes are synchronized. Once synchronized, any type of failover can be initiated as needed.

Validate the LiveSync SQL Agent Backup Schedule

A pre-configured SQL agent protects the production CommServe® server database. This agent protects all SQL databases configured on the CommServe® server and stores the data in the CommServeDR storage policy. A built-in schedule policy called 'System Created for Failover' controls the execution of the backup jobs. This schedule policy is made of the following three schedule patterns:

  • Full backup - Every Sunday at midnight
  • Differential backup - Every day at midnight except for Sunday
  • Transaction Log backup - Every 15 minutes 

Note: Although these schedules can be modified, it is recommended to leave as pre-configured.

Tip! After validating the schedule policy, run the first full backup immediately to synchronize both nodes as soon as possible. To do so, simply right-click the schedule policy and select 'Run Immediately.' Then select the full backup schedule to launch it.




To validate the CommServe® Server LiveSync backup schedule policy

1 - Expand Policies | Click Schedule Policies | Right-click System Created for Failover | Edit.

2 - The three built-in schedules are listed in the window.


Validate the LiveSync Replication Schedule

Once the production database is protected, the data must be replicated (restored) on the destination database (standby server). This operation is automated and controlled by a schedule called 'Live Sync Schedule for Failover Node.' This schedule restores data as soon as it is backed up from the source CommServe® server. By default, this schedule synchronizes all SQL databases hosted on the source server. 

Note: Although the database selection can be modified, it is recommended to select all databases.

Tip! Ensure that the first backup is complete, then run the replication schedule immediately by right-clicking it and selecting 'Run Immediately.'




To validate the CommServe® Server LiveSync replication schedule

1 - Right-click CommServe | View | Schedules.



2 - Right-click the schedule and click Edit.

3 - From the Schedule Pattern tab, define the frequency at which backups will be restored on the target CommServe® Server.



4 - By default, all databases are selected for replication.

5 - Database size and path information is listed on the Redirect tab.


Validate the Status of the Nodes

The node status can be monitored using the LiveSync instance Process Manager on the standby node. To ensure the right process manager is selected, validate the services running in the instance. The LiveSync instance has three services running that are displayed from the Services tab:

  • Commvault Communication Service
  • Commvault Client Manager Service
  • Commvault Network Daemon

Select the Failover Assistant tab and see the status of the nodes. The status is displayed by a colored circle next to the node's name. Green indicates that the node is up-and-running.




To validate the CommServe® Server LiveSync node status

1 - Locate and click the LiveSync Commvault Process Manager.

2 - Validate this is the LiveSync instance by ensuring only three services are running.



3 - A colored circle next to the node’s name indicate its status.



Perform a Failover

Performing a failover is the action of moving activity from the production to the standby node, or vice-versa (from the standby to the production node). There are different types of failovers that can be executed.

CommServe® server LiveSync failover types:

  • Test failover
  • Production failover (planned or unplanned)
  • Production Maintenance failover

It is important to understand the difference between each failover as it dictates the behavior of client communication, re-configuration, as well as the activity control.

Test Failover

Test failover is considered a non-disruptive failover that does not impact the production environment. It only validates if activity CAN be failed over to the standby node and if the CommServe® services have started. It WILL NOT re-configure client communication nor move the workload to the standby node. This is useful to test failover right after the initial deployment.

Production Failover

A production failover is considered a read/write failover. This means that upon a failover, not only will client communication be redirected to the standby node, but job activity is fully enabled for the standby® CommServe server. In other words, the entire workload is literally moved over to and handled by the standby server while any type of jobs can be executed (backup, restore, data aging, etc.). 

There are two types of production failover; planned or unplanned. The difference between the two types is how the activity is handled on the source server.

  • An unplanned failover is mainly in a disaster situation where the source CommServe® server is unavailable. Therefore, no reconfigurations of any sort can be achieved. The workload is simply failed over to the standby node and clients are reconfigured to communicate with it.

  • A planned failover is a situation where the source server is still available. In this case, the workload is failed over to the standby node and client communication is redirected to it. In addition, the source server's Commvault® services are stopped and disabled. This ensures that they are not inadvertently started, leading to a split-brain scenario, where two different CommServe® server instances are trying to interact with the Commvault environment.

Production Maintenance Failover

A production maintenance failover is considered a read-only. This means that during a failover, client communications are redirected to the standby node, but only recovery activity is allowed from the standby node. Basically, this node is used to install updates (firmware, OS, Commvault® software), so you can conduct restores or recall archived files during the maintenance. However, no other jobs (backup, data aging, etc.) can run. This is also the case for the production failover, during a maintenance failover, services are stopped on the source CommServe® server and are disabled to avoid inadvertently starting them.

Perform a Production Maintenance Failover

To install any updates, such as firmware, OS system updates, or Commvault® software, on a CommServe® server LiveSync environment, it is important to complete the steps in the exact order.

Steps to apply updates on a CommServe® server LiveSync environment

  1. Failover to the standby node
  2. Apply updates from the command line to the production server
  3. Reset the failover (failback to the production node)
  4. Apply updates from the command line to the standby node

Failover to the Standby Node

Failover to the standby node is initiated from the LiveSync instance Process Manager of the standby node. Select the type of failover to execute, as well as the standby node to failover to. In the case of a maintenance failover, select the Production Maintenance type. Ensure that the target node is the standby CommServe® server.




To execute a CommServe® Server Failover

1 - Locate and click the LiveSync Commvault Process Manager.

2 - Select the failover type to execute.

3 - Click to initiate the failover to the standby node.



4 - Type to confirm the failover and click OK.

5 - Information about the failover phase is provided in the status window.

6 - Once the failover is completed, click OK.



7 - To validate the failover, connect to the console using the standby node URL, and select the standby server from the list.



8 - From the CommCell® browser, validate that the database was restored.


Apply Updates to the Production CommServe® Server

Once operations have failed over to the standby node, updates can now be applied to the production node. First, apply any firmware and/or operating system updates to the system, then apply Commvault® software updates. The updates must be installed at the command line to prevent services from starting. To install the updates, copy the software on the server and run the following command syntax from the command line:

setup.exe /installupdates /silent /nostartsvc /instance Instance001

Once the updates are applied, continue to reset the failover.

Reset the Failover

Resetting a failover means to fail operations back to the production server. To achieve this task, simply conduct a traditional failover, execute it from the production server process manager, and select that same server as the failover destination. This reverts activity control and redirects all client communications to it. Once done, apply updates to the standby node.

Apply Updates to the Standby Node

The last step is to apply updates to the CommServe software on the standby server. As with the production server, it is recommended to install the updates from the command line to prevent services from starting.



Copyright © 2021 Commvault | All Rights Reserved.