Performance Overview Development

Quick Links to Topics:

Commvault^® software is a high-performance solution for protecting all data in any environment within defined protection windows. The software also provides many settings to improve performance. Before considering tuning Commvault software, it is important to understand capabilities and limitations of all hardware and software deployed within an environment.

There is no such thing as a static data center. Network infrastructures are constantly changing, new servers are added, mission critical business systems are moving to hybrid cloud, or public cloud infrastructures. Before considering Commvault tunables, it is first important to understand your environment including the capabilities and limitations of the infrastructure; specifically, the ability to transfer large amounts of data of production or backup networks.

When making modifications to an environment, changes that may positively impact one aspect of the environment can negatively affect another aspect. This is also true about Commvault settings. For example, enabling multiplexing when writing to tape drive can improve backup speeds. However, it may have a negative impact on restores if dissimilar data types are multiplexed to the same tape. Another example is using Commvault deduplication and setting a high number of data streams. Since client-side deduplication is being used, there will be a low impact to the network. But if the deduplication database needs to be sealed, the next set of backup operations may result in oversaturating the network while re-baselining blocks in storage.

Performance Benchmarks

Benchmarks can be divided into two kinds, component and system. Component benchmarks measure the performance of specific parts of a process, such as the network, tape or hard disk drive, while system benchmarks typically measure the performance of the entire process end-to-end.

Establishing a benchmark focuses your performance tuning and quantifies the effects of your efforts. Building a benchmark is made up of the following 5 steps:

• Understand the process
• Identify the resources involved
• Minimize outside influence
• Periodic test
• Write it down

Understand the process

You can't document or improve something if you don't know what's going on. More importantly, you need to understand what phases a job goes through and how much each phase affects the overall outcome.
For example, a backup job over a network to a tape library takes two hours to complete. You think it should take a lot less and you spend time, effort, and money to improve your network and tape drives and parallel the movement of data. The job now takes 1.8 hours to complete. You gained a 10% improvement.

Looking at the job in more detail we find that the scan phase of the job is taking 1.5 hours and the rest is the actual data movement. Switching the scan method reduces the scan phase time to 12 minutes. The job now takes .4 hours.  You gained a 78% improvement.
Knowing what phases a job goes through and how much each phase impacts the overall performance can help you focus your time, effort, and money on the real problems.

Identify the resources involved

Each hardware component is going to have a theoretical performance limit and a practical one. Attempting to get improvement beyond these limits without changing the resources involved is a waste of time. Consider using newer vs. older technologies, such as tape drives.

Minimize outside influence

Large data movements are usually done during non-production hours for two reasons – one, they can degrade production work, and two, production work can degrade the movement of data. You want to minimize competition for resources to get a fair benchmark of what performance is achievable. In those cases, where competition cannot be eliminated, you must accept the impact to performance or invest in more resources.

Periodic Test

A single measurement is not a benchmark. Tape devices have burst speeds that are not sustainable over the long run. Networks have various degrees of bandwidth availability over a period of time.  A single snapshot check of bandwidth will not give you a realistic expectation. Do periodic testing over the actual usage of a resource to determine its average performance. Try to level out the peaks and valleys - or at least try to identify what causes these variations.

Multiple measurements scattered over a day can also help in establishing if an unexpected external process is impacting the environment. For example, if you have a database server that is slowly backing up at night, but when you sample during the day, it is achieving expected performances, you can suspect an external process impacting the backup, such as a database administrator dumping the database and copying it to another server at the same time in this example.

Write it down

The hardest lessons are the ones you must learn twice. Once you've established your acceptable and/or expected performance levels for each resource and end-to-end, write them down and use them as the baseline for comparing future performance.

Environment Considerations

Before modifying Commvault^® software settings to improve performance, consider environmental capabilities and limitations. Ensure the environment is optimized to the best of your team's abilities. Commvault software can move data at high rates of speed, but it will ultimately be limited by bottlenecks on servers and network devices.

TCP/IP

TCP/IP is the most common network transmission protocol. Factors that can degrade TCP/IP performance are:

• Latency - Packet retransmissions over distance take longer and negatively impact overall throughput for a transmission path.
• Concurrency - TCP/IP was intended to provide multiple users with a shared transmission media. For a single user, it is an extremely inefficient means to move data.
• Line Quality - Transmission packet sizes are negotiated between sender/receiver based on line quality. A poor line connection can degrade a single link's performance.
• Duplex setting - Automatic detection of connection speed and duplex setting can result in a half-duplex connection. Full duplex is needed for best performance.
• Switches - Each switch in the data path is a potential performance degrader if not properly configured.
• Firewalls – Firewall is the first line of defense against hackers, malware, and viruses. There are hardware firewall appliances and software firewalls, such as operating system firewalls. Firewalls can have minor to moderate impacts on transfer performances.

Vendor documentation for most network interface cards (NIC), switches, and router provide useful information on optimization of their equipment.

SCSI/RAID

SCSI is the most common device protocol used and provides the highest direct connection speed.  An individual SCSI drive's speed is determined by spindle speed, access time, latency, and buffer.  Overall SCSI throughput is also dependent on how many devices are on the controller and in what type of configuration.  The limitation of SCSI is the distance between devices and the number of devices per controller.

• RAID arrays extend the single addressable capacity and random access performance of a set of disks. The fundamental difference between reading and writing under RAID is this: when you write data in a redundant environment, you must access every place where that data is stored; when you read the data back, you only need to read the minimum amount of data necessary to retrieve the actual data--the redundant information does not need to be accessed on a read.  Basically – writes are slower than reads.

• RAID 0 (striping) or RAID 1 (mirror) or RAID 1+0 with narrow striping are the fastest configurations when it comes to sequential write performance. Wider striping is better for concurrent use. A RAID 5 configured array can have poor write performance. The tradeoff in slower write performance is redundancy should a disk fail.

Fine tuning a RAID controller for sequential read/write may be counterproductive to concurrent read/write.  If backup/archive performance is an issue, a compromise must be arranged.

iSCSI/Fibre Channel

iSCSI or Fibre Channel protocol (FCP) is essentially serial SCSI with increased distance and device support. SCSI commands and data are assembled into packets and transmitted to devices where the SCSI command is assembled and executed.  Both protocols are more efficient than TCP/IP.  FCP has slightly better statistics than iSCSI for moving data. Performance tuning is usually setting the correct 'Host Bus Adapter' configuration (as recommended by the vendor for sequential I/O) or hardware mismatch.  Best performance is achieved when the hardware involved is from the same vendor. Given that configuration and hardware is optimum, then for both iSCSI and FCP, performance is inhibited only by available server CPU resources.

Disk I/O

Performing I/O to disks is a slow process because disks are physical devices that require time to move the heads to the correct position on the disk before reading or writing.  This re-positioning of the head is exacerbated by having many files or having fragmented files.  You can significantly improve read performance of the source data by de-fragmenting the data on a regular basis.  

Anti-Virus

Anti-viruses are intelligent software protecting a system against corrupted data by periodically scanning files systems and ensuring that every file accessed or opened by any processes running on the system is a legitimate file (and not a virus). You can easily imagine that when a backup runs and protects every system files, the anti-virus validation significantly decrease backup performances. It might also access and lock Commvault files, such as log files. It is recommended on all systems on which Commvault software is installed, to add exclusions to the anti-virus software for Commvault^® software folders, so that when Commvault related processes are in action, they do not trigger the anti-virus validation process.

Windows Recommended anti-virus exceptions

For V10 and earlier environments, any Content Store directory must be replaced by 'Simpana.' For V11 environments, any Content Store directory must be replaced by 'ContentStore.'

UNIX, Linux and Macintosh Recommended anti-virus exceptions