Disk Libraries - CommCell® Console

Storage Configuration Overview

Commvault^® software logically addresses storage systems to allow virtually any library type to be used. The three primary library types are disk, tape, and cloud.  

Disk Libraries

A disk library is a logical container which is used to define one or more paths to storage called mount paths. These paths are defined explicitly to the location of the storage as a drive letter or a UNC path. Within each mount path, writers are allocated which defines the number of concurrent streams for the mount path.

There are three primary types of disk libraries:

• Dedicated – disk libraries are created by first adding a disk library entity to the MediaAgent using either the right-click All Tasks menu or the Control Panel's Expert Storage Configuration tool. One or more mount paths can be created/added to the library. Mount Paths are configured as Shared Disk Devices. The Shared Disk Device in a dedicated disk library has only one Primary Sharing Folder.
• Shared – disk libraries are libraries with more than one Primary Sharing Folder configured on a Shared Disk Device. This enables other MediaAgents access to the same shared volume resource. A shared disk library can then be created and the 'Shared Disk Devices' added to the library. One path to the shared folder can be direct while the others are Common Internet File System (CIFS) shared directory paths. CIFS protocol is used to manage multiple MediaAgent access to the same directory. For UNIX hosted MediaAgents, Network File System (NFS) protocol can be used. NFS shared disks appear to the MediaAgent as local drives.
• Replicated – disk libraries are configured like a shared disk library with the exception that the Shared Disk Device has a replicated data path defined to a volume accessible via another MediaAgent. Replicated folders are read-only and replication can be configured for use with third party replication hardware.

There are three methods that disk library data paths can be configured:

• Network Attached Storage or NAS
• Storage Area Network or SAN
• Direct Attached Storage or DAS

Disk libraries best practices:

• If using DAS or SAN, format mount paths using a 64KB block size.
• If using DAS or SAN, try to create multiple mount path. For instance, if there are 10 mount paths, and there is a maintenance job, such as a defrag job running on one, the mount path can be set to read-only, leaving 90% of the disk library available for backup jobs.
• Set mount path usage to Spill and Fill, even if using only one mount path. If additional mount paths are added later, the streams will spill as expected.
• Share the disk library if required.
• From the CommCell^® console, validate the mount path speed and document for future reference.

The following explanations assume Commvault deduplication is being used.

Network-Attached Storage (NAS)

Network-Attached Storage provides the best connection method from a resiliency standpoint since the storage is accessed directly through the NAS device. This means that by using a Common Internet File System (CIFS) or a Network File System (NFS), Universal Naming Convention (UNC) paths can be configured to read and write directly to storage. In this case, the library can be configured as a shared library, where all MediaAgents can see stored data for data protection and recovery operations.

Disk library using Network Attached Storage (NAS)

Storage Area Network (SAN)

Storage Area Networks or SANs are very common in many data centers. SAN storage can be zoned and presented to MediaAgents using either Fibre Chanel or iSCSI. In this case, the zoned storage is presented directly to the MediaAgent providing Read / Write access to the disks.

When using SAN storage, each building block should use a dedicated MediaAgent, DDB and disk library. Although the backend disk storage in the SAN can reside on the same disk array, it should be configured in the Commvault^® software as two separate libraries; where Logical unit numbers (LUNs) are presented as mount paths in dedicated libraries for specific MediaAgents.

SAN storage provides fast and efficient movement of data but, if the building block MediaAgent fails, data cannot be restored. When using SAN storage, either the MediaAgent can be rebuilt or the disk library can be re-zoned to a different MediaAgent. If the disk library is rezoned, it must be reconfigured in the Commvault^® software to the MediaAgent that has access to the LUN.

Disk library using Storage Area Network (SAN)

Direct Attached Storage (DAS)

Direct-attached storage is when the disk library is physically attached to the MediaAgent. In this case, each building block is completely self-contained. This provides for high performance but does not provide resiliency. If the MediaAgent controlling the building block fails, data stored in the disk library cannot be recovered until the MediaAgent is repaired or replaced. Keep in mind that, in this case, all the data in the disk library is still completely indexed and recoverable, even if the index directory is lost. Once the MediaAgent is rebuilt, data from the disk library can be restored.

Disk library using Direct Attached Storage (DAS)

Add a Disk Library

Before adding a disk library, the operating system must have access to the storage. For SAN or DAS storage, volumes must be created and formatted in the operating system. For NAS, try to access the storage CIFS shares using the credentials.

Once the operating system has access to the storage, the library can be created.

Add Mount Paths to the Library

Creating the library also defines the first mount path. Any additional mount paths then need to be defined. Mount paths can also be added to a saturated library to expand the available space.

Share the Disk Library

If the disk library is using NAS storage, it can be shared with other MediaAgents. Sharing a disk library is achieved by sharing its mount paths. Apply the sharing procedure on each mount path.

Disk Library and Mount Path Configuration Options

Several options can be configured for the disk library and its mount paths. They are configured in the Properties pages of the library or any mount paths.

Disk Library Properties

Setting low disk space watermark

Disk library Properties | General tab

The 'Disk Library Low Watermark' is configured with a percentage threshold that reports an event to the Event Viewer when total disk capacity for all mount paths reaches or falls below the specified percentage. Alerts can also be configured to notify users when the threshold has been reached.

Instead of a low watermark, an estimated number of days before running out of space can be used. Based on the daily usage history, the system estimates when it expects the library to be saturated. When the number is equal or lower to the defined value, an event is sent in the Event Viewer.

Prevent disk content from being deleted through Windows Explorer

Disk library Properties | Mount Paths tab

The 'Prevent accidental deletion of data from mount paths' option locks disk mount paths content from being deleted for Windows MediaAgents. It is strongly recommended that this setting is selected. When it is enabled, and a user connects to the MediaAgent to delete data using Windows Explorer, the operating system displays an access denied error message.

Determine the order multiple mount paths are used

Disk library Properties | Mount Paths tab

The 'Mount Path Usage' section determines the order in which incoming client streams are distributed across mount paths, when multiple mount paths are configured for a disk library.

• Fill & Spill - The default setting for mount path usage, uses mount paths in the order in which they were created.
• Spill & Fill (load balancing) - Load balances device streams between all mount paths in the library.

Which should you use for best performance? For disk libraries, part of a bottleneck are the disks themselves, but the other part of the bottleneck is the I/O path to the disks. The use of Fill & Spill or Spill & Fill (load balancing) should be determined by how the mount paths were created and the I/O path to each mount path.

• Use Fill & Spill if the mount paths are part of the same disk array and there is a single I/O path to the mount paths.
• Use Spill & Fill if the mount paths are on separate disks and/or if there are separate I/O paths to the mount paths.

Tip: Spill & Fill for a Single Mount Path Library

If there is only one mount path in the library, consider setting mount path usage to Spill & Fill. The reason for this is if an additional disk array is later added, the disk library will automatically use the mount paths in a load balancing manner. A common misconfiguration is when disks are added to a library and the administrator or engineer forgets to set this option.

Disk space thresholds for managed disk space

Disk library Properties | Mount Paths tab

Managed disk space is a Commvault^® software feature that allows a job stored on disk to be retained beyond its standard retention settings if free space exists. Managed disk space thresholds do not apply to disk libraries using Commvault deduplication.

• Start Aging when data occupied on disk is (High Watermark) - determines the amount of disk space consumed before aged jobs will be pruned.
• Stop Aging when data occupied on disk is (Low Watermark) – stops purging jobs when amount of disk space consumed falls to the specified value.

Hide a decommissioned disk library

If a library is not to be used anymore for new backups, it does not mean that it can be deleted from the CommCell^® console. Doing so would result in destroying the data that might still be in the library and required for restores. Leaving the library in the CommCell console, however, could potentially lead to an administrator creating a new storage policy pointing to that library. If you want to prevent the administrator from using the disk library for any new storage policies and completely hide it in the console, the 'Hide Library' option can be used. Once hidden, if it is required to display the library, such as to later deleting it, the option to un-hide libraries is available from the Media Management applet.

Disk Library Associated Storage Policies

The Associations tab of a disk library displays all storage policies that have a copy sending data to the disk library. This is useful information when you plan on decommissioning a disk library or taking it offline for maintenance.

Disk Library Usage

The disk library usage tab provides several useful information about the disk library usage. It displays the space usage, deduplication space saving and data aging pruning information.

Keeping an eye on this tab helps in avoiding filled library situations and helps in planning the acquisition or replacement of hardware. Based on the average daily disk space consumption and the average space freed up by data aging, the system can estimate a possible date on which the library is expected to be filled.

Mount Path Properties

Determine the maximum concurrent writer streams for a mount path

Mount Path Properties | Allocation Policy tab
Mount Path Allocation determines the number of concurrent write operations that are performed to a mount path. The default is set to 'Maximum allowed.' If a defined number is set, increasing this number could potentially improve overall backup performance within the CommCell^® environment by allowing more jobs to run concurrently. However, setting this number too high could have a negative impact on the MediaAgent, network, and disk I/O performance.

Before modifying this setting consider the following key points:

• Increasing this setting will run more streams through the network, MediaAgent, and disk. For many environments increasing this number marginally may not have a negative impact on the overall environment. It is recommended to incrementally adjust this number higher (increments of three to five at a time) and carefully monitor the network, MediaAgent and disk performance.
• If multiple mount paths exist in the library, consider how the mount paths have been carved out of the disks. If two mount paths set to five writers are allocated from the same physical disk array then, there would be ten concurrent write operations to the disks, five from each mount path.
• Consider the I/O path from the MediaAgent to the disks as a potential bottleneck. Depending on the disk storage being used (Fibre, iSCSI or NAS) the I/O paths can become a bottleneck.
• Increasing the number of writers could have a positive impact on overall backup performance within the CommCell console by running more jobs but may slow down individual jobs due to the extra network and disk load.

Determine the space allocation for a mount path

Mount Path Properties | Allocation Policy tab

Space Allocation specifies hard cut-off limits for a mount path. By default, it is set to reserve 2GB of disk space. If more reserve space is needed, increase this setting or use the 'Do not consume more than' setting to limit how much disk space will be allocated to write operations. This can be useful if you require reserving more space for maintenance purposes, such as a defrag operation.

Validate a Mount Path

Mount path validation tests throughput speeds of library mount paths.

The validation operation is configured using the following parameters:

• MediaAgent
• File size
• Number of writers
• Block size
• Number of files

Move a Disk Library Mount Path

A disk library mount path can be moved to a different location using a 'Move Data Path' operation. This fully automated functionality is used when a disk library mount path is lacking space and needs to be moved on a bigger storage volume, or when an older storage unit is replaced.

The new storage must first be presented to the MediaAgent operating system, and then the 'Move Data Path' operation can be initiated.

Disk Library Mount Path Status

The status of each mount path of a disk library can easily be validated from the CommCell browser. Select the library and the view displays each mount path, their status such as 'ready' or 'offline' and the free space left.

Disk Library Maintenance

Disk library data is written in chunk files. When using Commvault^® deduplication, every written block that makes a chunk file is unique. When subsequently encountered in backup jobs, similar blocks are discarded, and references are made to the same block. When all jobs age and this block is no longer referenced to by any other jobs, it is marked to be purged. If the block is at the trail of a chunk file, it gets deleted and the chunk file is truncated. If it is in the middle of a chunk, and the library is hosted on DAS or SAN storage or NAS storage supporting the sparse file attribute, the block gets deleted and a hole is 'punched' in the chunk file.

This can lead to fragmentation over time, which could start degrading performances at some point. In this case, the mount paths can be defragmented using any defragmentation tool supporting the sparse file attribute. Windows built-in defragmentation tool does not support the sparse file attribute and therefore, cannot be used. Commvault does not support calls on any Third-party defragmentation tools. However, Disk keeper has certified its 2011 (and higher) release version for online volume defragmentation.

Before running any defragmentation jobs on a mount path, a disk library maintenance job and a report can be generated to provide information about fragmentation. Based on the result displayed in the report, the Commvault administrator can easily establish if maintenance is required.

Typical disk library maintenance procedure

1. Run a disk library maintenance analysis job to capture fragmentation information.
2. Execute a Library and Drive configuration report.
3. Disable the disk library or put the mount path in read-only.
4. Run the defragmentation tool on the mount path.

Disk Library Maintenance Job

Before defragmenting mount paths, you first need to establish if the mount paths require defragmentation. It is accomplished by first running a disk library maintenance job that captures the status of fragmentation for every mount path of a disk library. A report can subsequently be executed to display the result of the Disk Library Maintenance job.

The Disk Library Maintenance job can be scheduled from the Job Initiation tab.

Commvault recommends running the Disk Library Maintenance Job and the Library and Drive Configuration report once or twice a year.

Library and Drive Configuration Report

Once the Disk Library Maintenance job is completed, the Library and Drive Configuration report can be executed to display the result. Each mount path is displayed, and fragmentation related information is displayed in columns in the far right of the report. The report could be scheduled to run after the Disk Library Maintenance job schedule.

Disable Disk Library or Mount Path

Once the report indicates that a high fragmentation level has been reached, it is recommended to run a defragmentation tool. But before executing it, it is important to ensure that no backups are running, and then either disable the disk library or set a mount path as read-only.

Setting a mount path as read-only has an advantage over disabling the entire library. Restores can still be achieved and if there are enough additional mount paths, backup jobs can still run using the remaining mount paths.

Run the Defragmentation Tool

Run the defragmentation tool on the mount path. Consult your software vendor documentation, to ensure to use a tool that supports online defragmentation and that supports the sparse file attribute. Once the defragmentation job completes, do not forget to put the library back online or to put the mount path back to read-write.

Data Server

The Data Server feature allows the sharing of block-based storage among multiple MediaAgents. It also addresses the traditional limitation, which prevents sharing disk libraries between Linux and Windows^® MediaAgents. Security is increased since the Data Server uses a local service account to access the storage and then presents it to other MediaAgents on an as-needed basis.

When configuring the Data Server feature, there are three types of connections to storage/MediaAgent:

• Data Server IP - A MediaAgent presents local storage to other MediaAgents through the IP network as an NFS volume.
• Data Server SAN - A Linux MediaAgent acts as a proxy to present storage to other MediaAgents using Fibre Channel connections.
• Data Server iSCSI - A Linux MediaAgent acts as a proxy to present storage to other MediaAgents using iSCSI connections.

DataServer IP

DataServer IP provides a simple method to create shared libraries. Each MediaAgent will manage local storage and can access other MediaAgent mount paths in the shared library on demand. This provides a simple method to create large shared libraries and avoids access permission issues since it uses the local service account instead of UNC mount paths and user entered credentials.

How Data Server IP Works

When a mount path is required by another MediaAgent than the one hosting the mount path, such as for operation, it is mounted on-demand using a technology called '3dfs' server that presents the mount path as an NFS share to the target MediaAgent. The target MediaAgent can then have access and read any required blocks.

When setting up such a library, the MediaAgent hosting the library acts as the 3dfs server, which in Commvault^® software is called the DataServer IP.

In a Data Server IP environment, firewalls must be opened for some specific ports. Port requirements are as follows:

When the disk library is shared, you can create storage policies. Ensure that the DataServer IP is selected as the default data path.

By default, the mount path is shared with 'Read/Write' access and therefore is used for backups and restores. However, you can select the 'Read Only' access type from the list to use the mount path only for restores by the remote MediaAgent.

Data Server SAN and Data Server iSCSI

Data Server SAN and Data Server iSCSI allow multiple MediaAgents to connect to storage through a Linux proxy MediaAgent using Fibre Channel or iSCSI connections. The Linux MediaAgent can host direct-attached disk storage which is then presented to other MediaAgents as a shared disk library. This allows multiple clients / MediaAgents to backup over the SAN or iSCSI connection to a single shared disk library instead of each client / MediaAgent using a dedicated library.

When using the Data Server San or iSCSI and Commvault deduplication, better deduplication ratios can be achieved. SAN Data Server is best suited for protecting clustered applications or when clients / MediaAgents have SAN connectivity and it's desirable to avoid using the IP network to conduct backup operations. It also has the advantage of allowing to share the block-based storage between Linux and Windows MediaAgents.

Disk library using a SAN Data Server

Sharing a Mount Path using Data Server

To share a library using a Data Server IP, first create a traditional disk library along with mount paths. Then share the mount paths using the Data Server transport mode.