File System Configuration - CommCell® Console

Agents, Backup Sets, and Subclients Overview

Commvault^® software uses agents to provide application-aware data protection using native Application Program Interfaces (APIs) and scripting methods. An example of API-based data protection is using Windows file system APIs to protect files on a Windows server. An example of scripting is using Oracle's Recovery Manager (RMAN) scripts to protect Oracle databases. The agent communicates directly with the file system or application requiring protection. Within an agent, a backup set is used as a primary container, which manages all data the agent is responsible to protect. Within the backup set, one or more sub-containers, called subclients are used to explicitly define data requiring protection and to define specific rules for how the data is protected.

Agent, backup set, and subclient high-level concept

File System Agents

File system agents provide the fundamental data protection strategy in all data centers. File agents are supported for all major operating systems and include inherent file system capabilities based on the operating system being protected. An example is the ability to use Volume Shadow Services (VSS) to quiesce Windows operating systems.

Backup Sets

A backup set is used as a primary container for all data managed by the agent. This creates a hierarchical structure allowing great flexibility in how specific data is protected. Along with defining subclients within the backup set to define specific management rules, multiple backup sets can be created to provide configurations for special use cases. One example is using two backup sets, one for streaming backups, and the other for hardware snapshot operations. Using two backup sets provides a simplified administration by splitting subclients that manage data for streaming backups and subclients where the IntelliSnap^® feature is enabled.

Each backup set represents all data managed by the agent. This means that creating multiple backup sets creates the potential for protecting data redundantly from the client. It is critical to only use multiple backup sets in special situations to avoid redundant data protection. Storage policies and policy copies provide the ability to create multiple copies of data in backend storage.

Two backup sets managing streaming backups and IntelliSnap^® operations

Subclients

Subclients are used to define data that is protected in a containerized format. Each subclient container manages specific content within a backup set. Each backup set has one or more subclients.

Key points for subclients:

• Subclient contents are defined as drives, folders, files or UNC paths.
• A storage policy is defined to manage the subclient.
• Scripts are inserted prior/after the scan and prior/after backups.
• Filter settings are configured for global and local filters.
• IntelliSnap^® technology is enabled and storage array managing subclient data is defined.
• Data transfer options including compression, deduplication, and encryption are configured.

Example of subclient design

Default Subclient

By default, most agents have a default subclient. During the initial installation of the agent software, an option to associate agent data with a storage policy is provided. This determines the storage policy that manages the default subclient data. All subclients must be associated with a storage policy to protect the data.

The default subclient acts as a catch-all for all data managed within a backup set. This means the default subclient automatically detects and protects all data the agent is responsible to protect. When custom subclients are defined, any data managed by the custom subclients is automatically excluded from the default subclient. This means that the data is mutually exclusive to the subclient in which it is defined and cannot be defined in multiple subclients within the backup set The concept of Commvault^® software is to 'Copy Once and Reuse Extensively (CORE).' In other words, protect the data to the storage policy and use secondary copies to create additional copies of data. There are situations where protecting data from the source location multiple times may be required. To accomplish this, you can create additional backup sets.

Tip: Mutual Exclusiveness of Subclients
Early in version 11, the mechanism of mutual exclusiveness of content was taken out of the product, allowing duplicate content in different subclients within the same backup set to be defined. However, this mechanism has been re-introduced in V11 SP5. If your subclient design strategy relies on defining exclusive content between subclients, update the CommCell^® environment to the latest service pack.

Virtual Server Agent (VSA) Default Subclient

The VSA also has a default subclient that acts as a catch-all. It protects virtual machines that are not protected by other custom subclients. However, the VSA does not have the mutual exclusiveness mechanism and behaves differently than other agents. It allows duplicate content to be defined between custom subclients.

For example, if the first subclient is defined to protect VMs in DataStore 1, and a second subclient is configured to protect Linux VMs, if there are Linux VMs in DataStore 1, they will be protected by both subclients.
VMs defined in a custom subclient will always be filtered from the default subclient.

Modifying Contents of the Default Subclient

The content of the default subclient is represented by a slash (backslash for Windows-based agents and forward slash for Linux/Unix based clients). Do NOT modify the contents of the default subclient. Modifying this content disables the auto detect functionality of the default subclient. If this is done, any future content required for protection must be explicitly added to the subclient contents.

Defining Subclient Contents

The Contents tab is used to define the content for the subclient. Make sure to review the following important points prior to configuring the subclient content:

• Do NOT modify the contents of the default subclient. If only certain drives are to be protected, use the Filter tab to exclude those drives.
• Add content by using the Browse button or manually entering it using the Add Paths button.
• Define content drives, folders, or file types such as Office documents or Media files.
• Use the Add Paths button to enter UNC paths to protect data on systems that do not have Commvault agents installed. You will be prompted to enter a user account with proper permissions to read the data from the shared location. This feature is only recommended when protecting small amounts of data.

Content Rules for the Default and Custom Subclients

The following rules apply when defining content in subclients:

• Any data defined in a custom subclient is automatically filtered from the default subclient.
• An explicit path such as f:\ cannot be defined in two separate subclients within the same backup set.

Content Rules Prior to V11 SP5:

• If one subclient has the f:\ drive defined as content and a second subclient has the f:\users folder defined, the f:\users folder will be backed up by both subclients.
• If one subclient has the f:\users folder defined and a second subclient has f:\users\finance defined, the f:\users\finance folder will be backed up by both subclients.
• If one subclient has the f:\ drive defined and a second subclient has content defined as *.DOCX, both subclients will protect all DOCX files redundantly.
• If it is necessary to configure subclients using the above examples and you want to avoid protecting redundant data, ensure paths are filtered from other subclients.

Example:

• Two subclients f:\ and f:\users.
• f:\ drive subclient contains an f:\users local filter.

Content Rules as of V11 SP5 and Higher:

• If one subclient has the f:\ drive defined as content and a second subclient has the f:\users folder defined, the f:\users folder will be protected only by the f:\users subclient and will automatically be excluded from f:\drive.
• If one subclient has the f:\users folder defined and a second subclient has f:\users\finance defined, the f:\users\finance folder will be protected only by the f:\users\finance subclient and will automatically be excluded from f:\users.
• If one subclient has the f:\ drive defined and a second subclient has content defined as *.DOCX, both subclients will protect all DOCX files redundantly.
• To prevent file types from being protected in multiple subclients, filter the file type from other subclients. Filter options include the 'content library' option.

Subclient Configuration

Rename a Subclient

Right-click the subclient | Click Properties | General tab

The name of a subclient can be modified at any given time without impacting any operations such as schedules. Feel free to rename a subclient as desired.

Tip: Renaming the Default Subclient
If you rename the default subclient, always leave the string 'default' in the new name. This helps to identify the default subclient that acts as a catch-all.

Define Contents

Right-click the subclient | Click Properties | Content tab

The Contents tab is used to define the content for the subclient.

Key points to review before configuring subclient content:

• The contents of a default subclient, for most backup agents, is a \ (Windows-based) or / (Linux/Unix based). This represents an auto-detection functionality that protects any newly added volumes. If the default subclient's content is modified, the \ or / is removed and auto-detection is disabled. Do NOT modify the contents of the default subclient. If only certain drives are to be protected, use the Filter tab to exclude the unwanted drives.
• Add content by using the Browse button or manually entering it using the Add Paths button.
• Use the Add Paths button to enter UNC paths to protect data on systems that do not have Commvault agents installed. You will be prompted to enter a user account with proper permissions to read the data from the shared location. This feature is only recommended when protecting small amounts of data
• The option to 'Backup System State' is used to protect system state data for Windows servers. By default, the default subclient automatically protects system state data. If required, a separate subclient can be defined to specifically protect system state. Only one subclient within a backup is be designated to protect system state data.

Defining Content Using Wildcards and Patterns

When defining content using the Add Paths button, regular expressions (or wildcard characters) can be used. Wildcard expressions are characters such as * or ?. Regular expressions include patterns such as [a-f] or *.[l-n]df. The terms are interchangeable.
You can use the regular expressions for defining content at any level in the data path. For example:

F:\Users
[A-L]* or *.pst.

The expression ** represents any folder structure. For instance, :*\data\ means that any data folder, from any drives, in any folder structure, is backed up.

Defining Windows system folders as content

The system folders, such as My Documents, Desktop, Music, etc. can also be defined as content. The '%Document%' path is useful when protecting laptops with different operating system versions and languages.

Data Readers

Right-click the subclient | Click Properties | Advanced | Performance tab

Data Readers determine the number of concurrent read operations that are performed when protecting a subclient. For File System agents, by default, the number of readers permitted for concurrent read operations is based on the number of physical disks available. The limit is one reader per physical disk. If there is one physical disk with two logical partitions, setting the readers to 2 will have no effect. Having too many simultaneous read operations on a single disk could potentially cause the disk heads to thrash, slowing down read operations and potentially decreasing the life of the disk. The Data Readers setting is configured in the General tab of the subclient and defaults to two readers.

Multiple readers on a single drive server

Multiple readers on a multiple drives server

Allow multiple readers within a drive or mount point

When a disk array containing several physical disks is addressed logically by the OS as a single drive letter, the 'Allow multiple readers within a drive or mount point' is used as an override. This allows a backup job to take advantage of the fast-read access of a RAID array.

Allowing multiple readers on the same drive

Storage Policy Association

Right-click the subclient | Click Properties | Storage Device tab | Data Storage Policy subtab

Traditionally data protection has always been approached at the server level where an entire server is protected as one piece. Commvault^® software handles data protection at the subclient level where each subclient, when protected, runs as a separate job. If three subclients are defined within a backup set and, if they all run at the same time, three jobs will appear in the job controller.

Content defined at the subclient level is directed to storage through a storage policy. This means that data defined in different subclients on the same server can be directed to different storage and have different retention settings.

Tip: Different Retention Requirements for Data on the Same Server
Scenario: You have a large file server which is 7TB in size. The backups of that server must be kept for 30 days. On the same server, there is a 500MB critical financial data folder, for which backups are requiring 7 years retention. How can the protection of that server be addressed?

Solution: There is no need to keep the entire server for 7 years. Create a subclient that has the critical folder defined as contents. Associate it with a 7-year retention storage policy. Use the default subclient to protect the rest of the server and associate it with a 30-day retention storage policy.

System State for Windows Subclients

For Windows operating systems, ensuring proper protection of System State is critical when performing full system restores. VSS should be used for System State backups to ensure all components are properly being protected. By default, the default subclient is enabled to protect the System State. This is changed by unchecking the 'Backup System State' checkbox.

By default, the System State is backed up on every backup job. But the System State backup can sometimes be large (up to 20GB with the newest versions of Windows). Therefore, running an incremental backup job on a server can result in a System State backup being much larger than the actual incremental data. An option is available to execute the System State backup, but only during full backup jobs. This means that it is skipped on an incremental or a differential backup job.

System State and Domain Controllers

For Domain Controllers, System State backups protect the Active Directory database. This is used when restoring Domain Controllers, which can be restored in an Authoritative or Non-Authoritative restore mode. Although Microsoft recommends, and many administrators choose to rebuild Domain Controllers, the ability to use the Commvault^® software to restore a Domain Controller is critical, especially in a disaster situation. Therefore, do NOT use the 'Backup System State Only With Full Backups' option and instead back it up on every job.

System State and Microsoft^® Clusters

A critical volume, called the quorum drive, is implemented when deploying Microsoft^® Clustering Services. The quorum drive is essential to a full recovery of a clustered environment. When the physical servers of the cluster are backed up, the System State backup of the server that happens to be the active node during the backup captures the quorum drive content.

Subclient Administration

Add Subclients

Right-click the backup set | Click All tasks | New subclient

Key points for adding subclients:

• When new subclients are added, the content defined in the new subclient is automatically removed from the default subclient.
• Content within a backup set cannot be defined in more than one subclient, except for VSA subclients.

Delete a Subclient

Administration | Servers | Agent | BackupSet | Actions button of the subclient | Delete

Key points for deleting subclients:

• Deleting a subclient will NOT result in deleting data already protected for the subclient. However, the data cannot be restored from the Commvault Command Center^TM.
• When a subclient is deleted, the content that was defined for the deleted subclient is automatically included in the default subclient for future data protection jobs.
• To browse previous backup jobs for the deleted subclient, browse at the backup set level using the CommCell^® Console.

Filtering

Right-click on Subclient | Click Properties | Filters tab

Filters are defined at the global and subclient level to remove specific folders and objects that do not require protection. Global filters defined in the Global Filters applet in Control Panel are automatically or manually associated with subclients. If global filters are associated with a subclient, the choice to override the global filters is available.

Key points for global and subclient filters:

• Global filters can be defined for Windows, UNIX, Exchange (Classic), Hadoop, IBM i, Macintosh, NAS, NFS, OES, OpenVMS, and Virtual Servers.
• To enforce global filters to subclients, enable the 'Use Global Filters on all Subclients' checkbox.
• Subclient settings to inherit global filters are configured as:
  • On – always use global filters.
  • Off – never use global filters.
  • Cell-Level Policy – only use global filters if 'Use Global Filters on all Subclients' checkbox has been enabled.
• Subclient filters include exclusion and exception filter entries:
  • Exclusion filters – determines which folders and/or objects will be excluded from the subclient.
  • Exception – is an override for exclusion and global filters. This means any folders and/or objects defined in the exception entry is protected by the subclient.

Similar to defining content, wildcards and patterns are used to define exclusion filters and exceptions. 

Handling Opened and Locked Files

Right-click on Subclient | Click Properties | General tab

Frequently during backup jobs, several files can be opened or locked. This results in the backup system skipping them. Few options are available in such cases.

Filter Files

If the files are not important and you do not wish to protect them, filter them out. This keeps the backup summary report clean, and if an important file is skipped later, it is easy to notice from the report. If hundreds of unimportant files are not addressed and always skipped during backups, they are listed in the report, making it hard to spot important files scattered in so many files.

Use Scripts to Start and Stop Service

If the files are important, and a service or an application is locking the files, you can execute scripts to stop and then start the service before and after the backups.

But the first to consider is using a mechanism to quiesce the file system during the backup. Commvault^® software can interact with such mechanisms.

VSS Option to Quiesce Windows File Systems

For Windows servers subclients, the option to use VSS is available to quiesce the file systems. This engages Windows Volume Shadow Services to properly snap the volumes, which conducts a consistent backup of the files that would be otherwise locked. There is no need to configure any Volume Shadow Services options nor schedule any snapshot. The only requirement is that the Microsoft Volume Shadow Service is enabled and set to either Manual or Automatic.

For All Files vs. For Locked Files Only

When enabling the VSS option, there are two available settings that can also be enabled, 'For All Files', or 'For Locked Files Only.' There is a significant difference between the two settings.

• For All Files – This option engages a VSS snapshot at the beginning of the backup phase, right after the scan completes, for all the volumes that have content defined in the subclient. Once snapped, the backup phase starts protecting all files. While the file system is snapped, all changes occurring are stored in the VSS cache, which is on the same volume by default. The snapshot remains engaged as long as the backup job runs. Once the job completes, the snapshot is released, and the changes that were cached are then applied to the file system.
• For Locked Files Only (Default) – This option is similar but yet different than the previous setting. At the beginning of the backup phase, no VSS snapshot is engaged. But when a locked file is encountered, then a VSS snapshot is conducted. The file is backed up and the snapshot gets immediately released. When another locked file is encountered, the same process occurs again. This option has a significant advantage over using VSS for all files.

Example: Consider a very large server with a high change rate. If VSS is used for all files, the volumes are snapped. But the server is large, therefore, it can take a long time — maybe a few hours to protect it. Since it also has a high change rate, a large amount of changes is cached while it is being backed up. When the snapshot is released, these changes are committed to the file system. Sometimes, it impacts the server in such a way that it becomes unresponsive for a few seconds. The setting to 'use VSS for locked files only' engages VSS more frequently during a backup, but since it is just to protect one file at a time and then it is released, a very small amount of changes needs to be applied, avoiding any impact on the server.

Running Scripts from a Subclient

Commvault^® software allows any desired script to run before and/or after any phase of the backup job. The scripts must be copied to the client and then added as a pre/post process script in the subclient properties.
Backup job phases:

• Scan – creates the list of files requiring protection
• Backup – sends a copy of the files to Commvault^® storage for protection
• Snapshot – creates the hardware snapshot when IntelliSnap^® Technology is used

Popular use cases to implement scripts:

• Stopping/starting services or application to backup locked files
• Executing any database maintenance scripts before/after database backups
• Disconnecting users from applications

Block-Level Backup

In Version 11 block-level backups integrate software-based snapshots with block storage mechanisms to increase speed and efficiency. This option is recommended for servers with a very large number of files.

Once enabled, the block-level backup feature replaces the traditional file (or object-level backup). This capability provides faster backups by reading data directly from the disk rather than from the database or file system. It also allows granular recovery by restoring individual items from the block-level backup. 

Block-Level for the Windows File System agent requires the installation of the block filter driver on the client server. For more information on requirements, please refer to the Commvault^® online documentation.

Subclient Design Strategy

Defining custom subclients allows for much greater flexibility in crafting a comprehensive protection strategy. User-defined subclients allow data to be explicitly defined within a subclient container. This subclient container can then be protected and managed independently within Commvault^® protected storage. There are many advantages to using custom subclients.

The following list highlights the primary advantages of using custom subclients:

• Allows for better media management by meeting protection needs based on specific content.
• Better performance by using multiple streams within a subclient, protecting multiple subclients concurrently, or by stagger scheduling subclients over a longer time period.
• Allows for custom configurations for specific data such as open file handling, filtering or Pre/Post Process scripts.

Special Protection Needs

Data being protected by an agent is, by default, protected by the default subclient. Custom subclients can be explicitly defined to manage specific data such as a folder or database. Each subclient container can be managed independently in protected storage. This reduces the amount of data that needs to be protected by associating just subclient data to the storage policy copy to meet protection requirements.

Example: A file server with 800GB of data has a file share containing critical financial data that must be retained for 10 years. The folder containing the data can be defined in a separate subclient and the subclient can be associated with a storage policy copy with a retention of 10 years. The result is instead of keeping all 800GB of data for 10 years, only the financial data will be kept for the required period.

Performance Requirements

Each defined subclient is an independent job and uses independent streams when being protected. There are several reasons why this improves performance:

• Multiple Stream Backups
• Stagger Schedule Backups

Multiple Stream Backups

A subclient is configured to use multiple streams for supported agents. This is useful when data is being stored in a RAID array. To take advantage of RAID's fast read access; multiple streams are used to improve the performance of data protection operations.
Multiple Subclients running concurrently will result in multiple stream data protection operations. This is especially useful when the application does not inherently support multi-stream backups, such as Exchange message-level backups or archives.

Stagger Schedule Backups

By creating separate subclients, you can stagger schedule data protection operations. Instead of trying to get a full backup done in one night, different subclients can be scheduled to run full backups throughout the week or month and incremental backups on other days. This is especially useful for virtual machine backups or NAS with large file counts.

Filters

Filters are applied through the Global Filter applet in Control Panel or locally at the subclient level. If specific folder locations require special filters, a dedicated subclient should be used. Define the subclient content to the location where the filters will be applied and configure local filters for that subclient. The option to use Global Filters can still be used allowing the global and local filters to be combined.
If global filters are being used but specific subclient data should not have certain filters applied, define the content in a separate subclient. Global filters can still be enabled for the subclient, but the exceptions list can be used to override the global filter settings for specific file/folder patterns.

Open-File Handling

Open-File Handling using Microsoft VSS is used to ensure open files are protected. VSS is an available option for Windows 2003 or higher agents.

Pre/Post Scripts

Pre/Post Process scripts are used to quiesce applications prior to protection. This is very useful when protecting proprietary database systems or for quiescing databases within virtual machines prior to using the Commvault Virtual Server Agent for snapping and backing up the virtual machine.