Commvault

Processes

Owned by Carl Brault (Unlicensed)
Last updated: Nov 30, 2021 by tdopko@commvault.com (Unlicensed)
9 min read

Quick Links to Topics:



Processes

Understanding Commvault processes and associated log files will help you learn how to better troubleshoot common problems. Conceptually, it is difficult to become good at troubleshooting – practice is key. Knowing the processes and logic flow of operations, such as backups and auxiliary copy jobs, will help improve troubleshooting abilities over time.


The types of log files that reside on each component of the CommCell® environment (CommServe® server, MediaAgent, Client computer) might differ depending on its role. For example, a CommServe server contains only the CommServe log files. A component that is both a CommServe and a Client contains the log files for both entities.

Base Services

Base Services are installed on components, such as the CommServe® server, the MediaAgent or a client. These services are made of components and processes that allow Commvault® software to communicate between the servers.


Base Service Process and Log File Names:

Process Name

Log File Name

Description

CVD

CVD.log
CVPerfMgr.log

The main communication process responsible to send or fetch metadata information and send status updates to the CommServe® server. It is also responsible to open a pipe between the client and the MediaAgent for data transfer.

ClMgrS, for the Domino Mailbox Archiver, File System Agent, File Share Archiver Client, Virtual Server Agent, and Web Server components

ClMgrS.log

Responsible for running and controlling recall jobs and failover operations on clusters. It uses SSL certificate to authenticate connections with clients. For Virtual Server Agents (VSA), the service tracks Hyper-V block changes.

Cvlaunchd, for Linux systems only

cvlaunchd.log

A daemon that runs on the Linux operating system responsible for spawning Commvault® processes.




CommServe® Server Processes

The following diagram illustrates how the CommServe server's main processes operate within the CommCell® environment.


CommServe® Server processes:




CommServe® Server processes and log files


Process Name

Log File Name

Description

DownloadUpdates

DownloadSoftware.log

Download the Commvault® Software updates from Commvault's FTP site.

EvMgrS

EvMgrS.log
CS_CVMMInst.log
WorkQueue.log

Responsible for receiving messages and updates from clients. It also feeds information to the CommCell console.

DistributeUpdates

DistributeSoftware.log

Responsible for pushing updates to client servers. It also coordinates activity on the client using the InstallUpdates and RemoveUpdates processes.

JobMgr

JobManager.log

Responsible for initiating and controlling jobs and communicating with storage resources. It acts as the primary coordinator for all data movement operations. The JobManager.log is typically the first log to view when troubleshooting data movement problems. Every starting and stopping of processes during a data movement operation is logged in the JobManager.log.
JobMgr process example for an auxiliary copy Job:
The JobMgr initiates the auxiliary copy job by communicating with the source MediaAgent to reserve storage resources for the source job. It then communicates with the destination MediaAgent to reserve destination storage resources. It then communicates with the CVODS process to generate required data for the auxiliary copy job. Once the auxiliary copy job has completed, the JobMgr.exe reports the job as complete.
Note that in this example not only does the JobMgr communicate with the CVODS process, but it also communicates with both the source and destination MediaAgents to allocate storage resources.

ArchPrune

DataAging.log

Is initiated during data aging operations to clear out data that has exceeded retention. It is also used to prune index data from the index directory based on index retention settings.

MediaManager

MediaManager.log
MediaManagerPrune.log
ArchMgr.log
DBTblStats.log

Responsible for controlling the hardware devices that are part of a CommCell® environment.

IndexingService

StartRestore.log StartSynthFull.log

Responsible for coordinating restore and synthetic full operations.

CVODS

AuxCopyMgr.log

Responsible for communicating with the AuxCopy process on source and destination MediaAgents to control auxiliary copy and DASH Copy operations. It controls auxiliary copy jobs and sends information about what is required to be copied.




MediaAgent Processes

The following diagram illustrates how the MediaAgent's main processes operate within the CommCell® environment.


MediaAgent processes



MediaAgent processes and log files:


Process Name

Log File Name

Description

IndexingService

IndexingService.log

Responsible for creating, updating and purging indexes and index databases in the index directory.

AuxCopy

AuxCopy.log on the primary copy MediaAgent.
CVD.log on the secondary copy MediaAgent.

Responsible for reading and sending chunks required to be copied to the target MediaAgent. It receives the list of chunks to copy from the CVODS process running on the CommServe® server.

CVMountD

CVMA.log
IndexCacheServer.log

Responsible for interacting with the hardware devices that are attached to the MediaAgent and are defined in the CommCell® environment.

SynthFull

SynthFull.log

Responsible for synthetic full restore and backup when generating a synthetic full backup.

ArchiveIndex

ArchiveIndex.log

Responsible for copying the index of the backup job, from the index directory, onto the disk or tape media.

SIDB2

SIDBEngine.log
SIDBPrune.log
SIDBPhysicalDeletes.log

Is the Deduplication Database (DDB) process used when deduplication is enabled.




Client Processes

The following information describes client processes and log files and how they function within the CommCell® environment.


Client processes and log files:


Process Name

Log File Name

Description

iFind

FileScan.log

Responsible for scanning the client file systems and generating collect files containing the list of files requiring protection.

CLBackup

clBackup.log

Responsible for reading and collecting the files during the backup job.

CLRestore

clRestore.log

Responsible for writing files on a client during the restore job.

CLIFRestore

clRestore.log

Responsible for restoring data on a client during an index free restore job, such as a Restore By Job.




Non-Deduplicated Backup Job Process

The following graph illustrates how multiple processes operate during a non-deduplication backup. Understanding the order in which each process runs helps troubleshoot issues:


Non-deduplicated backup job



Non-deduplicated backup job steps:

  1. The JobMgr service running on the CommServe® server initiates the backup job by sending a signal to the client.
  2. The client uses the iFind process to generate a list of files requiring protection. Once completed, the client sends the status to the CommServe® server.
  3. The CommServe® server uses the MediaManager process to interface with the MediaAgent to request mounting the required disk, tape or cloud library.
  4. The MediaAgent mounts the library using the CVMountD process. It then sends the status to the CommServe® server.
  5. The client and the MediaAgent open a pipe using the CVD process.
  6. The client collects the data using the CLBackup process and sends it to the MediaAgent, through the pipe.





Commvault Deduplicated Backup Job Process

Commvault® deduplicated and non-deduplicated jobs function similarly, but there are additional steps to handle the deduplicated backup process. Understanding the order in which each process runs helps troubleshoot issues:


Deduplicated backup job



Deduplicated backup job steps:

  1. The JobMgr service running on the CommServe® server initiates the backup job by sending a signal to the client server.
  2. Once the client completes the scan phase and the MediaAgent mounting the library, the client opens the pipe with the MediaAgent using the CVD process.
  3. The MediaAgent starts the deduplication database engine, therefore the SIDB2 process.
  4. The SIDB2 process gets job and deduplicated data from the CommServe® server.
  5. The client starts collecting data using the CLBackup process. This process hashes the data and generates signatures that get sent to the MediaAgent SIDB2 process. If the client-side disk cache option is enabled, the CLBackup process also triggers the CLDBEngine process, which is the small client-side cache.
  6. Once the signatures are received, the MediaAgent SIDB2 process validates if the signatures were already encountered. If not, it requests the block of data from the client. If yes, it increases the counters for that signature and has the client discard the block.





DASH Full Processes

A DASH Full job is a deduplicated synthetic full backup, which involves the SIDB2 process. Understanding the order in which each process runs helps troubleshoot issues.


DASH Full backup job



DASH Full backup job steps:

  1. The JobMgr process running on the CommServe® server initiates the backup job.
  2. The MediaAgent SIDB2 process is initiated. It first accesses the most recent index from the index directory to get the most recent image file. This file lists all the files that were present on the client during the last backup job (these files are required to build the new synthetic full backup).
  3. The SIDB2 process reads the block signatures that are in the metadata chunk files of the disk library. The data is not re-hashed, nor re-copied.
  4. The SIDB2 process validates if the signatures are in the Deduplication Database (DDB) as expected.
  5. The SIDB2 process increases the counters for the signatures, referring to the new DASH Full job.





Auxiliary Copy Processes

The auxiliary copy job copies data from a primary storage policy copy to a secondary storage policy copy. In some cases the source can also be a secondary copy from which another secondary copy gets synchronized. Understanding the order in which each process runs helps troubleshoot issues:


Auxiliary copy job



Auxiliary copy job steps:

  1. The JobMgr process running on the CommServe® server initiates the auxiliary copy job.
  2. The CommServe® server JobMgr and MediaManager processes then send reservation requests to the source MediaAgent for stream resources.
  3. The CommServe® server JobMgr and MediaManager processes send reservation requests to the destination MediaAgent for stream resources.
  4. The destination MediaAgent allocates the resources by mounting the library using the CVMountD process.
  5. The CommServe® server CVODS process creates the list of chunks required to be copied.
  6. The CVODS process sends the list of chunks to the source MediaAgent AuxCopy process.
  7. The source and the destination MediaAgents open a pipe using the CVD process.
  8. The source MediaAgent AuxCopy process sends a batch of chunks to the destination MediaAgent AuxCopy process through the established pipe.
  9. Once all chunks are copied, the destination MediaAgent AuxCopy process requests additional chunks from the source.
  10. The destination MediaAgent AuxCopy process sends status updates about the copy to the CommServe® server until all chunks are transferred and the job completes.





DASH Copy Processes

A DASH Copy is an optimized auxiliary copy job that requires only changed blocks to be sent to a second disk target. For instance, A DASH Copy can be:


A primary deduplicated copy from the main data center that gets replicated to a deduplicated copy at the disaster recovery site.
A primary deduplicated copy that gets replicated to a deduplicated cloud copy.


Understanding the order in which each process runs helps troubleshoot issues:


DASH Copy job



DASH Copy job steps:

  1. The CVODS process, on the CommServe® server, sends to the source MediaAgent AuxCopy process the list of chunks required to be sent to the destination MediaAgent.
  2. The source MediaAgent AuxCopy process generates signatures and sends them to the target MediaAgent to validate if the blocks are already present in the destination library. The signatures can be generated in two different ways, based on the selected option:
  • If 'Network Optimized' is selected, the AuxCopy process reads the chunk files and generates signatures for each block.
  • If 'Disk Read Optimized' (default setting) is selected, the AuxCopy process does not process the data. Instead, it gets the signatures already stored in the chunk metadata files.

It is recommended to use this default setting, unless required or directed by Commvault® support team.

3. Optionally, if the Enable Source Side Disk Cache option is selected, the CLDBEngine process on the source MediaAgent is used. Signature lookups are first validated from there, prior to being sent to the destination MediaAgent.

4. The destination MediaAgent Deduplication Database (DDB), which is the SIDB2 process, receives the signature queries. It validates if the signature already exists in the DDB, and therefore the block is already in the library:

a. If it is not there, it asks the source MediaAgent to send the block of data.

b. If it is already there, it asks the source MediaAgent to drop the block and it adds a counter for that block in the DDB.


Note that even if the source copy is deduplicated, the source MediaAgent SIDB2 process is never queried, nor involved in the process.





Copyright © 2021 Commvault | All Rights Reserved.