Processes

Understanding Commvault processes and associated log files will help you learn how to better troubleshoot common problems. Conceptually, it is difficult to become good at troubleshooting – practice is key. Knowing the processes and logic flow of operations, such as backups and auxiliary copy jobs, will help improve troubleshooting abilities over time.

The types of log files that reside on each component of the CommCell^® environment (CommServe^® server, MediaAgent, Client computer) might differ depending on its role. For example, a CommServe server contains only the CommServe log files. A component that is both a CommServe and a Client contains the log files for both entities.

Base Services

Base Services are installed on components, such as the CommServe^® server, the MediaAgent or a client. These services are made of components and processes that allow Commvault^® software to communicate between the servers.

Base Service Process and Log File Names:

Process Name	Log File Name	Description
CVD	CVD.log CVPerfMgr.log	The main communication process responsible to send or fetch metadata information and send status updates to the CommServe^® server. It is also responsible to open a pipe between the client and the MediaAgent for data transfer.
ClMgrS, for the Domino Mailbox Archiver, File System Agent, File Share Archiver Client, Virtual Server Agent, and Web Server components	ClMgrS.log	Responsible for running and controlling recall jobs and failover operations on clusters. It uses SSL certificate to authenticate connections with clients. For Virtual Server Agents (VSA), the service tracks Hyper-V block changes.
Cvlaunchd, for Linux systems only	cvlaunchd.log	A daemon that runs on the Linux operating system responsible for spawning Commvault^® processes.

CommServe^® Server Processes

The following diagram illustrates how the CommServe server's main processes operate within the CommCell^® environment.

CommServe^® Server processes:

CommServe^® Server processes and log files

Process Name	Log File Name	Description
DownloadUpdates	DownloadSoftware.log	Download the Commvault^® Software updates from Commvault's FTP site.
EvMgrS	EvMgrS.log CS_CVMMInst.log WorkQueue.log	Responsible for receiving messages and updates from clients. It also feeds information to the CommCell console.
DistributeUpdates	DistributeSoftware.log	Responsible for pushing updates to client servers. It also coordinates activity on the client using the InstallUpdates and RemoveUpdates processes.
JobMgr	JobManager.log	Responsible for initiating and controlling jobs and communicating with storage resources. It acts as the primary coordinator for all data movement operations. The JobManager.log is typically the first log to view when troubleshooting data movement problems. Every starting and stopping of processes during a data movement operation is logged in the JobManager.log. JobMgr process example for an auxiliary copy Job: The JobMgr initiates the auxiliary copy job by communicating with the source MediaAgent to reserve storage resources for the source job. It then communicates with the destination MediaAgent to reserve destination storage resources. It then communicates with the CVODS process to generate required data for the auxiliary copy job. Once the auxiliary copy job has completed, the JobMgr.exe reports the job as complete. Note that in this example not only does the JobMgr communicate with the CVODS process, but it also communicates with both the source and destination MediaAgents to allocate storage resources.
ArchPrune	DataAging.log	Is initiated during data aging operations to clear out data that has exceeded retention. It is also used to prune index data from the index directory based on index retention settings.
MediaManager	MediaManager.log MediaManagerPrune.log ArchMgr.log DBTblStats.log	Responsible for controlling the hardware devices that are part of a CommCell^® environment.
IndexingService	StartRestore.log StartSynthFull.log	Responsible for coordinating restore and synthetic full operations.
CVODS	AuxCopyMgr.log	Responsible for communicating with the AuxCopy process on source and destination MediaAgents to control auxiliary copy and DASH Copy operations. It controls auxiliary copy jobs and sends information about what is required to be copied.

MediaAgent Processes

The following diagram illustrates how the MediaAgent's main processes operate within the CommCell^® environment.

MediaAgent processes

MediaAgent processes and log files:

Process Name	Log File Name	Description
IndexingService	IndexingService.log	Responsible for creating, updating and purging indexes and index databases in the index directory.
AuxCopy	AuxCopy.log on the primary copy MediaAgent. CVD.log on the secondary copy MediaAgent.	Responsible for reading and sending chunks required to be copied to the target MediaAgent. It receives the list of chunks to copy from the CVODS process running on the CommServe^® server.
CVMountD	CVMA.log IndexCacheServer.log	Responsible for interacting with the hardware devices that are attached to the MediaAgent and are defined in the CommCell^® environment.
SynthFull	SynthFull.log	Responsible for synthetic full restore and backup when generating a synthetic full backup.
ArchiveIndex	ArchiveIndex.log	Responsible for copying the index of the backup job, from the index directory, onto the disk or tape media.
SIDB2	SIDBEngine.log SIDBPrune.log SIDBPhysicalDeletes.log	Is the Deduplication Database (DDB) process used when deduplication is enabled.

Client Processes

The following information describes client processes and log files and how they function within the CommCell^® environment.

Client processes and log files:

Process Name	Log File Name	Description
iFind	FileScan.log	Responsible for scanning the client file systems and generating collect files containing the list of files requiring protection.
CLBackup	clBackup.log	Responsible for reading and collecting the files during the backup job.
CLRestore	clRestore.log	Responsible for writing files on a client during the restore job.
CLIFRestore	clRestore.log	Responsible for restoring data on a client during an index free restore job, such as a Restore By Job.

Non-Deduplicated Backup Job Process

The following graph illustrates how multiple processes operate during a non-deduplication backup. Understanding the order in which each process runs helps troubleshoot issues:

Non-deduplicated backup job

Non-deduplicated backup job steps:

The JobMgr service running on the CommServe^® server initiates the backup job by sending a signal to the client.
The client uses the iFind process to generate a list of files requiring protection. Once completed, the client sends the status to the CommServe^® server.
The CommServe^® server uses the MediaManager process to interface with the MediaAgent to request mounting the required disk, tape or cloud library.
The MediaAgent mounts the library using the CVMountD process. It then sends the status to the CommServe^® server.
The client and the MediaAgent open a pipe using the CVD process.
The client collects the data using the CLBackup process and sends it to the MediaAgent, through the pipe.

Commvault Deduplicated Backup Job Process

Commvault^® deduplicated and non-deduplicated jobs function similarly, but there are additional steps to handle the deduplicated backup process. Understanding the order in which each process runs helps troubleshoot issues:

Deduplicated backup job

Deduplicated backup job steps:

The JobMgr service running on the CommServe^® server initiates the backup job by sending a signal to the client server.
Once the client completes the scan phase and the MediaAgent mounting the library, the client opens the pipe with the MediaAgent using the CVD process.
The MediaAgent starts the deduplication database engine, therefore the SIDB2 process.
The SIDB2 process gets job and deduplicated data from the CommServe^® server.
The client starts collecting data using the CLBackup process. This process hashes the data and generates signatures that get sent to the MediaAgent SIDB2 process. If the client-side disk cache option is enabled, the CLBackup process also triggers the CLDBEngine process, which is the small client-side cache.
Once the signatures are received, the MediaAgent SIDB2 process validates if the signatures were already encountered. If not, it requests the block of data from the client. If yes, it increases the counters for that signature and has the client discard the block.

DASH Full Processes

A DASH Full job is a deduplicated synthetic full backup, which involves the SIDB2 process. Understanding the order in which each process runs helps troubleshoot issues.

DASH Full backup job

DASH Full backup job steps:

The JobMgr process running on the CommServe^® server initiates the backup job.
The MediaAgent SIDB2 process is initiated. It first accesses the most recent index from the index directory to get the most recent image file. This file lists all the files that were present on the client during the last backup job (these files are required to build the new synthetic full backup).
The SIDB2 process reads the block signatures that are in the metadata chunk files of the disk library. The data is not re-hashed, nor re-copied.
The SIDB2 process validates if the signatures are in the Deduplication Database (DDB) as expected.
The SIDB2 process increases the counters for the signatures, referring to the new DASH Full job.

Auxiliary Copy Processes

The auxiliary copy job copies data from a primary storage policy copy to a secondary storage policy copy. In some cases the source can also be a secondary copy from which another secondary copy gets synchronized. Understanding the order in which each process runs helps troubleshoot issues:

Auxiliary copy job

Auxiliary copy job steps:

The JobMgr process running on the CommServe^® server initiates the auxiliary copy job.
The CommServe^® server JobMgr and MediaManager processes then send reservation requests to the source MediaAgent for stream resources.
The CommServe^® server JobMgr and MediaManager processes send reservation requests to the destination MediaAgent for stream resources.
The destination MediaAgent allocates the resources by mounting the library using the CVMountD process.
The CommServe^® server CVODS process creates the list of chunks required to be copied.
The CVODS process sends the list of chunks to the source MediaAgent AuxCopy process.
The source and the destination MediaAgents open a pipe using the CVD process.
The source MediaAgent AuxCopy process sends a batch of chunks to the destination MediaAgent AuxCopy process through the established pipe.
Once all chunks are copied, the destination MediaAgent AuxCopy process requests additional chunks from the source.
The destination MediaAgent AuxCopy process sends status updates about the copy to the CommServe^® server until all chunks are transferred and the job completes.

DASH Copy Processes

A DASH Copy is an optimized auxiliary copy job that requires only changed blocks to be sent to a second disk target. For instance, A DASH Copy can be:

A primary deduplicated copy from the main data center that gets replicated to a deduplicated copy at the disaster recovery site.
A primary deduplicated copy that gets replicated to a deduplicated cloud copy.

Understanding the order in which each process runs helps troubleshoot issues:

DASH Copy job

DASH Copy job steps:

The CVODS process, on the CommServe^® server, sends to the source MediaAgent AuxCopy process the list of chunks required to be sent to the destination MediaAgent.
The source MediaAgent AuxCopy process generates signatures and sends them to the target MediaAgent to validate if the blocks are already present in the destination library. The signatures can be generated in two different ways, based on the selected option:

If 'Network Optimized' is selected, the AuxCopy process reads the chunk files and generates signatures for each block.
If 'Disk Read Optimized' (default setting) is selected, the AuxCopy process does not process the data. Instead, it gets the signatures already stored in the chunk metadata files.

It is recommended to use this default setting, unless required or directed by Commvault^® support team.

3. Optionally, if the Enable Source Side Disk Cache option is selected, the CLDBEngine process on the source MediaAgent is used. Signature lookups are first validated from there, prior to being sent to the destination MediaAgent.

4. The destination MediaAgent Deduplication Database (DDB), which is the SIDB2 process, receives the signature queries. It validates if the signature already exists in the DDB, and therefore the block is already in the library:

a. If it is not there, it asks the source MediaAgent to send the block of data.

b. If it is already there, it asks the source MediaAgent to drop the block and it adds a counter for that block in the DDB.

Note that even if the source copy is deduplicated, the source MediaAgent SIDB2 process is never queried, nor involved in the process.