Commvault

Deduplication Database Operations

Owned by Jordan Cannata (Unlicensed)
Last updated: Nov 30, 2021 by tdopko@commvault.com (Unlicensed)
7 min read

Quick Links to Topics:

Credits:

Great thanks to Mike Byrne for his hard work with the screen captures.

Move the DDB

If the Deduplication Database (DDB) needs to be relocated to a different MediaAgent or disk, the 'Move Partitions' option is used. Before moving the database, ensure no data protection jobs, DDB backups, data verification, or DDB reconstructs are running.




To change the location of the DDB partitions

1 - Right-click the DDB partition | All Tasks | Move Partitions.

2 - Click Choose Path…

3 - Select the MediaAgent and path for the new DDB location.


Cross Operating System DDB Move

Cross operation Deduplication Database (DDB) Move is the process of moving a DDB between MediaAgents running on different operating systems. For instance, a DDB affiliated with a Windows MediaAgent can be moved to a Linux MediaAgent. This is especially useful when migrating a library to a new MediaAgent.

In this illustration, a network attached storage (NAS) library is migrated from a Windows MediaAgent to a Linux MediaAgent.

Library migration between MediaAgents using different operating systems



Seal the DDB

Sealing the Deduplication Database (DDB) closes the active database and starts a new one. It also marks all folders in the deduplication store as full which means no new data can be written to the folders. The database remains in the DDB drive location as it is still needed for data aging and pruning operations. Once all data from the sealed DDB is aged and pruned, the system automatically removes the database. The database should never be manually removed. Once all data has aged and all records deleted, the system will remove the DDB automatically.

Reasons to seal the DDB:

  • Block lookups take too long and data protection jobs and the pruning process slow substantially. By sealing the database, a new DDB is started—which results in faster data protection jobs, but diminishes deduplication ratios.
  • Organizations with large amounts of data protected through a single storage policy and DDB. Since there is an upper limit to the DDB, sealing the database may be required. This is dependent on how long the data is being retained for, since blocks pruned from disk have signature records pruned from the DDB.
  • The DDB is stored on slow disks or inefficient protocols such as NFS, CIFS, or iSCSI are being used.
  • Using SILO storage to tape. A store can be sealed when it grows to a certain size, after a defined number of days, or through time intervals such as once a quarter. If you wanted to have a set of tapes representing a fiscal quarter, then sealing the database every three months would result in all volume folders being sealed and placed in SILO storage—isolating the data based on the fiscal quarter.

If the environment is designed and scaled appropriately, then sealing the database is not necessary. If you are managing large amounts of data, it is recommended to consult with Commvault® experts for assistance in designing a deduplication solution.


If data protection or pruning performance is degrading significantly, contact Commvault® Support before sealing the DDB.




To manually seal a deduplication database

1 - Expand the desired deduplication engine | Right-click the partition | All Tasks | Seal Deduplication Database.

2 - Click Yes to confirm warning about new baseline.

3 - Type the phrase to confirm that the required space is available.

4 - The system seals the DDB and starts a new one.



Data Verification

With all the benefits of Commvault® deduplication, it is critical to consider the integrity of deduplicated data. A corrupt block in the deduplication store can result in data from multiple jobs not being recoverable. Commvault® V11 provides live data verification operations that are conducted while data protection jobs are running. To use data verification, the MediaAgent options 'Validation on Media' and 'Validation on Network' must be enabled, which they are by default.

There are four verification options:

  • Verification of existing jobs on disk and deduplication database
  • Verification of deduplication database
  • Quick verification of deduplication database
  • Incremental verification

Verification of Existing Jobs on Disk and Deduplication Database

This verification method uses checksum data to verify block integrity by reading data chunks (Sfiles), uncompressing, and decrypting, and using Cyclic Redundancy Check (CRC) information to validate block integrity. This option also verifies chunk metadata using CRC checks. Any blocks failing the check will be marked in the DDB. New blocks generating the same signature as a block marked bad are re-written to disk and a new signature entry is written to the DDB. This verification method also verifies chunk integrity between the DDB and disk library.

Verification of Deduplication Database

This verification method performs all the same tasks as the 'Verification of Existing Jobs on Disk and the Deduplication Database' except metadata chunk validation.

Quick Verification of Deduplication Database

The verification method quickly verifies chunk integrity between DDB and disk library.

Incremental Verification

This method verifies data integrity for new jobs added since the last verification job. This option is available when running 'Verification of Deduplication Database' or 'Verification of Existing Jobs on Disk and the Deduplication Database' options. Commvault® introduced a DDB verification schedule that executes an incremental verification every day, at 11 a.m. Since this method only verifies new jobs, full verification jobs should periodically be executed, such as once a month or once a quarter. 
The best way to protect against potential data corruption, whether using deduplication or not, is to always have multiple copies of data.



To edit the DDB Verification schedule

1 - Click Schedule Policies | Right-click the System Created DDB Verification schedule policy | Edit.

2 - Select the schedule.

3 - Click Edit.



4 - Modify schedule options if required.

5 - Check to run an incremental verification or un-check to run a full verification.

6 - Select desired verification options.



To run a data verification job

1 - Right-click DDB partition | All Tasks | Run Data Verification.

2 - Data verification job can be scheduled, run immediately or saved as a script.

3 - Configure data verification options.



Deduplication Database Resynchronization

Sometimes there can be inconsistencies between the deduplication database entries and the CommServe® server database job history. If this happens, the deduplication database is switched to maintenance mode. When in this mode, data aging, backup, or auxiliary copy jobs cannot run using the DDBs. Also, recovering the CommServe server database to a previous point-in-time can also lead to inconsistencies between the two databases.

For example, client backups are executed every hour and the DR backup is scheduled for 10:00 a.m. If the CommServe® server crashes at 1:00 p.m. and is restored, it uses the 10:00 a.m. DR backup. However, since some client backups ran during 10:00 a.m. and 1:00 p.m., the deduplication database contains block entries created after 10:00 a.m. Therefore, these orphaned blocks entries are not known by the CommServe® server database. 

To resolve any discrepancies, a deduplication database resynchronization must be executed.

DDB resynchronization following a CommServe® server database restore


Note that after a CommServe® server database restore, the deduplication databases may be in maintenance mode, which requires resynchronization. But the resync process will work only if the CommServe server database is restored from a DR backup that is less than five days old. If the DR backup used is older than five days, the deduplication databases must be sealed, leading to a re-baseline of the deduplication store.




To resync deduplication databases

1 - Expand Deduplication Engines | Expand any DDB | Click a partition.

2 - Validate that the partition status is set to maintenance mode.



3 - Right-click Deduplication Engines and synchronize all databases.

4 - Acknowledge the warning.



5 - Open the Event Viewer view.

6 - Validate that all partitions are re-synced and online.



Copyright © 2021 Commvault | All Rights Reserved.