Job Based Retention

A data retention strategy is important for managing storage in your CommCell^® environment. With Commvault^® software, you can define retention for multiple copies of data with each copy having different retention requirements. Additionally, retention may be required at the object-level and not just the data protection operation. Commvault software makes this strategy straight forward to implement by using storage policy copies, subclient object-level retention, and Exchange configuration retention policies.

In Version 11, Commvault software has three primary retention methods:

• Job based retention – Configured at the storage policy copy level, job schedule level, or manually by selecting jobs or media to retain, and applying different retention.
• Subclient object based retention – Configured at the subclient level, it applies retention-based on the deletion point of an object. Object-based retention is based on the retention setting in the subclient properties plus the storage policy copy retention settings.
• Configuration policies – Currently used for Exchange mailbox protection. These policies include archive, retention, cleanup, and journaling. Configuration policies provide the ability to define complete retention and destruction policies, including the capability of deleting messages from the production Exchange environment.

Retention Basics

Commvault^® software provides extensive retention control for protected data. For basic retention requirements, follow the general guidelines and best practices for retention configuration.

Retention general guidelines:

• Disk storage:
  • Leave the Cycles retention set at the default of two
  • Use the Days retention to govern retention policies for each copy
  • Never use extended retention rules when using Commvault deduplication
• Tape storage:
  • Set the Cycles retention based on the number of complete sets of tape copies you want to retain. For example, if you want 30 days of data stored off-site, which includes at least four full backups and all dependent jobs (incremental or differential), for complete recovery from any tape set, set the Cycles retention to four.
  • Set the Days retention based on standard retention requirements.

Job Based Retention

Job-based retention places a standard retention for an entire job. Jobs are retained based on storage policy copy retention rules. Additionally, job-based retention is applied through the job schedule or modified after the job completes.

Storage Policy Copy Retention Rules

Right-click the desired storage policy copy | Click Properties | Retention tab

Policy-based retention settings are configured in the storage policy copy Retention tab. The settings for backup data are Days and Cycles. For archive data, the retention is configured in Days. Retention is also set through schedules or applied retroactively to a job in a storage policy copy.

Days

A day is a 24-hour time period defined by the start time of the job. Each 24-hour time period is complete whether a backup runs or not. This way, a day is considered a constant.

Cycles

A cycle is defined as all backup jobs required to restore a system to a specific point-in-time. Traditionally, cycles are defined as a complete full backup, all dependent incremental backups, differential backups, or log backups; up to, but not including the subsequent full backup. A cycle is referenced as Active or Complete, which means that as soon as a full backup completes successfully it starts a new cycle which is the active cycle. The previous active cycle is marked as a complete cycle.

An active cycle is marked complete only if a new full backup finishes successfully. If a scheduled full backup does not complete successfully, the active cycle remains active until such time that a full backup does complete. On the other hand, a new active cycle begins and the previous active cycle is marked complete when a full backup completes successfully regardless of scheduling. In this way, a cycle can be thought of as a variable value based on the successful completion or failure of a full backup. This also helps to break away from the traditional thought of a cycle being a week long, or even a specified period of time.

Days and Cycles Relation

Cycles and days should directly or indirectly equal each other:

• 2 cycles and 14 days with weekly full backups
• 4 cycles and 30 days being approximately 1 month
• 12 cycles and 365 days for month end full backups being retained for a year

But what about 52 cycles and 365 days? In situations like this it is rather irrelevant how many cycles are set. The truth is, 2 cycles and 365 days is good enough. You will meet your retention requirements since you are keeping data for one year. If backups don't run for over a year, you are still guaranteed to have at least 2 cycles of data in storage based on the aging entire cycles rule.

When setting retention in the policy copy, base it on the primary reason data is being protected. If it is for disaster recovery, ensure the proper number of cycles are set to guarantee a minimum number of backup sets for full backup restore. If you are retaining data for data recovery, then set the days to the required length of time determined by retention policies. If the data recovery policy is for three months, 12 cycles and 90 days or 1 cycle and 90 days will still meet the retention requirements.

With the release of Commvault Version 11 software, the default retention for a storage policy primary copy is 15 days and 2 cycles. A secondary copy's default retention is 30 days and 4 cycles.

Retention Rules for Storage Policy Copy Retention

There are several retention rules that are applied to jobs:

• Both Days and Cycles criteria must be met for aging to occur
• Data is aged in complete cycles
• Days criteria is not dependent on jobs running on a given day

Rule 1: Both CYCLES and DAYS criteria must be met

Commvault^® software uses AND logic to ensure that both retention parameters are satisfied. Another way of looking at this is the longer of the two values of cycles and days within a policy copy always determines the time data is retained for.

Example: Retention for a storage policy copy is set to 3 days and 2 cycles. This is not a typical example, but it's used to logically prove the statement that both days and cycles criteria must be met for data to age. By Monday 3 full backups have been performed. If Friday's full backup is aged, there would be 2 full backups left meeting our criteria of 2 cycles. However, the days criteria calls for 3 days, and if the Friday full backup was aged, only 2 days would be counted. The Friday full backup would therefore age on Tuesday.

Monday at 12 PM the data aging operation runs and determines no data can be marked aged

Tuesday at 12 PM the data aging operation runs and determines the Friday full backup can be marked aged

Rule 2: Data is retained based on complete cycles

Backup data is managed within a storage policy copy as a cycle or a set of backups. This includes the full backup which designates the beginning of a cycle and all incrementals or differentials backups. When data aging is performed and retention criteria allows for data to be aged, the entire cycle is marked as aged. This process ensures that jobs will not become orphaned resulting in dependent jobs (incremental or differential) existing without the associated full backup.

Example: This is another retention example used to prove the rule. Retention is configured for 7 days and 2 cycles. Full backups are being performed on Fridays and Mondays, and incremental backups on all other days. On Saturday the cycles criteria of 2 has been met since there are 3 full backups. If a cycle is removed there would be 2 left, a complete cycle (Monday – Thursday) and the full backup on Friday night.

However, since we prune entire cycles we would have to age the Friday full backup and the incremental backups from Saturday and Sunday. This results in only 5 days, which does not meet our day's retention requirements of 7. So on Monday when the data aging operation runs (default 12PM daily) there will now be 7 days and 2 cycles which will allow the first cycle to be aged.

Retention has been defined for 7 Days and 2 Cycles. When the data aging operation runs on Saturday, the cycles criteria has been met but not the days criteria

Retention has been defined for 7 Days and 2 Cycles. When the data aging operation runs on Monday both Cycles and Days criteria have been met and the first cycle will be marked as aged

Rule 3: Day is based on a 24 hour time period

A day is measured as a 24 hour time period from the start time of a data protection job. Days are considered constants, since regardless of a backup being performed or completed successfully, the time period will always be counted. If a backup fails, backups are not scheduled, or if power goes out, a day will still count towards retention. Therefore it is so critical to measure retention in cycles and days. If retention was just managed by days and no backups were run for a few weeks, all backup data may age off leaving no backups.

Example: Defining retention in both days and cycles is very important. For example, during a Friday night backup power is lost in the building. Power is restored on Sunday resulting in two days elapsing and counting towards retention. Note that since the Friday full backup failed, the cycle continues into the next scheduled full (following Friday).

A failure of a full backup on Friday due to a power outage results in a cycle continuing until a valid full is completed

Additional Storage Policy Copy Retention Rules

Spool Copy

Right-click the primary storage policy copy | Click Properties | Retention tab

The Spool Copy option is used for fast disk read/write access and its multi-streaming capabilities – when there is limited capacity available on the disks. A spool copy is not a retention copy. Data is spooled to disk and then copied to a secondary copy. Once the data is successfully copied to the secondary copy, the data on disk is pruned, immediately freeing up space for new backups.

The Spool Copy option is not available when using deduplication.

Extended Retention

Right-click the desired storage policy copy | Click Properties | Retention tab

Standard retention allows you to define the length of time based on cycles and days that you want to retain data. Extended retention allows you to define specific retention in days that you want to keep full backups for. It allows you to extend the basic retention by assigning specific retention to full backups based on criteria configured in the extended retention settings. Basically, it allows you to set a grandfather, father, son tape rotation scheme.

Extended retention rules are not designed to be used with disk storage and will have significant negative effects on aging and pruning of deduplicated data.

Example: You want to retain backups for 4 cycles and 28 days. You also want to retain a monthly full backup for three months, a quarterly full backup for a year, and a yearly full backup infinitely.

To accomplish this, you configure retention as follows:

• Standard retention is set for (4,28)
• Extended retention is configured for:
  • 90 days keep monthly full backups
  • 365 days keep quarterly full backups
  • An infinite keep yearly full backup

Extended retention rules are like selective copies in that they only apply to full backups. However, a selective copy creates an additional copy of a full backup and assigns it a specific retention. Extended retention applies retention to an existing full backup and does not create an additional copy. Determine which solution is more appropriate when planning retention strategies.

Managed Disk Space

Right-click the desired storage policy disk copy | Click Properties | Retention tab

Managed Disk Space is a feature used with disk libraries that do not use Commvault^® deduplication. This feature allows data to reside on the disk beyond its retention settings and increases the chance of recovering data faster from primary storage on disk without changing retention settings. Managed data on disk is treated the same as retained data for data recovery.

Managed data is held on the disk beyond the standard retention settings until an upper threshold is reached. A monitoring process detects data exceeding the upper threshold and then deletes aged jobs from the media until a lower threshold is reached.

It is important to note that only aged jobs will be pruned. If all aged jobs are pruned and the lower threshold is not met, then pruning will not occur. As a rule of thumb, the upper threshold should be set to allow one hour of backups to run after the threshold is reached. The lower threshold should be set so that the Managed Disk Space pruning operation will not run more than once in a backup time-period, as the pruning operation will have a negative effect on the performance of backups.

Managed disk thresholds are configured in the disk library properties and can be enabled in each storage policy copy. For more information on configuring library settings for managed disk space, refer to the Commvault Online Documentation.

Zero Cycle Retention

It is possible to configure a storage policy copy for a zero-cycle retention. However, this can cause undesired results where data is pruned, but not stored. This is a common mistake when administrators don't fully understand how Commvault cycles and days retention works.
It is NOT recommended to set zero cycles for a policy copy unless another copy has been configured with at least one cycle defined. 

Applying Retention at the Job Level

Typically, retention is based on company policy and therefore managed through storage policy retention settings that affect all data being managed by the policy. There may be situations where jobs retention need to be individually set. There are two methods to apply retention to specific jobs: through schedules or through storage policy copy job history.

Retention Set Through Schedules

Edit the backup job or schedule| Click Advanced | Media tab

Retention can be extended beyond the defined storage policy primary copy retention through a schedule or schedule policy. This is done by setting the Extend Job Retention options in the Media tab of Advanced Options. You can set schedule-based retention for a specified number of days or infinitely retain the data. Retention settings at the schedule level cannot be shorter than the retention defined in the storage policy primary copy.

Retention Applied to Job in Policy Copy

Right-click the storage policy copy | Click View | Jobs | Right-click the desired job | Click Retain Job

Retention for a job in a primary or secondary storage policy copy can be retroactively changed in the job history of the copy. The job can be retained infinitely or until a specific date. The job icon changes to reflect that the job has been 'pegged down'.