Storage Dedupe Reaches The Mainstream

DEMYSTIFYING DEDUPE
Deduplication, also called "dedupe," removes duplicate information as data is backed up or archived. It can be done at the file level, where duplicate files are replaced with a marker pointing to one copy of the file, and/or at the subfile or byte level, where duplicate bytes of data are removed and replaced by pointers, resulting in a significant decrease in storage capacity requirements.

Dedupe products can be classified according to where the de-dupe process takes place. Source dedupe dedupes the data before it is sent across a LAN or WAN. Doing so results in fewer files and less data being sent over the network, but it can also affect the performance of the backup because of the processing overhead caused by the dedupe process.

Post-processing dedupe starts the dedupe process after the data is copied onto a destination device such as a virtual tape library. This mitigates the bottleneck by accepting the full data set and then eliminating duplicates as it is stored, but requires more storage capacity to temporarily store the entire data set.

In-line dedupe uses a separate appliance, which sits between the data source and the data target to dedupe the data as it is moved. This cuts the performance and capacity requirements, but adds a new appliance that needs to be managed.