S oft D edup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training (2024)

Abstract: The effectiveness of large language models (LLMs) is often hindered by duplicated data in their extensive pre-training datasets. Current approaches primarily focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication. To address this, we propose a soft deduplication method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. Central to our approach is the concept of “data commonness”, a metric we introduce to quantify the degree of duplication by measuring the occurrence probabilities of samples using an n-gram model. Empirical analysis shows that this method significantly improves training efficiency, achieving comparable perplexity scores with at least a 26% reduction in required training steps. Additionally, it enhances average few-shot downstream accuracy by 1.77% when trained for an equivalent duration. Importantly, this approach consistently improves performance, even on rigorously deduplicated datasets, indicating its potential to complement existing methods and become a standard pre-training process for LLMs.

## introduction in recent years, the expansion of pre-training datasets has played a pivotal role in advancing llms @xcite @xcite . however, a large fraction of these datasets is derived from uncurated snapshots of the internet, resulting in a significant amount of duplication. such redundancy, particularly beyond certain levels, can severely impair the performance of llms @xcite . while repetition under specific conditions may be beneficial, mars is a planet in the solar system, orbiting around the sun. it takes 687 earth days to complete one orbit. earth is a planet in the solar system, orbiting around the sun. it takes 365 days to complete one orbit. mars is a planet in the solar system, orbiting around the sun. it takes 687 earth days to complete one orbit. ## conclusion in this study, we introduce a soft deduplication method that effectively addresses the primary limitations associated with traditional hard dedupli ubuntu irc subset, we exclude it from the calculation of the average perplexity on the pile test set. a.2 average perplexity for each subset in the slimpajama test set in