Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning (2024)

Abstract: Through pretraining on a corpus with various sources, Large Language Models (LLMs) have gained impressive performance. However, the impact of each component of the pretraining corpus remains opaque. As a result, the organization of the pretraining corpus is still empirical and may deviate from the optimal. To address this issue, we systematically analyze the impact of 48 datasets from 5 major categories of pretraining data of LLMs and measure their impacts on LLMs using benchmarks about nine major categories of model capabilities. Our analyses provide empirical results about the contribution of multiple corpora on the performances of LLMs, along with their joint impact patterns, including complementary, orthogonal, and correlational relationships. We also identify a set of “high-impact data” such as Books that is significantly related to a set of model capabilities. These findings provide insights into the organization of data to support more efficient pretraining of LLMs.

## introduction under the data-driven paradigm, large language models (llms) have demonstrated promising performance and showcased immense potential in further promotion @xcite @xcite . previous analyses have suggested that the composition of the pretraining corpus may exert a significant impact upon the performance of llms @xcite . however, how different sources and types of pertaining corpora influence the knowledge and reasoning ability of llms largely remains opaque, or stays at the qualitative level. as a result, it still heavily relies on the experiences of trainers to organize the pre-training corpus. such experiences may deviate from the optimal, and hence limit the efficiency and effectiveness of model training. in this paper, we propose to quantify how components with different sources and types in the pretraining corpus contribute to the performance of llms. previous literature refers to such analyses as data influence analysis (dia) @xcite . however, due to the limitations of previous dia methods, the dia of llms remains challenging. primary dia methods can be mainly categorized into two lines: the retraining-based methods and gradient-based methods. retraining-based methods work by removing specific data from the training corpus and retraining the model, then comparing changes in model performance. considering the prohibitive training cost of llms, retrainingbased methods would be impractical @xcite . while the gradient-based methods may not be applicable in analyzing the source of the complex reasoning ability of llms, as they assume that the performance upon a test instance is determined by several independent training instances. however, such an assumption may not hold for llms, especially for the ability to complete reasoning tasks, as it may originate from groups of correlated instances that jointly contribute to the performance of llms. for example, solving math problems requires understanding a knowledge taxonomy, and the taxonomy is described by a set of interdependent instances holistically. missing one component would lead to the collapse of the whole taxonomy. hence, the gradient-based methods may fail to trace the influence of such a whole corpus, which is of vital importance @xcite . to address these issues, we resort to another strand of method, machine unlearning. prior research @xcite suggests that machine unlearning can selectively erase specific knowledge from a model through gradient ascent on corresponding instances. this enables us to investigate the influence of a certain pretraining corpus on an llms by "unlearning" instances from it, and then compare the performance of the "forgotten" llms with the original llms. meanwhile, different from previous machine unlearning methods, to avoid unintended impacts on non-targeted samples, we incorporate additional regularization by retraining samples from non-targeted domains during the unlearning process. experiments demonstrate that our method can effectively remove information contained in the target samples, without significantly affecting other unrelated samples. based on our customized machine unlearning method, we systematically investigated the quantitative contribution of multiple important resources and types of training corpora on the performance of llms. we covered widely adopted high-quality corpora github, wikipedia, arxiv, books, stack-exchange and c4, and conducted an in-depth analysis of their contributions to the model performance by segmenting them into subsets based on the type of knowledge. from the content dimension, the abovementioned corpora covered text, commonsense knowledge, domain-specific knowledge, math, and coding. additionally, to investigate the source of the reasoning abilities of llms, we analyzed the impact of over ten kinds of programming languages with different coding paradigms, and 17 kinds of common algorithms such as dynamic programming. programming paradigms essentially represent different abstractions of real-world problems, and an algorithm corresponds to a common solution for a particular type of reasoning problems. to comprehensively assess the impact of these corpora, we evaluated the "forgotten" llms across various downstream tasks as illustrated in figure 1 and detailed illustrated in figure 4 . with our proposed gradient ascent-based machine unlearning method, we identify a range of corpora that have a significant impact on the capabilities of llms, and reveal previously unreported influences of certain corpus (such as algorithms) on model capabilities. furthermore, we discovered the interaction among corpora which jointly affects the capabilities of llms, and the existence of "high-energy" corpora, like books in the data. our research underscores the importance of further studying the impact of pre-training data, to provide foundations for future research on the optimization of pre-training datasets to support a more efficient pre-training process. based on our analysis results, we provide several hints about the organization of the pretraining corpus, such how should the proportion of each category of pretraining corpus be set, the arrangement of datasets during the pretraining process, and the evaluation of the pretraining process of the llms. ## main analysis after the validity analysis, we employ grace to investigate the impact of various corpora on the performance of llms. specifically, section 3.1 introduces the experimental settings. section 3.2 explains how different types of data affect model performance individually. section 3.3 discusses the joint impact of various types of data on the abilities of llms. ## related work the data influence analysis (dia) task aims at finding how each training data contributes to a model's performance. dia methods can be mainly classified into two categories @xcite . the first category, the retraining-based approach @xcite @xcite , assesses the influence of certain instances by comparing the model performance with and without these instances. however, due to the prohibitive training costs, these methods have only been extensively applied in the 'small" models @xcite @xcite . the second category gradient-based methods @xcite @xcite discover training samples with greater influence by comparing gradient similarities between training and test instances. these methods demonstrate efficacy in finding instances contributing to knowledge memorization of models. however, they may fail to find instances related to the reasoning abilities of models, which is especially important for llms, as it may originate from groups of correlated instances that jointly contribute to the performance of llms. considering the limitations of these methods, we propose a machine unlearning-based dia approach to investigate the impact of corpora upon llms. machine unlearning is devised to erase certain knowledge from a model. prior research @xcite @xcite suggests that machine unlearning can selectively erase specific knowledge from a model through gradient ascent on corresponding instances. @xcite has shown that gradient ascent methods would still be effective in llms, and can accurately unlearn targeted samples. in this paper, we further extend the gradient ascent-based machine unlearning methods by involving an additional retraining process and a random text-based stop criterion. ## conclusion in this study, we employed a machine-unlearningbased data influence analysis method grace to investigate the complex effects of diverse types of pretraining data on the performance of large llms. we gained empirical analysis results about how specific components of the pretraining corpus influence llms capabilities, and how they jointly contribute to multiple capabilities of llms. our findings suggest the nuanced impact of data selection and organization in llms development. the identification of high-impact data and the delineation of complementary, antagonistic, and orthogonal data relationships offer guidance for optimizing pre-training data organization. in future work, we consider adapting our analysis methodology to other parts of llms training such as supervised fine-tuning.