Data Similarity is Not Enough to Explain Language Model Performance (2023)

Abstract: Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model’s pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.

## introduction large language models (lms) are pretrained on large datasets, such as c4 @xcite and the pile @xcite , and used for downstream tasks. many hypothesize that task performance is a matter of how similar the task data is to the pretraining data @xcite @xcite . despite the prevalence of this "similarity hypothesis," it has not yet been evaluated beyond data overlap @xcite . can we measure the impact of data similarity? similarity matters at a coarse level: models pretrained on english data do not perform as well on other languages @xcite . robustness to distribution shifts has been extensively studied @xcite , though dissimilarity is usually assumed through dataset construction rather than explicitly measured. we measure distributional similarity between pretraining datasets and benchmarks using a 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 kl-divergence from pretraining token distribution 30 40 50 60 70 80 english finnish spanish french german indonesian swahili normalized score stack exchange forum classification 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 kl-divergence from pretraining token distribution 20 10 0 10 20 normalized score big-bench lite multiple choice tasks range of similarity metrics, finding that similarity correlates with performance in a controlled setting but not across common benchmarks (figure 1 ). example-level similarity offers a more targeted hypothesis that controls for task difficulty: within a downstream dataset, examples that are similar to examples in the pretraining data should be easier than dissimilar examples. it has been shown that examples with words and entities more frequently found in the pretraining dataset (e.g. obama or the number 24) are more often answered correctly in arithmetic and factoid qa tasks @xcite @xcite . we ask a stronger version of this question: will a model perform well on an example if that example is similar to any pretraining document? we test the hypotheses that distributional similarity and example-similarity determine performance using three measurements of similarity from prior work: word similarity, embedding similarity, and language model perplexity. our contributions are the following. first, we show that similarity correlates with performance for multilingual datasets. the same underlying task is represented in different languages, each with a different aggregate similarity to the pretraining dataset, and languages that are more similar to the english pretraining data afford higher performance. second, we show that multiple measures of distributional similarity are surprisingly not correlated with zero-shot performance on big-bench lite multiple-choice tasks @xcite across models and pretraining datasets. third, we show that within a downstream dataset, examples that are more similar to any example in the pretraining dataset are not easier than less-similar examples. we do this by comparing examples from glue @xcite and big-bench lite with all of c4 and the pile. fourth, we find that the similarity measures are not even correlated with each other. we conclude with several examples that show the limits of similarity's predictive power. although similarity is a common post-hoc justification for performance, we find it is misleading. this indicates the importance of model-based interpretability approaches, like training data attribution @xcite . foot_0 ## related work the impact of pretraining data on downstream performance is an active area of research. similarity to a model's pretraining dataset is assumed to determine the model's performance on tasks @xcite @xcite . prior work has focused on repeated entities in arithmetic @xcite and qa tasks @xcite . finetuned downstream performance benefits from continued pretraining on a similar domain @xcite ). 2 contemporaneous work quantifies task similarity to downstream training datasets @xcite . task data leakage is an extreme case of data similarity, where the presence of task examples in the pretraining dataset impacts downstream performance on that task @xcite . lm perplexity has been assumed to be a proxy for similarity to pretraining data @xcite , though it is poorly calibrated for downstream tasks @xcite . training data attribution uses influence functions, gradients, and model internals rather than data similarity @xcite @xcite . other approaches quantify task difficulty but do not explicitly account for the role of pretraining data @xcite @xcite @xcite . ## similarity correlates with performance in a controlled setting the similarity hypothesis implies that a downstream dataset's similarity to the pretraining data should matter. as an initial check of this hypothesis, we construct multiple versions of a task. the underlying task is held constant, but we decrease the similarity of the data to the training set in a way that we expect to decrease lm performance. datasets in multiple languages are a starting point because text in non-english languages is presumably less similar to english pretraining data. while popular english pretraining datasets do contain non-english text that enables some transfer to other languages, lms trained on english do not perform as well on multilingual tasks @xcite @xcite . we consider two tasks that have datasets in english and several other languages, where the task is fixed across languages. for an explicitly easy task, we extend a stack exchange forum classification dataset @xcite ) by translating it from english into multiple languages. for a harder task, we use xnli @xcite . details are in appendix a. we measure zero-shot performance of pythia-6.9b @xcite , which was pretrained on the pile @xcite , a large dataset of mostly english text. figure @xcite shows that performance tracks the kl-divergence of unigram token distributions between the pile and stack exchange datasets. the english dataset is the most similar dataset to the pile and affords the highest performance. all other languages are much less similar to the pile and have lower performances. note that token distribution similarity does not allow fine-grained comparisons between the languages with high kl-divergence. we also test whether performance smoothly degrades as we titrate in a given amount of dissimilar- ## measuring similarity having established that distributional similarity correlates with performance in a setting that holds task difficulty constant, can we say the same across different benchmark tasks? and is performance higher for more similar examples? if the similarity hypothesis holds more generally, so that similarity to training examples can consistently predict performance on tasks, we expect that multiple measures of similarity will correlate with accuracy. similarity measures. we implement three categories of similarity measures on different text representations: first, kl-divergence of n-gram distributions. second, embedding-based similarities using sentence-t5 embeddings @xcite , including: (a) mauve score @xcite of a downstream dataset against samples of the pretraining dataset, (b) the maximum cosine similarity between a downstream example and the entire pretraining dataset, and (c) the mean cosine similarity between a downstream example and the closest 1,000 neighbors in the entire pretraining set. third, language modeling loss of a pretrained model evaluated on a downstream example.we use the perplexity of the input sequence and the perplexity of the correct target conditioned on the input. we further measure similarity at two different scales: 1) aggregate similarity between an entire downstream dataset and the pretraining dataset, and 2) example-level similarity between an individual downstream example and the pretraining dataset. cosine similarities and lm loss are example-level, so we take the mean over a dataset's examples for aggregate similarity. see appendix b for details. ## results tasks that are more similar to the pretraining data do not have higher few-shot performance. we first use big-bench lite to examine whether datasets that are more similar in aggregate to the pretraining data are easier for a given model. if the similarity hypothesis is true, we expect to see significant high-magnitude correlations between performances and similarities to the corresponding pretraining dataset. examples that are more similar to any pretraining document are not easier. we can control for different task difficulties by focusing on individual examples within a dataset. if the example-level similarity hypothesis is true, then we expect to see that within a task, examples that are more similar to the pretraining dataset are classified correctly more frequently than less similar examples. figure 3 shows that big-bench lite examples classified correctly by pythia-6.9b do not have significantly higher embedding cosine similarity to any document in the pile than those that were misclassified. this holds for all the tasks we examined. figure 6 in the appendix shows that accuracy of more similar examples is not consistently higher than accuracy of dissimilar examples. figure 7 in the appendix 4 the probability of any false positives is at most 0.05. mnli matched mnli mismatched qnli qqp sst2 rte dataset 0.85 0.90 0.95 1.00 max cosine similarity similarity to pretraining data finetuned t5 prediction correct incorrect mnli matched mnli mismatched qnli qqp sst2 rte dataset 0.85 0.90 0.95 1.00 similarity to finetuning data shows this result for t5 performance and maximum similarity to c4 documents. we also test the hypothesis in the finetuning paradigm by using embedding similarity and t5 3b finetuned on glue. here we compare examples from the validation split to both the pretraining dataset and the glue training split. figure 4 shows no correlation between performance on an example and that example's maximum similarity to documents in the pretraining or finetuning datasets. ## discussion our goal is to evaluate the similarity hypothesis: that tasks that are more similar to pretraining data are easier. we have so far shown that similarity does not in practice correlate with performance for common benchmark tasks. in this process we found additional reasons to doubt the hypothesis. task difficulty can be orthogonal to similarity. tasks can have language similar to the pretraining corpus and still be hard. to underscore that similarity is not sufficient to predict performance, we use another stack exchange classification task. rather than predicting whether a given post was from the bicycles or cstheory forum, we instead choose a much more difficult task: predicting whether it was posted before or after noon. again, we translate the dataset into a range of other languages. by construction, the similarity to the pretraining data is nearly the same as in figure 1 , but the accuracy is random (figure 8 , appendix). similarity measures are uncorrelated. do different similarity measures capture the same notion of similarity? surprisingly, different measures of aggregate similarity between downstream datasets and pretraining datasets are not significantly correlated with each other when comparing big-bench lite tasks with c4 and the pile ( scope of our findings. we do not disprove the similarity hypothesis. one possibility is that all datasets in big-bench lite and glue are so similar to c4 and the pile that similarity differences are not great enough to impact performance. our experiments are on models up to the 7b-parameter scale. perhaps there is a model scale threshold at which similarity matters more for performance. it is clear that at least some level of performance variation is necessary to see if similarity matters. for example, a model that performs poorly on all tasks (or conversely if it solves all tasks perfectly) will not be able to show any meaningful influence of similarity. and while we evaluate a broad range of similarity measures, they do not capture all possible similarities. rather, our results make it increasingly unlikely that similarity-broadly construed-is the single most determinative factor in this setting. ## conclusion at a broad level, it is clear that pretraining data matters. data curation can impact robustness, memorization, and toxicity generation @xcite @xcite . specific examples of high-level task similarity, such as prevalence of entity mentions, have been observed to correlate with performance on arithmetic and qa tasks @xcite . we find that while similarity is correlated with performance when task difficulty can be kept constant, it is surprisingly hard to connect textual similarity to performance for standard benchmarks in the few-shot setting. both distributional and finegrained example-based similarity to the pretraining dataset are not enough to explain performance across benchmarks, and this remains true for every future work. future work can study the advantages of training data attribution over raw similarity, as begun by @xcite . pretraining on datasets constructed to be more or less similar to specific downstream tasks might also be able to determine if there is a causal relationship between similarity and performance.