P re C og: Exploring the Relation between Memorization and Performance in Pre-trained Language Models (2023)

Abstract: Large Language Models (LLMs) are impressive machines with the ability to memorize, possibly generalized learning examples. We present here a small, focused contribution to the analysis of the interplay between memorization and performance of BERT in downstream tasks. We propose PreCog, a measure for evaluating memorization from pre-training, and we analyze its correlation with the BERT’s performance. Our experiments show that highly memorized examples are better classified, suggesting memorization is an essential key to success for BERT.

## introduction large language models (llms) @xcite are intriguing machines dominating the arena of nlp tasks with their ability to memorize generalizations of texts in synthetic neurons. after long pre-training on large amounts of unlabeled data, llms have been shown to learn effectively downstream tasks with limited labeled data @xcite and generalize in out-of-distribution examples @xcite . extensive studies have shown that these models tend to mimic traditional linguistic syntactic models @xcite and traditional nlp. hence, a crucial issue is to clarify why pltms exploit pre-training better than traditional nlp modules exploit annotated corpora. understanding the learning process of llms may help in understanding their results in downstream tasks and in improving their linguistic representations in scenarios where they fail @xcite . indeed, unlike traditional general nlp modules in pipelines, llms need to be fine-tuned for the specific tasks @xcite and, eventually, domain-adapted on the specific language of the novel corpus @xcite . moreover, as with many other machine learning models, finetuned ptlms lose their ability to solve a task if subsequently fine-tuned to another task @xcite although they apparently do not change their language models @xcite . this phenomenon is known as catastrophic forgetting @xcite in machine learning. then, it is still unclear how these models exploit pre-training and training examples. llms, such as bert @xcite , have shown to have an impressive ability to memorize and possibly generalize learning examples. this ability has been largely investigated as it may be extremely harmful. in fact, these models may reveal sensitive information that has been acquired during pre-training. for example, memories of gpts @xcite have been violated and produced phone numbers, and usernames @xcite . however, this simple ability to memorize may play a crucial role in the performances of llms in downstream tasks @xcite . this paper presents a small, focused contribution to the role of memorization in the performance of bert in downstream tasks. we propose precog, a very simple measure of coverage that evaluates how much pre-training covers the information needed to model a given example or, better, if bert has already partially seen the example -it pre-cognizes the example. the aim is to evaluate if precog precognizes which examples bert adapted to a downstream task performs better inferences. we have extensively experimented with precog by using bert over the glue tasks @xcite ), and we observed the ability of precog to predict examples where a task-adapted bert performs better. besides being a predictive measure, precog showed that example memorization is a crucial part of the success of llms. ## related work the ability of linguistic neural models to memorize facts is out of doubt @xcite . this ability has been deeply explored as it is a problem for privacy issues. indeed, lstm language models remember facts so well that individual facts can be retrieved during inference @xcite . these facts may reveal sensitive personal information such as names and addresses associated with people. moreover, revitalizing the idea of sparse distributed memories @xcite , @xcite hypothesized that large language models might be used as clever and inexpensive ways to build up effortlessly knowledge bases. even in other areas like image classification, it appears that large neural networks may memorize entire datasets as these networks achieve very low error rates over datasets with randomly generated target labels @xcite . this also proves to be a problem for the de-biasing phenomenon @xcite . yet, it is still unclear to what extent this ability to memorize facts helps neural networks in downstream tasks. a key research question is to understand how large pre-trained neural networks generalize over memorized examples. pre-training seems to be a winning strategy to boost generalization. in fact, pre-trained models generalize better on outof-distribution data and can detect such data better than non-pre-trained methods @xcite . however, these models need a significant number of training instances to exploit this generalization ability in downstream tasks @xcite . hence, since fine-tuning on specific datasets seems to be connected to catastrophically forgetting examples @xcite , generalization and memorization can be strictly correlated. to explore the correlation between memorization and performance on downstream tasks, we propose a mechanism for analyzing sentence coverage. in particular, we investigate how many sentences are seen in the pre-training phase in transformerbased plms using perturbation masking methods. these methods allow us to observe the impact of pre-training on the performance of downstream tasks. this novel measure is needed as current measures for understanding coverage, such as "forgetting event" @xcite and counterfactual memorization @xcite , mix performance, and actual memorization. ## method and data this section introduces precog, which is our measure to evaluate how much pre-training covers the information needed to model a given example (sec. 3.1), two comparative measures lenght and lexcov (section 3.2), and the experimental setting (section 3.3). ## experimental results and discussion accuracies reported in fig. 1a and fig. 1c and used in tab. 2 are the weighted sum of accuracies in each glue task. this guarantees that the 20-point bins have a sufficient set of samples to compute stable accuracies. precog correlates with the accuracy of bert f t better than lenght and lexcov (see fig. 1a and tab. 2). accuracies of precog in the different bins degrade more uniformly than the other two measures (red solid line in fig. 1a ). moreover, the pearson's correlation between precog values and the accuracies of bert f t is 0.9737 with a p-value of 0.005 and it is higher than the ones of both lexcov, 0.9014 with a p-value of 0.037, and length which is not correlated (see tab. 2 ). precog values better separate examples in testing sets. at first glance, lexcov may seem a better model to separate samples with high with respect to those with fewer accuracy expectations. samples with a value of lexcov less than 40 have low accuracy (see fig. 1a ). however, samples having lexcov between 0 and 40 are rare (fig. 1b ). better observations are derived by plotting accuracies over bins rescaled according to their coverage (fig. 1c ). indeed, precog separates samples better than lexcov (red solid line vs. dashed blue line in fig. 1c ): samples from 18,000 to 55,000 fall in two bins for precog and in only one bin for lexcov. hence, precog has better discriminative power than lex-cov. results are substantially confirmed on task basis: precog is a better predictor of the accuracy on tasks and a better separator of classes of samples (see tab. 1 ). accuracies of bert f t are generally higher for samples with precog in the interval @xcite than for samples with the other two measures in the same interval. lexcov has higher accuracy for samples in [80, 100] only for rte. moreover, accuracies of samples in the interval [80, 100] are always higher than those in the interval [0, 80] for both precog and lexcov. yet, precog partitions more evenly samples, and the differences in accuracies between intervals [80, 100] and [0, 80] are generally higher. moreover, domain adaptation is not changing the above findings. accuracies for bert da are generally higher than those without domain adaptation for all the tasks except for sst2 and wnli (tab. 2). moreover, focusing on precog, the overall increase in accuracies in cola, mnli, and rte derives from an increase in the samples of the interval @xcite . this fact suggests that bert da is gaining a better model for these samples. as a final observation, bert seems to behave better on sentences that have been, at least, partially seen during pre-training. indeed, precog is a measure capturing how much the sentence is covered with the pre-training task masked language model (mlm). typically, bert overfits mlm during pretraining. then, precog is a measure telling whether sentences have already been partially seen. instead, lexcov describes how many words of sentences are covered by bert's vocabulary. since there is a great difference in predicting accuracy on tasks between precog and lexcov, we can conclude that bert behaves better when general knowledge of the target sentence is already acquired during pre-training. ## conclusion memorization of pre-training examples plays a very important role in the performance of bert. indeed, our precog, which measures how much memorized pre-training knowledge cover target examples, is highly correlated with bert's performance in inference. precog can also be used to measure confidence for bert-based decisions in downstream tasks. as bert success is partially due to simple memorization of examples and given the overwhelming presence of chatgpt, one area of future research should be on better understanding the relation between actual training examples and inferences in order to give credit to knowledge producers.