An Empirical Study of Memorization in NLP (2022)

Abstract: A recent study by Feldman (2020) proposed a long-tail theory to explain the memorization behavior of deep learning models. However, memorization has not been empirically verified in the context of NLP, a gap addressed by this work. In this paper, we use three different NLP tasks to check if the long-tail theory holds. Our experiments demonstrate that top-ranked memorized training instances are likely atypical, and removing the top-memorized training instances leads to a more serious drop in test accuracy compared with removing training instances randomly. Furthermore, we develop an attribution method to better understand why a training instance is memorized. We empirically show that our memorization attribution method is faithful, and share our interesting finding that the top-memorized parts of a training instance tend to be features negatively correlated with the class label.

## introduction in recent years, there has been an increasing amount of interest in the machine learning community to understand the memorization behaviour of deep neural network models. studies have shown that deep learning models often have sufficient capacities to "memorize" training examples @xcite . a number of recent studies tried to understand how memorization helps generalization @xcite @xcite @xcite in nlp, memorization of training examples by deep learning models is also often observed @xcite @xcite , and existing studies usually see memorization as something that hinders generalization. for example, @xcite tried to measure the amount of "data leakage" in nlp datasets in order to assess a model's ability to memorize vs. its ability to generalize. however, recently @xcite proposed a long-tail theory, which states that memorization is necessary for generalization if the data follows a long-tail distribution. this theory was later empirically validated by @xcite , but their validation was done in only the computer vision domain. it is therefore interesting and useful for us to study whether the long-tail theory also holds in nlp; such validation would help us better understand the utility of memorization in the context of nlp. the long-tail theory states that if the training data form a long-tail distribution, where there are many small "sub-populations" that are atypical instances, and if these small sub-populations are also present in the test data, then memorizing these atypical instances helps the model generalize to the test data. in order to validate this long-tail theory in the context of nlp, we follow the experiments and analyses on image classification done by @xcite . specifically, we aim to answer the following questions in this paper: (1) on a few typical nlp tasks, are the training instances memorized by deep learning models indeed atypical instances? (2) does memorizing these training instances lead to lower generalization error on the test instances? in addition, observing that it is not always straightforward to understand why a training instance is being memorized, we study the following novel research question: (3) can we provide some explanation about why a training instances is memorized? to be more specific, can we attribute the memorization score of a training instance to its individual tokens such that we can quantify which tokens require the most memorization by the model? to answer these research questions, we first adopt self-influence @xcite as our memorization scoring function. compared with the estimator proposed by @xcite , our self-influence function is also theoretically mo-tivated but has the advantage that it is easy for us to derive a memorization attribution method for the third research question above. we present the self-influence function in section 2.1, and in section 2.2, we present our novel memorization attribution method. we conduct experiments on three nlp tasks: sentiment classification, natural language inference (nli) and text classification. our experiments and analyses demonstrate that the training instances with the highest memorization scores tend to be atypical, at least on sentiment classification and nli. on all three tasks, we find that removing the top-memorized training instances results in significantly dropped test performance, and the drop is markedly higher compared with removing a random subset of training instances. we also evaluate our memorization attribution method and find that our method can indeed identify input tokens that require the most memorization. finally, we apply our memorization attribution method to sentiment classification and to an image classification dataset, and we share the interesting finding that the highly-memorized input features tend to be those that are negatively correlated with the class labels. our code and data are available at https://github.com/ xszheng2020/memorization . ## our approach to validate the long-tail theory in the context nlp, let us first review the main claims of the theory. first, the long-tail theory hypothesizes that training instances with the same class label has a long-tail distribution, with instances at the tail end being those atypical instances that need to be memorized. to verify this assumption, we first identify those training instances that are memorized by a trained deep learning model and then check if they are indeed atypical. specifically, we follow feldman and zhang (2020) and adopt "self-influence" to measure memorization, but we use the influence function proposed by @xcite to define self-influence. second, the long tail theory states that memorization of the atypical training instances leads to lower generalization error, because the atypical training instances belong to subpopulations that also have presence in the test data. to verify this statement, we check whether removing the memorized training instances would lead to more significant performance drop on the test data than removing a random sample of training instances. it is worth noting that the approach outlined above follows the experiments conducted by feldman and zhang (2020) to validate the long tail theory on image classification. furthermore, we want to pinpoint which parts of a memorized instance are most critical for memorization. in other words, since each training instance is assigned a memorization score, can we attribute the memorization score to different parts of the input of this instance? this presumably can help us better understand which parts of the input need to be memorized the most. we follow the idea from integrated gradients (ig) @xcite and derive a formula to compute memorization attribution. ## experiments with the memorization score defined in eqn 2 and the memorization attribution score defined in eqn 5, we now conduct experiments to answer the three research questions raised in section 1. ## related work the long-tail theory: the long-tail theory proposed by @xcite is relatively new and has not been systematically validated in nlp. our work is the first to empirically check the validity of this theory on nlp tasks. @xcite used the long-tail theory to explain hallucinations under source perturbations in neural machine transla-tion. they assume the theory holds in nmt rather than validating the theory itself as we do. @xcite investigated the memorization phenomenon for variational auto-encoder also via self-influence. memorization vs. generalization: it is wellknown that deep learning models possess strong capabilities to memorize training instances @xcite . in the context of nlp, @xcite showed that bert is more likely to memorize shallow patterns from the training data rather than uncover abstract properties. some recent work has tried to combine interpolation methods with deep learning methods to generalize via memorization @xcite . however, previous work using interpolation methods did not explain why memorization is necessary in the first place. our work follows the long-tail theory that views memorization as beneficial to generalization when the data follows a certain type of long-tail distribution. there has also been some work studying "forgetting," which is related to memorization @xcite . however, in this paper we do not study this "forgetting" phenomenon. influence functions: influence functions have been studied for large-scale deep neural networks by @xcite and gained much attention in recent years. in the context of nlp, @xcite explored the usage of influence functions to explain model predictions and unveil data artifacts. @xcite proposed a combination of gradient-based methods and influence functions to examine training history and test stimuli simultaneously. our work, however, adopts influence function as a tool to measure memorization. ## conclusions in this paper, we empirically examine a recently proposed long-tail theory in the context of nlp. we use sentiment classification, natural language inference and text classification to check the validity of the long-tail theory in nlp. we also propose a memorization attribution method to reveal which parts of an instance are being memorized. there are two major takeaway messages: (1) our experiments empirically validated the long-tail theory on the three nlp datasets, showing that memorization is important for generalization, offers an alternative view and helps nlp researchers to see the value of memorization. (2) our attribution method can be a tool to help model developers better understand the memorization behaviours of a model and possibly further improve the model.