Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (2021)

Abstract: Public datasets are often used to evaluate the efficacy and generalizability of state-of-the-art methods for many tasks in natural language processing (NLP). However, the presence of overlap between the train and test datasets can lead to inflated results, inadvertently evaluating the model’s ability to memorize and interpreting it as the ability to generalize. In addition, such data sets may not provide an effective indicator of the performance of these methods in real world scenarios. We identify leakage of training data into test data on several publicly available datasets used to evaluate NLP tasks, including named entity recognition and relation extraction, and study them to assess the impact of that leakage on the model’s ability to memorize versus generalize.

## introduction shared tasks that provide publicly available datasets in order to evaluate and compare the performance of different methods on the same task and data are common in nlp. held-out test sets are typically provided, enabling assessment of the generalizability of different methods to previously unseen data. these datasets have played a key role in driving progress in nlp, by defining focus tasks and by making annotated data available to the broader community, in particular in specialized domains such as biomedicine where data can be difficult to obtain, and quality data annotations require the detailed work of domain experts. examples of tasks where benchmark data sets exist include open domain question answering (qa) @xcite and biomedical named entity recognition @xcite . in the context of machine learning models, effectiveness is typically determined by the model's ability to both memorize and generalize @xcite . a model that has huge capacity to memo-rize will often work well in real world applications, particularly where large amounts of training data are available @xcite . the ability of a model to generalize relates to how well the model performs when it is applied on data that may be different from the data used to train the model, in terms of e.g. the distribution of vocabulary or other relevant vocabulary. the ability to memorize, taken to the extreme, can be considered equivalent to an exact match lookup table @xcite and the ability to generalize captures how well it can deal with degrees of variations from the lookup table . an effective combination of memorization and generalization can be achieved where a model selectively memorizes only those aspects or features that matter in solving a target objective given an input, allowing it to generalize better and to be less susceptible to noise. when there is considerable overlap in the training and test data for a task, models that memorize more effectively than they generalize may benefit from the structure of the evaluation data, with their performance inflated relative to models that are more robust in generalization. however, such models may make poor quality predictions outside of the shared task setting. the external validity of these evaluations can therefore be questioned @xcite . in this paper, we assess the overlap between the train and test data in publicly available datasets for named entity recognition (ner), relation extraction (rel) and text classification (cls) tasks, including sst2 @xcite , biocreative @xcite and aimed @xcite datasets, and examine the significant impact of not taking into account this overlap on performance evaluation. we argue that robustness in generalization to unseen data is a key consideration of the performance of a model, and propose a framework to examine inadvertent leakage of data between data set splits, in order to enable more controlled assessment of the memorization vs. generalization characteristics of different methods. ## related work the issue of memorization vs. generalization has been previously discussed in the context of question answering datasets, where, given only a question, a system must output the best answer it can find in available texts. @xcite identify 3 distinct issues for open domain qa evaluation: a) question memorization -recall the answer to a question that the model has seen at training time; b) answer memorization -answer novel questions at test time, where the model has seen the answer during training; and c) generalization -question and answer not seen during training time. they find that 58-71% of test answers occur in the training data in 3 examined data sets, concluding that the majority of the test data does not assess answer generalization. they also find that 28-34% have paraphrased questions in training data, and a majority of questions are duplicates differing only by a few words. similarly, min (2020) identified repeating forms in qa test sets as a problem. the work proposed a novel template-based approach to splitting questions into paraphrase groups referred to as "templates" and then controlling train/test data splits to ensure that all questions conforming to a given template appear in one segment of the data only. this was tested on the emr clinical question answering dataset emrqa @xcite and the overnight dataset @xcite ; it was demonstrated that models perform significantly worse on test sets where strict division is enforced. this paraphrase-based splitting methodology was also employed in their recent work on emrqa @xcite . ## approach a common practice to create a train and test set is to shuffle data instances in a dataset and generate random splits, without taking into account broader context. however, this can inadvertently lead to data leakage from the train set to test set due to the overlaps between similar train and test instances. the type of overlap between train and test dataset depends on the type of the nlp task. generally speaking, the leakage can occur either in the algorithm 1 compute overlap 1: procedure compare(testset, trainset) 2: ## conclusion we conclude that quantifying train/test overlap is crucial to assessing real world applicability of machine learning in nlp tasks, given our reliance on annotated data for training and testing in the nlp community. a single metric over a held-out test set is not sufficient to infer generalizablity of a model. stratification of test sets by similarity enables more robust assessment of memorization vs. generalization capabilities of models. further development of approaches to structured consideration of model performance under different assumptions will improve our understanding of these tradeoffs.