Quantifying Train-Evaluation Overlap with Nearest Neighbors (2023)

Abstract: Characterizing benchmark datasets is crucial to interpreting model performance. In this work, we study train-evaluation overlap as a measure of an individual dataset’s adequacy to evaluate model generalization over a wide range of datasets. We quantify the overlap with a simple novel metric based on a nearest neighbors approach between the training and evaluation sets. We identify nearest training examples for each evaluation example by mapping instances with generic and task-specific embedding methods. Our study on eleven classification and extractive QA tasks reveals a wide range of train-evaluation overlap, and we show that the data collection method of the dataset and the difficulty of the task may play a role in the amount of overlap. Lastly, we use our nearest neighbor analysis to identify challenging or potentially mislabeled examples. Our analysis quantifies train-evaluation overlap, providing insights for constructing datasets to study generalization.

## introduction benchmark datasets in nlp @xcite are invaluable for driving and tracking progress in the field. while evaluating on a held-out set of data ideally tests for generalizability to new data, frequent overlap between training and evaluation sets hinders assessing a model's generalization capacity @xcite @xcite . in this paper, we quantify the overlap between the training and evaluation splits in datasets through a simple metric based on a nearest neighbor approach, and analyze datasets along the axis of dataset collection method. we categorize data collection methods frequently used in the literature into four categories, based on how naturally the language is captured; some datasets harvest user generated content (e.g., movie reviews paired with their scores), while language in other datasets is written by crowdworkers to fool existing models @xcite or syn-thetically generated from templates @xcite . we analyze the train-evaluation overlap in eleven nlp datasets varying in data collection method on two tasks -classification and extractive question answering -through a nearest neighbors approach. to quantify the overlap between training and evaluation datasets, we identify the nearest train neighbor to each evaluation example using cosine similarity between the input representations. we experiment with two types of representationsgeneral sentence embeddings @xcite and task-specific embeddings (after task-specific training @xcite ). then, we copy the label of the nearest training example to each evaluation example, constructing a simple nearest neighbor baseline model. in nearly every setting, we show that copying labels from the nearest train example alone achieve a competitive baseline, indicating overlap in content between the training and evaluation sets without any task specific training. we find that naturally-collected datasets exhibit stronger training and evaluation set overlap compared to more synthetic and adversarially-generated data. we introduce a new metric, named inssim, which summarizes the distance from each evaluation example to its nearest training examples, indicating the train-evaluation overlap. we use the nearest neighbor classifier and inssim score to estimate the difficulty of individual evaluation examples, and suggest splitting evaluation datasets into challenging and easier subsets. our analysis motivates careful benchmark designs @xcite ) that aims to capture both natural language usage and distributional shifts. ## related work representing a sequence of tokens as a single, fixed dimensional vector @xcite @xcite has been studied extensively. such an encoder can act as a dense passage retriever @xcite , paired with an efficient similarity search method @xcite . two prior studies in question answering @xcite look in-depth into the overlap between the training and evaluation sets. they identify the most similar training example either by answer string match or comparing the question embedding constructed for passage retrieval. the follow up work further develops the qa model @xcite for copying the answer from the nearest training example, after augmenting training examples with generated question answer pairs. our study in section 4.3 extends this setting for a wide range of tasks and different embedding methods. similar to our work, @xcite examine train-test overlap for text classification tasks. they also compute the similarity for each test instance to the entire training set using a similarity function. however, they utilize a bag-of-words approach to represent text (where we use sentence embeddings). in addition, we provide analysis for a broad range of datasets. many works have explored whether models simply memorize the training dataset or actually learn the task, thus generalizing to unseen examples. our nearest-neighbor match classification method resembles protobert @xcite , which shows promising performance in rare classes. the model classifies examples by comparing distance to the centroid of training examples belonging to each class. our method is simpler, without estimating a probability distribution over the output classes. @xcite also study the effect of dataset size and model size on memorization, but look at the dynamics of memorization in language models during training, finding that larger language models tend to memorize data faster, and that certain parts of speech are memorized faster than others. other work studies different subsets of datasets and how this can change evaluation. @xcite study dataset difficulty in terms of the lack of usable information to a particular model v , as well as difficulty of data subsets using a measure of pointwise v -information for individual data instances. as in our work, @xcite study difficulty of individual instances, although they focus on the training rather than evaluation set. similarly, @xcite propose a method for better evaluation of generalization on more difficult examples (those assigned lower likelihood by a pretrained lm), focused on creating the train-eval split. in our work, we introduce a very simple and generalizable method of splitting examples by whether classification with the nearest training example can succeed. recent work @xcite @xcite focuses on modeling distributional shifts in carefully constructed real world datasets, such as simulating shifts by having training set from one region and the test set from another region. this can be one path to mitigate frequent train-evaluation overlap in naturally occurring datasets. ## categorizing dataset collection method nlp datasets are collected through diverse methods for multiple purposes -some datasets mirror the user-facing applications closely (e.g., question answering datasets and machine translation datasets), while other datasets are carefully designed for diagnostic purposes. with the rise of harder to interpret, high capacity models @xcite , many datasets are designed to probe model qualities. would different data collection method yield different level of train evaluation overlap? to investigate this, we first categorize the data collection method of datasets below. we propose a discrete scale of naturalness, from purely synthetic to user-generated, as follows: we note that our definition of synthetic data includes highly-constrained crowd-sourced text, by which we mean that the annotators have limited freedom in the content of their annotations. for example, for the winogrande dataset workers are instructed to choose an anchor word to use in the twin sentences, they are given a range for sentence length, and they are asked to maintain 70% overlap between sentences. this is less natural than what the human might have generated on their own. we provide examples of the datasets of each type we study here, approximately ordered from the least to most natural datasets. winogrande a crowd-sourced, commonsense reasoning benchmark inspired by the winograd schema challenge, in which twin sentences with a small edit distance each have a missing word and two possible options @xcite . csqa 2.0 (commonsense question answering 2.0) a corpus of crowdsourced yes/no commonsense reasoning questions (e.g., "a playing card is capable of cutting soft cheese?") @xcite . anli (adversarial nli) a natural language inference corpus with data collected "adversarially" in three rounds using a human-in-the-loop approach @xcite . mnli (multi-genre natural language inference) a corpus of sentence pairs (crowdsourced) with annotations for textual entailment (given a premise and hypothesis, does the first entail, contradict, or is neutral to the other). we conduct experiments using both the matched (in-domain) and mismatched (cross-domain) evaluation sets @xcite . squad 2.0 (stanford question answering dataset 2.0) a corpus of crowdsourced questions (along with a wikipedia context), and annotated answer spans. unlike squad 1.1, not all questions have answers @xcite . mrpc (microsoft research paraphrase corpus) a corpus of sentence pairs extracted from online news sources, where each pair is annotated for whether the sentences are semantically equivalent @xcite . the sentences was paired based on heuristics (e.g., "two sentences share at least three common words"). nq (natural questions) a corpus of questions from popular google search queries, paired with a retrieved wikipedia document, annotated with an answer. we use simplified mrqa version, which removes unanswerable questions, yes/no questions or questions without a short answer span and considers paragraph containing a short answer span as context instead of the entire document @xcite . tweeteval a corpus of tweets containing multiple classification tasks @xcite ), though we used the subset of the dataset specifically for sentiment analysis. we also pre-process the data to remove examples with the neutral label, making the classification task binary (positive/negative) for out-domain evaluation with sst-2. sst-2 (stanford sentiment treebank) a corpus of movie review sentences with annotations for sentiment (positive/negative) @xcite . ag news a corpus of news articles from the web, categorized into four topics (business, sci/tech, sports, world) @xcite . imdb (imdb review dataset) a balanced corpus of movie reviews from imdb with negative (@xmath0. ## nearest neighbor analysis with two types of encoders we begin studying overlap with an analysis of nearest neighbor data instances between the train and evaluation datasets. we define the nearest neighbor for each evaluation example x e in the given training dataset x train . this is dependent on the embedding function e(x), and the training dataset x train . following prior work @xcite , we define the similarity between two examples x i and x j as the cosine similarity between their embeddings, e(x i ) and e(x j ). we describe how to encode each example below. ## quantifying overlap with instance similarity in this section, we introduce a new metric, instance similarity (inssim), and use it to identify easy and challenging instances in the evaluation dataset. defining inssim we define a metric, inssim(x e ), for each individual evaluation example x e based on its nearest neighbors in the provided training dataset. we notate topn(x e , x train , k) as set of k nearest examples in the total training dataset x train of x e according to the similarity function described in section 4. we conduct our analysis with a default setting of @xmath1. interpreting inssim the higher inssim(x e ), the easier for a machine learning model to estimate p (y e |x e ), if the label of the example matches its nearest train neighbors (we study this further in this section). an alternative metric would be estimating the input distribution p (x) based on the training data and evaluate the likelihood of x e according to this distribution. while p (x) will estimate how likely x e is with respect to the entire training set x train , inssim will only consider the k closest elements in the training dataset. given strong few-shot learning ability of recent pre-trained models @xcite , we anticipate this metric can more effectively capture the predicted performance on example x e . we report the average inssim score on each dataset in to better evaluate model generalization, we propose to divide evaluation examples into two subsets -(1) match: examples where the evaluation label equals the nearest gold train label, and (2) mismatch: examples where the evaluation label does not match the nearest gold train label. we use general sentence embeddings (simcse) for the representations for better generalizability. we hypothesize that the match subset is easier for models. how does model performance differ between match and mismatch subsets? we show roberta-large performance on each of these subsets, along with the difference between them, in ## conclusion in this paper, we analyze eleven downstream nlp datasets for train-evaluation overlap using a nearest neighbors approach, quantified with a simple measure of instance similarity. we categorize datasets according to their data collection method, and find that more naturally-collected data and easier tasks tend to demonstrate higher train-eval overlap than more synthetically-generated data and difficult tasks. lastly, we suggest using nearest neighbor analysis to split the evaluation data into more easy and challenging subsets, determined by the overlap with the training set, and advocate studying model performance on these subsets as well as the full dataset for a more comprehensive evaluation of model generalizability.