Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets (2021)

Abstract: Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question formulations with answers seen during training, to generalizing to completely novel questions with novel answers. However, single aggregated test set scores do not show the full picture of what capabilities models truly have. In this work, we perform a detailed study of the test sets of three popular open-domain benchmark datasets with respect to these competencies. We find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding train sets. In addition, we find that 60-70% of answers in the test sets are also present in the train sets. Using these findings, we evaluate a variety of popular open-domain models to obtain greater insight into what extent they can generalize, and what drives their overall performance. We find that all models perform substantially worse on questions that cannot be memorized from train sets, with a mean absolute performance difference of 61% between repeated and non-repeated data. Finally we show that simple nearest-neighbor models outperform a BART closed-book QA model, further highlighting the role that train set memorization plays in these benchmarks

## introduction open-domain question answering (odqa) is a task that examines the ability of models to produce answers to natural language factoid questions drawn from an open set of domains. odqa has received significant attention for its potential practical applications, and more recently as a popular method to analyse how well nlp systems can capture and recall factual knowledge. this interest in odqa as a challenging "knowledge-intensive" task has led to a flurry of recent works that have driven test-set performance on standard odqa datasets to new heights @xcite @xcite @xcite . whilst there have been several works examining other kinds of qa datasets @xcite @xcite , however, we know comparatively little about how the questions and answers are distributed in odqa benchmarks, making it hard to understand and contextualize the results we are observing. in this work, we address these issues via an analysis of the test sets of three popular odqa datasets, namely webquestions @xcite , triv-iaqa @xcite and open naturalquestions @xcite . we identify three types of desired behaviour of a trained odqa system, in increasing order of difficulty: 1) to recall the answer to a question that the model has seen at training time. 2) to answer novel questions at test time and choose an answer from the set of answers it has seen during training. 3) to answer novel questions which have answers which are not contained in the training data. it is not clear to what extent our current odqa datasets measure each of these three behaviours. to address this, we stratify the test sets of these datasets. firstly, we split the test data by whether answers in the test set also appear somewhere in the train set. we find that 58-71% of test answers also occur somewhere in the training data, demonstrating that the majority of the test data does not probe for answer generalization. secondly, we annotate 1,000 question-answer pairs from each test set for near-duplicate questions in their respective train sets. we find that a surprisingly high 28-34% have paraphrased questions in the training data, the vast majority of which are near-duplicates differing by one or two words. this result implies that 30% of the test set of these datasets only probe for how well models can memorize question-answer pairs seen at training. equipped with these observations, we compute the performance of several recently proposed odqa models on our test subsets. we test both open-book approaches, which leverage retrieval from a large corpus of documents and closed-book approaches, which focus on training large parametric seq2seq models with no external knowledge source @xcite . we find that test data with train-overlapping data contribute the bulk of the overall performance of all the models studied. these issues seem to be more acute for closedbook models. strikingly, we find that a closedbook bart-based model @xcite is incapable of producing answers not observed at training time, and achieves very low scores on nonoverlapping questions, suggesting this model is only capable of memorizing question-answer pairs from training time. with this in mind, we build simple nearest-neighbor models which outperform this bart model, despite having virtually no capacity to generalize beyond training data. in summary, we make the following contributions: 1) we provide insights into how answer entities are distributed between dataset splits for odqa datasets 2) we provide annotated subsets of odqa test sets indicating whether test-time questions are duplicates of training time questions. foot_0 3) we evaluate a variety of models on our dataset splits, and derive insights into what kinds of question answering behaviour different models achieve. ## datasets we analyse three widely used odqa datasets, we-bquestions @xcite , triviaqa @xcite , and open-naturalquestions @xcite . all three datasets consist of factual natural language questions and short multi-token answers, differing slightly in the style of questions and format of answers. webquestions consists of 3,778 train and 2,032 test instances. questions were obtained by mining a search engine, and answers are freebase entities @xcite annotated by crowdworkers. the odqa task consists of predicting the name of the freebase entity. we use the standard train/test splits from @xcite and the development split from @xcite , which was randomly split from the train set. triviaqa consists of 78,785 train, 8,837 development and 11,313 test instances obtained by scraping trivia websites. answers are wikipedia entities, and any alias for the answer entity is considered a correct answer. we use the odqa splits, which correspond to the unfiltered-train and unfiltereddev reading comprehension splits @xcite @xcite . open-naturalquestions consists of search engine questions with crowdsourced answer spans in wikipedia articles. the odqa version consists of question-answer pairs from naturalquestions which have short answer spans less than 6 tokens in length. we use the standard open-domain splits in our experiments, consisting of 79,168 train, 8,757 development and 3,610 question answer pairs. for all three datasets, the canonical train, development and test splits were obtained by randomly splitting the question-answer pairs, and there are no exact duplicate questions in any dataset. we exclude development data from our overlap analyses, and focus purely on train-test overlap to explicitly assess the effects of training memorization. ## test-train overlaps we explore two ways of examining the test sets based on overlaps between training and test data. consider a question-answer pair (q, a) from the test set d test where the answer consists of at least one answer reference @xmath0..s n }. we can consider answer overlap where there exists at least one (q , a ) ∈ d train which shares at least one answer reference with (q, a). we can also consider question overlap, where there exists some (q , a ) ∈ d train where q is a duplicate of q, such that q and q are paraphrases and have the same answer. answer overlap following @xcite , we apply answer normalization (lowercasing, stripping punctuation, removing articles and normalizing whitespace) on answer references question-overlap unlike answer overlap, question overlap cannot be easily computed automatically, as searching for duplicates via rules or paraphrase classifiers may lead to both false positives and negatives. thus, we turn to manual annotation to investigate question overlap. to obtain a representative sample for each dataset, we annotate a random subset of 1,000 (q, a) pairs for each test set. annotators are shown a list of up to 50 training questions which have a similar answer reference. 2 this answer similarity function is designed for high recall to obtain a tight lower bound on question overlap. if there were no questions with similar answers in the train set, the question was automatically annotated as not overlapping. three expert annotators annotated the remaining questions and indicated if any were paraphrases of the test question and had the same answer. the results from the annotation can be seen in ## implications for modelling earlier we identified three classes of answering behaviours: 1) questions that can be memorized at training time, 2) novel questions that can be answered with answers memorized at training time, 3) novel questions with novel answers. we refer to these behaviours as question memorization, answer classification and qa generalization. question memorization to perform well on the question-overlap subset, a model only needs to memorize (q, a) pairs at training time, then recognize which training question matches a test-time question. the reasoning required ranges from trivial duplicate detection for very similar questions such as "who played pink in pink floyd the wall" and "who played pink in the movie the wall", to more challenging inference problems for subtler duplicates like "on which island in the north sea did both st aidan and st cuthbert live?" and "irish born missionary st aidan founded a monastery in 653 on which english island which is also the name of a 1970s uk folk-rock band?". manual annotation of 100 question-overlap pairs showed 81% were simple duplicates differing by one or two words, 14% needed some paraphrase recognition ability, and 5% needed more sophisticated language understanding. to measure performance on question memorization, we build a test subset of (q, a) pairs with question overlap to the train set. answer classification for a model to handle answer-overlap questions, a classifier over train set answers would be sufficient, as answers never appear at test time that do not appear at training time. we build a test subset of (q, a) pairs which have answer overlap, but not question overlap. questionoverlap pairs are excluded as they are significantly easier and would inflate scores. ## related work examining what behaviours are learnt by models has received attention in language understanding tasks, such as glue @xcite , which includes tools for probing for different reasoning types. there has also been critical and careful analysis of qa systems and datasets. @xcite , @xcite and @xcite analyse the difficulty of various machine reading datasets, and @xcite show that visual qa models memorize common question-answer relationships in training data. févry et al. (2020) analyse various closed-book models' triviaqa predictions. @xcite note that the machine reading naturalquestions dataset has train-test overlap of wikipedia titles, and provide baselines for "long-answer" qa. @xcite observe answer overlap effects in a related modality (knowledgebase qa), but no not consider question overlap. ## conclusion we performed an analysis of popular open-domain qa datasets. we found that 60% of test set answers overlap with the train set and 30% of test set questions have at least one duplicate in the train set. given these observations, we contextualize performance of seven odqa models, stratifying by the extent of overlap, exploring how well these models generalize verses simply memorizing training data. it is clear that performance on these datasets cannot be properly understood by overall qa accuracy. in the future, a greater emphasis should be placed on more behaviour-driven evaluation, rather than pursuing single-number overall accuracy.