Do Question Answering Modeling Improvements Hold Across Benchmarks? (2023)

Abstract: Do question answering (QA) modeling improvements (e.g., choice of architecture and training procedure) hold consistently across the diverse landscape of QA benchmarks? To study this question, we introduce the notion of concurrence—two benchmarks have high concurrence on a set of modeling approaches if they rank the modeling approaches similarly. We measure the concurrence between 32 QA benchmarks on a set of 20 diverse modeling approaches and find that human-constructed benchmarks have high concurrence amongst themselves, even if their passage and question distributions are very different. Surprisingly, even downsampled human-constructed benchmarks (i.e., collecting less data) and programmatically-generated benchmarks (e.g., cloze-formatted examples) have high concurrence with human-constructed benchmarks. These results indicate that, despite years of intense community focus on a small number of benchmarks, the modeling improvements studied hold broadly.

## introduction the nlp community has created a diverse landscape of extractive question answering (qa) benchmarks-their context passages may come from different sources, their questions may focus on different phenomena or be written by different populations, or other aspects of the data collection process may differ. driven to improve benchmark performance, researchers have proposed a variety of qa modeling approaches. however, not all benchmarks receive equal attention from the community @xcite ; many qa modeling approaches are developed on a small handful of benchmarks, especially those with popular leaderboards (e.g., squad; @xcite . as a result, it is conceivable that some modeling improvements may not hold because they are (perhaps inadvertently) benchmark-specific, while others figure 1 : two benchmarks have high concurrence if they rank a set of modeling approaches similarly. surprisingly, we find that human-constructed benchmarks (e.g., squad, naturalquestions) have high concurrence with other human-constructed benchmarks, downsampled human-constructed benchmarks, and even programmatically-generated cloze benchmarks (e.g., the children's book test; cbt). in addition, we are able to construct synthetic benchmarks that have high concurrence with human-constructed benchmarks despite lacking natural language passages or questions. in this work, we evaluate whether improvements from modeling approaches hold (e.g., choices in model architecture or training procedure)-if a particular modeling approach improves performance when trained and evaluated on one benchmark, does it also improve performance on others? although much existing work studies whether systems generalize (i.e., a model with a particular set of parameters; @xcite @xcite , research value often comes not from the systems themselves (e.g., model weights), but from the underlying ideas, techniques, and approaches. we study the comparatively under-investigated question of whether such modeling approaches generalize. to study whether modeling improvements hold across benchmarks, we introduce the notion of concurrence. we say that two benchmarks have high concurrence on a set of modeling approaches if they rank the modeling approaches similarly. to assess whether modeling improvements hold across the space of qa benchmarks, we measure the concurrence between 32 diverse qa benchmarks on a testbed of 20 representative modeling approaches introduced between 2016 and 2020. overall, we find that benchmarks that differ substantially still often have high concurrence. human-constructed benchmarks (e.g., squad and mrqa naturalquestions) have high concurrence with each other, despite differences in crowdsourcing setups, passage and question distributions, and even linguistic phenomena of focus ( §3). how different can a benchmark be, while still maintaining high concurrence with humanconstructed benchmarks? in §4.1, we investigate the role of training dataset size by measuring concurrence with downsampled training datasets (e.g., using 20k squad training examples rather than the full 88k). we find that downsampled training datasets are sufficient for high concurrence with other human-constructed benchmarks. in §4.2, we measure concurrence between human-constructed and programmatically-generated benchmarks (e.g., cloze-formatted or synthetic) to better understand the importance of human-written questions and passages. we find that cloze-formatted benchmarks have high concurrence with human-constructed benchmarks, so human-written questions and passages are not strictly necessary for concurrence. however, programmatically-generated synthetic benchmarks (e.g., the babi task suite) have low concurrence. having found this breaking point of low concurrence, we construct two minimal synthetic benchmarks that achieve high concurrence with human-constructed benchmarks, despite lacking linguistic structure. intuitively, the benchmarks that concur with human-constructed benchmarks are those that require model capabilities that are also useful for better performance on human-constructed benchmarks (e.g., identifying paraphrase and lexical overlap; §4.3-4.5). our results have several implications for the future development of benchmarks and modeling approaches. to summarize: ## measuring concurrence informally, we say that two benchmarks have high concurrence on a set of modeling approaches if the two benchmarks rank the modeling approaches similarly. we compare the performance of a modeling approach when trained and tested on one benchmark with its performance when trained and tested on another benchmark-we use each benchmark's original i.i.d. train-test split, so all evaluation is indomain. repeating this process for many modeling approaches, we can assess whether performance gains between modeling approaches are generally preserved when moving between benchmarks. formally, define a benchmark b as a pair of datasets (d train , d test ), where d train ⊆ x × y and d test ⊆ x × y for an input space x and an output space y. a system is a function s : @xmath0.e., a trained model with a particular set of parameters). in contrast, a modeling approach (i.e., a neural architecture coupled with a training procedure) is a function a that takes in a training dataset d train and outputs a system. let eval denote an evaluation function, where eval(a, b) returns the performance (under a given evaluation function, e.g., exact match) of a modeling approach a when trained on the train split of b and tested on the test split of b. finally, concur(b 1 , b 2 ; a, eval) is the concurrence between the benchmarks b 1 and b 2 with respect to a set of modeling approaches a and the evaluation function eval. let a ∼ uniform(a), where uniform(a) denotes the uniform distribution over the set of modeling approaches a. defining the random variables p 1 = eval(a, b 1 ) and p 2 = eval(a, b 2 ), we finally define we use the squad exact match (em) metric as our evaluation function eval, and we consider the pearson correlation coefficient (r) and the kendall rank correlation coefficient (τ ) as our correlation functions corr. the former measures whether the relationship between model performance on the two benchmarks is approximately linear, whereas the latter measures whether pairwise rank comparisons between models are preserved between benchmarks. as a rough guideline, we consider τ > 0.8 to be high concurrence, though interpreting concurrence often requires more than comparing overall correlation. extractive qa modeling approaches. to assess concurrence in this work, we use a representative set of 20 diverse modeling approaches introduced between 2016 to 2020 (a). these modeling approaches include rasor @xcite , bidaf @xcite , documentreader @xcite , qanet @xcite , bidaf++ @xcite , mnemonicreader @xcite , fusionnet @xcite , bert @xcite , albert @xcite , roberta @xcite , electra @xcite , and spanbert @xcite . 1 10 of our 20 modeling approaches are nonpretrained. these approaches generally propose (1) better sequence encoders for passages and questions (e.g., @xcite @xcite and/or (2) improved attention mechanisms for question-passage interactions (e.g., @xcite @xcite . in contrast, the other 10 of our 20 modeling approaches are pre-trained; these modeling approaches all use the transformer architecture @xcite , but improve performance by proposing better pre-training procedures and objectives. these pre-trained modeling approaches are generally evaluated on a suite of downstream tasks, in contrast to non-pretrained modeling approaches, which generally evaluate on a single benchmark. all of these modeling approaches were originally evaluated on squad, though several (e.g., spanbert) were also evaluated on other qa benchmarks. we evaluate each modeling approach on each benchmark with the same training hyperparameters used for squad, as well as 5 additional randomly sampled hyperparameter settings. extractive qa benchmarks. in this work, we study concurrence between three broad classes of extractive qa benchmarks: (i) human-constructed, (ii) cloze, and (iii) synthetic. human-constructed benchmarks contain human-written natural language questions and passages; examples include squad, newsqa @xcite , and naturalquestions @xcite . on the other hand, cloze benchmarks (e.g., children's book test or cnn; @xcite contain cloze questions, which are "fill-inthe-blank" statements with masked answers. these questions are usually automatically-generated from human-written natural language passages. finally, synthetic benchmarks contain programmaticallygenerated questions and passages (e.g., the babi task suite; @xcite . ## do modeling improvements hold across human-constructed benchmarks? many extractive question answering benchmarks are human-constructed-they contain humanwritten natural language questions and passages. however, differences in the data collection procedure may yield benchmarks with dramatically different passage and question distributions. do modeling improvements hold across benchmarks despite these differences? setup. we study the concurrence between six human-constructed benchmarks: squad, newsqa, naturalquestions, drop @xcite , hotpotqa @xcite , and qamr @xcite . we use the mrqa versions of newsqa, naturalquestions, drop, and hot-potqa @xcite . ## exploring the limits of concurrence our results in §3 indicate that human-constructed benchmarks have high concurrence with each other, despite differences in their phenomena of interest and passage and question distributions. just how different can a benchmark be, while maintaining high concurrence with human-constructed benchmarks? in §4.1 we investigate the role of training dataset size on concurrence-while larger training datasets often yield better systems with higher end-task accuracy, are they necessary for comparing modeling approaches? in §4.2, we measure concurrence between human-constructed and cloze benchmarks to better understand the role of human-written questions and passages in concurrence. cloze benchmarks have high concurrence with human-constructed benchmarks, indicating that human-written questions and passages are not necessary for concurrence with human-constructed benchmarks. to take this to an extreme, §4.3 evaluates concurrence between programmaticallygenerated synthetic benchmarks (the babi task suite) with human-constructed benchmarks. our results show that the babi tasks have low concurrence with human-constructed benchmarks. having found this breaking point, we work backwards to build a minimal benchmark with high concurrence, which will enable us to better understand sufficient conditions for concurrence. in §4.4, we construct a benchmark that has no linguistic structure or complex reasoning but still has high concurrence with human-constructed benchmarks over non-pretrained models. finally, §4.5 shows that a synthetic benchmark that requires richer reasoning between question and passage tokens can achieve high concurrence with human-constructed benchmarks on both pre-trained and non-pretrained modeling approaches. ## related work a recent line of work examines whether systems have overfit to particular test sets by taking existing systems and evaluating them on newly-constructed test sets @xcite @xcite . recent work has also studied whether higher-performing systems are more robust by studying the correlation between in-domain and out-of-domain improvements @xcite . in contrast, this work examines whether improvements from modeling approaches hold across benchmarks. we train and test modeling approaches on a variety of existing and newlyconstructed benchmarks. in this regard, our work is similar to the study of @xcite , who find that performance improvements on im-agenet are well-correlated with performance improvements on other benchmarks. ## conclusion this work studies whether qa modeling improvements hold across the diverse landscape of qa benchmarks. we develop the notion of concurrence, which quantifies the similarity between benchmarks' rankings of modeling approaches. experiments with 32 qa benchmarks and 20 diverse modeling approaches indicate that humanconstructed benchmarks largely have high concurrence amongst themselves, even when their passage and question distributions or linguistic phenomena of focus are very different. to better understand how different benchmark attributes affect concurrence, we explore downsampled benchmarks and various programmatically-generated benchmarks, the latter having high concurrence only when they target phenomena that are also useful for better performance on human-constructed benchmarks (e.g., identifying paraphrase and lexical overlap). our results indicate that the modeling improvements studied hold broadly, despite years of intense community focus on a small number of benchmarks.