What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021)

Abstract: Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.

## introduction a large and impactful thread of research on natural language understanding (nlu) has focused on improving results on benchmark datasets that feature roughly independent and identically distributed (iid) training, validation, and testing sections, drawn from data that were collected or annotated by crowdsourcing @xcite @xcite . recent methodological progress combined with longstanding issues in crowdsourced data quality has made it so state-of-the-art systems are nearing the maximum achievable values on most of these benchmarks and thus are unlikely to be able to measure further improvements @xcite . at the same time, these apparently high-performing systems have serious known issues and have not achieved human-level competence at their tasks @xcite . ## background the problem performance on popular benchmarks is extremely high, but experts can easily find issues with high-scoring models. the glue benchmark @xcite , a compilation of nlu evaluation tasks, has seen performance on its leaderboard approach or exceed human performance on all nine of its tasks. the follow-up superglue benchmark project @xcite solicited dataset submissions from the nlp research community in 2019, but wound up needing to exclude the large majority of the submitted tasks from the leaderboard because the bert model @xcite was already showing performance at or above that of a majority vote of human crowdworkers. of the eight tasks for which bert did poorly enough to leave clear headroom for further progress, all are now effectively saturated @xcite . state-of-the-art performance on the highly popular squad 2 english reading-comprehension leaderboard @xcite has long exceeded that of human annotators. ample evidence has emerged that the systems that have topped these leaderboards can fail dramatically on simple test cases that are meant to test the very skills that the leaderboards focus on @xcite . this result makes it clear that our systems have significant room to improve. however, we have no guarantee that our benchmarks will detect these needed improvements when they're made. most were collected by crowdsourcing with relatively limited quality control, such that we have no reason to expect that perfect performance on their metrics is achievable or that the benchmark will meaningfully distinguish between systems with superhuman metric performance. while the true upper bound on performance for any task (bayes error) is not measurable, the fact that our systems have exceeded serious estimates of human performance leaves us with no reason to expect there to be much more headroom. in addition, many of our best models display socially-relevant biases that render them inappropriate for deployment in many applications. 2 our best current benchmarks do little or nothing to dis-courage harmful biases and, by building largely on crowdsourced or naturally-occurring text data, they likely incentivize the development of models that reproduce problematic biases, at least to some degree. the goal this paper lays out four criteria that we would like our benchmarks to satisfy in order to facilitate further progress toward a primarily scientific goal: building machines that can demonstrate a comprehensive and reliable understanding of everyday natural language text in the context of some specific well-posed task, language variety, and topic domain. among language understanding tasks, we focus on those that use labeled data and that are designed to test relatively general language understanding skills, for which the design of benchmarks can be especially difficult. we distinguish between a task and a benchmark: a task, in our terms, is a language-related skill or competency that we want a model to demonstrate in the context of a specific input-output format. a benchmark attempts to evaluate performance on a task by grounding it to a text domain and instantiating it with a concrete dataset and evaluation metric. as a rough example, multiple-choice reading-comprehension question answering is a task, which the cosmos benchmark @xcite attempts to test using an accuracy metric over a specific sample of passages and questions from the english personal narrative domain. there is no general way to prove that a concrete benchmark faithfully measures performance on an abstract task. nevertheless, since we can only evaluate models on concrete benchmarks, we have no choice but to strengthen the correspondence between the two as best we can. we set aside the evaluation of computational efficiency and data efficiency, despite its relevance to many specific applications of language technology. we will not fully set aside issues of social bias. even though it is possible for the same system to demonstrate both adept language understanding and harmful social prejudices, foot_2 ethical concerns prompt us to argue that community-wide benchmarks should identify and disincentivize potentially harmful biases in models. the widespread sharing of trained models among nlu researchers and en-gineers and the fast pace of nlp r&d work mean that it is easy for systems designed with scientific goals in mind to be deployed in settings where their biases can cause real harm. while recent initiatives around data documentation should reduce the accidental deployment of models built on inappropriate data @xcite , we see room to do more. we will also set aside few-shot learning, in which tasks are made artificially difficult by training models only on small subsets of the available training data (as was prominently used for gpt-3 by @xcite . this paper focuses instead on the case where one is interested in reaching excellent performance on some language task and is willing to collect data or otherwise expend resources to make that possible. while few-shot learning represents a potentially impactful direction for engineering research, and success on some task in a few-shot setting is clear evidence of success more generally, artificial constraints on the use of training data do not fit the broad goals laid out above and do not fit many applied settings. ## four challenges this paper focuses on four criteria, outlined in figure 1 , that we argue effective future benchmarks for nlu tasks should satisfy. we believe that no current benchmark for any difficult broad-domain nlu task satisfies all four: ## sketching a solution building new benchmarks that improve upon our four axes is likely to be quite difficult. below we attempt to sketch out some possible directions for improvement along each axis. ## related work the nlp and ml research communities are increasingly interested in issues surrounding data and evaluation. this section surveys relevant positions and issues that don't quite fit our schema. @xcite advocate for the more precise reporting of the focus and abilities of test sets and metrics in ml broadly, with a focus on issues surrounding statistical power. @xcite and @xcite advocate for explicit freestanding datasheets documenting dataset releases of all kinds, with a focus on making potential harmful mismatches between data and application visible, and @xcite argue along similar lines for a broader program of transparency and stakeholder engagement in data creation. @xcite lay out a set of best practices for results reporting, with a focus on the impact of hyperparameter tuning on model comparison. ethayarajh and jurafsky (2020) advocate for the inclusion of efficiency considerations in leaderboard design. @xcite describe ways that trivia competitions can provide a model for carefully-considered dataset design. church and hestness (2019) revisit the arguments that motivated the nlp community's shift toward quantitative benchmarking in the early 1990s and warn that the overwhelming success of this shift has indirectly laid the groundwork for the widespread use of poor-quality benchmarks. blodgett et al. (2020) challenge researchers working on social bias in nlp to focus more precisely on specific types of harm to specific populations of users, a challenge that our broad position piece does not fully meet. nlp has had longstanding debates over the types of tasks that best test substantial language understanding skills. many task-specific papers contribute to this debate, as does a prominent recent thread advocating for an increased focus on grounding of various kinds by @xcite , @xcite , and others. ## conclusion benchmarking for nlu is broken. we lay out four major criteria that benchmarks should fulfill to offer faithful, useful, and responsible measures of language ability. we argue that departing from iid evaluation (as is seen with benchmark datasets collected by adversarial filtering) does not help to address these criteria, but lay out in broad strokes how each criterion might be addressed directly. nonetheless, important open research questions remain. most centrally, it is still unclear how best to integrate expert effort into crowdsourced data collection, and we do not yet see a clear institutional model by which to ensure that bias metrics are built and used when they are most needed.