Dynabench: Rethinking Benchmarking in NLP (2021)

Abstract: We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.

## introduction while it used to take decades for machine learning models to surpass estimates of human performance on benchmark tasks, that milestone is now routinely reached within just a few years for newer datasets (see figure 1 ). as with the rest of ai, nlp has advanced rapidly thanks to improvements in computational power, as well as algorithmic breakthroughs, ranging from attention mechanisms @xcite , to transformers @xcite , to pre-trained language models @xcite @xcite @xcite . equally important has been the rise of benchmarks that support the development of ambitious new data-driven models and that encourage apples-to-apples model comparisons. benchmarks provide a north star goal for researchers, and are part of the reason we can confidently say we have made great strides in our field. in light of these developments, one might be forgiven for thinking that nlp has created models with human-like language capabilities. practitioners know that, despite our progress, we are actually far from this goal. models that achieve super-human performance on benchmark tasks (according to the narrow criteria used to define human performance) nonetheless fail on simple challenge examples and falter in real-world scenarios. a substantial part of the problem is that our benchmark tasks are not adequate proxies for the sophisticated and wide-ranging capabilities we are targeting: they contain inadvertent and unwanted statistical and social biases that make them artificially easy and misaligned with our true goals. we believe the time is ripe to radically rethink benchmarking. in this paper, which both takes a position and seeks to offer a partial solution, we introduce dynabench, an open-source, web-based research platform for dynamic data collection and model benchmarking. the guiding hypothesis be-hind dynabench is that we can make even faster progress if we evaluate models and collect data dynamically, with humans and models in the loop, rather than the traditional static way. concretely, dynabench hosts tasks for which we dynamically collect data against state-of-theart models in the loop, over multiple rounds. the stronger the models are and the fewer weaknesses they have, the lower their error rate will be when interacting with humans, giving us a concrete metrici.e., how well do ai systems perform when interacting with humans? this reveals the shortcomings of state-of-the-art models, and it yields valuable training and assessment data which the community can use to develop even stronger models. in this paper, we first document the background that led us to propose this platform. we then describe the platform in technical detail, report on findings for four initial tasks, and address possible objections. we finish with a discussion of future plans and next steps. ## background progress in nlp has traditionally been measured through a selection of task-level datasets that gradually became accepted benchmarks @xcite . recent well-known examples include the stanford sentiment treebank @xcite , squad @xcite , @xcite , and multinli @xcite . more recently, multi-task benchmarks such as senteval @xcite , decanlp @xcite , glue @xcite , and super-glue @xcite were proposed with the aim of measuring general progress across several tasks. when the glue dataset was introduced, "solving glue" was deemed "beyond the capability of current transfer learning methods" @xcite . however, glue saturated within a year and its successor, superglue, already has models rather than humans at the top of its leaderboard. these are remarkable achievements, but there is an extensive body of evidence indicating that these models do not in fact have the humanlevel natural language capabilities one might be lead to believe. ## dynabench dynabench is a platform that encompasses different tasks. data for each task is collected over multiple rounds, each starting from the current state of the art. in every round, we have one or more target models "in the loop." these models interact with humans, be they expert linguists or crowdworkers, who are in a position to identify models' shortcomings by providing examples for an optional context. examples that models get wrong, or struggle with, can be validated by other humans to ensure their correctness. the data collected through this process can be used to evaluate state-of-the-art models, and to train even stronger ones, hopefully creating a virtuous cycle that helps drive progress in the field. figure 2 provides a sense of what the example creation interface looks like. as a large-scale collaborative effort, the platform is meant to be a platform technology for humanand-model-in-the-loop evaluation that belongs to the entire community. in the current iteration, the platform is set up for dynamic adversarial data collection, where humans can attempt to find modelfooling examples. this design choice is due to the fact that the average case, as measured by maximum likelihood training on i.i.d. datasets, is much less interesting than the worst (i.e., adversarial) case, which is what we want our systems to be able to handle if they are put in critical systems where they interact with humans in real-world settings. however, dynabench is not limited to the adversarial setting, and one can imagine scenarios where humans are rewarded not for fooling a model or ensemble of models, but for finding examples that models, even if they are right, are very uncertain about, perhaps in an active learning setting. similarly, the paradigm is perfectly compatible with collaborative settings that utilize human feedback, or even negotiation. the crucial aspect of this proposal is the fact that models and humans interact live "in the loop" for evaluation and data collection. one of the aims of this platform is to put expert linguists center stage. creating model-fooling examples is not as easy as it used to be, and finding interesting examples is rapidly becoming a less trivial task. in anli, the verified model error rate for crowd workers in the later rounds went below 1-in-10 (nie et al., 2020), while in "beat the ai", human performance decreased while time per valid adversarial example went up with stronger models in the loop @xcite . for expert linguists, we expect the model error to be much higher, but if the platform actually lives up to its virtuous cycle promise, that error rate will go down quickly. thus, we predict that linguists with expertise in exploring the decision boundaries of machine learning models will become essential. while we are primarily motivated by evaluating progress, both anli and "beat the ai" show that models can overcome some of their existing blind spots through adversarial training. they also find that best model performance is still quite far from that of humans, suggesting that while the collected data appears to lie closer to the model decision boundaries, there still exist adversarial examples beyond the remit of current model capabilities. ## caveats and objections there are several obvious and valid objections one can raise. we do not have all the answers, but we can try to address some common concerns. won't this lead to unnatural distributions and distributional shift? yes, that is a real risk. first, we acknowledge that crowdsourced texts are likely to have unnatural qualities: the setting itself is artificial from the perspective of genuine communication, and crowdworkers are not representative of the general population. dynabench could exacerbate this, but it also has features that can help alleviate it. for instance, as we discussed earlier, the sentiment analysis project is using naturalistic prompt sentences to try to help workers create more diverse and naturalistic data. second, if we rely solely on dynamic adversarial collection, then we increase the risks of creating unnatural datasets. for instance, @xcite show that training solely on adversarially-collected data for qa was detrimental to performance on non-adversarially collected data. however, they also show that models are capable of simultaneously learning both distributions when trained on the combined data, retaining if not slightly improving performance on the original distribution (of course, this may not hold if we have many more examples of one particular kind). ideally, we would combine adversarially collected data with non-adversarial-preferably naturally collecteddata, so as to capture both the average and worst case scenarios in our evaluation. finally, we note that dynabench could enable the community to explore the kinds of distributional shift that are characteristic of natural languages. words and phrases change their meanings over time, between different domains, and even between different interlocutors. dynabench could be a tool for studying such shifts and finding models that can succeed on such phenomena. what if annotators "overfit" on models? a potential risk is cyclical "progress," where improved models forget things that were relevant in earlier rounds because annotators focus too much on a particular weakness. continual learning is an exciting research direction here: we should try to understand distributional shift better, as well as how to characterize how data shifts over time might impact learning, and how any adverse effects might be overcome. because of how most of us have been trained, it is natural to assume that the last round is automatically the best evaluation round, but that does not mean that it should be the only round: in fact, most likely, the best way to evaluate progress is to evaluate on all rounds as well as any high-quality static test set that exists, possibly with a recency-based discount factor. to make an analogy with software testing, similar to checklists @xcite , it would be a bad idea to throw away old tests just because you've written some new ones. as long as we factor in previous rounds, dynabench's dynamic nature offers a way out from forgetting and cyclical issues: any model biases will be fixed in the limit by annotators exploiting vulnerabilities. another risk is that the data distribution might be too heavily dependent on the target model in the loop. when this becomes an issue, it can be mitigated by using ensembles of many different architectures in the loop, for example the top current state-of-the-art ones, with multiple seeds. 5 how do we account for future, not-yet-in-theloop models? obviously, we can't-so this is a very valid criticism. however, we can assume that an ensemble of model architectures is a reasonable approximation, if and only if the models are not too bad at their task. this latter point is crucial: we take the stance that models by now, especially in aggregate, are probably good enough to be reasonably close enough to the decision boundaries-but it is definitely true that we have no guarantees that this is the case. how do we compare results if the benchmark keeps changing? this is probably the main hurdle from a community adoption standpoint. but if we consider, e.g., the multiple iterations of se-meval or wmt datasets over the years, we've already been handling this quite well-we accept that a model's bleu score on wmt16 is not comparable to wmt14. that is, it is perfectly natural for benchmark datasets to evolve as the community makes progress. the only thing dynabench does differently is that it anticipates dataset saturation and embraces the loop so that we can make faster and more sustained progress. what about generative tasks? for now dynabench focuses on classification or span extraction tasks where it is relatively straightforward to establish whether a model was wrong. if instead the evaluation metric is something like rouge or bleu and we are interested in generation, we need a way to discretize an answer to determine correctness, since we wouldn't have ground truth annotations; which makes determining whether a model was successfully fooled less straightforward. however, we could discretize generation by re-framing it as multiple choice with hard negatives, or simply by asking the annotator if the generation is good enough. in short, going beyond classification will require further research, but is definitely doable. do we need models in the loop for good data? the potential usefulness of adversarial examples can be explained at least in part by the fact that having an annotation partner (so far, a model) simply provides better incentives for generating quality annotation. having the model in the loop is obviously useful for evaluation, but it's less clear if the resultant data is necessarily also useful in general for training. so far, there is evidence that adversarially collected data provides performance gains irrespective of the model in the loop @xcite @xcite . for example, anli shows that replacing equal amounts of "normally collected" snli and mnli training data with anli data improves model performance, especially when training size is small @xcite , suggesting higher data efficiency. how-ever, it has also been found that model-in-the-loop counterfactually-augmented training data does not necessarily lead to better generalization @xcite . given the distributional shift induced by adversarial settings, it would probably be wisest to combine adversarially collected data with nonadversarial data during training (anli takes this approach), and to also test models in both scenarios. to get the most useful training and testing data, it seems the focus should be on collecting adversarial data with the best available model(s), preferably with a wide range of expertise, as that will likely be beneficial to future models also. that said, we expect this to be both task and model dependent. much more research is required, and we encourage the community to explore these topics. is it expensive? dynamic benchmarking is indeed expensive, but it is worth putting the numbers in context, as all data collection efforts are expensive when done at the scale of our current benchmark tasks. for instance, snli has 20k examples that were separately validated, and each one of these examples cost approximately @xmath00.64 each (p.c., multinli authors). by comparison, the average cost of creation and validation for anli examples is closer to $1.00 (p.c., anli authors). this is a substantial increase at scale. however, dynamic adversarial datasets may also last longer as benchmarks. if true, then the increased costs could turn out to be a bargain. we should acknowledge, though, that dynamic benchmarks will tend to be more expensive than regular benchmarks for comparable tasks, because not every annotation attempt will be model-fooling and validation is required. such expenses are likely to increase through successive rounds, as the models become more robust to workers' adversarial attacks. the research bet is that each example obtained this way is actually worth more to the community and thus worth the expense. in addition, we hope that language enthusiasts and other non-crowdworker model breakers will appreciate the honor that comes with being high up on the user leaderboard for breaking models. we are working on making the tool useful for education, as well as gamifying the interface to make it (even) more fun to try to fool models, as a "game with a purpose" @xcite , for example through the ability to earn badges. ## conclusion and outlook we introduced dynabench, a research platform for dynamic benchmarking. dynabench opens up exciting new research directions, such as investigating the effects of ensembles in the loop, distributional shift characterisation, exploring annotator efficiency, investigating the effects of annotator expertise, and improving model robustness to targeted adversarial attacks in an interactive setting. it also facilitates further study in dynamic data collection, and more general cross-task analyses of humanand-machine interaction. the current iteration of the platform is only just the beginning of a longer journey. in the immediate future, we aim to achieve the following goals: anyone can run a task. having created a tool that allows for human-in-the-loop model evaluation and data collection, we aim to make it possible for anyone to run their own task. to get started, only three things are needed: a target model, a (set of) context(s), and a pool of annotators.