WANLI : Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022)

Abstract: A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. We introduce a novel approach for dataset creation based on worker and AI collaboration, which brings together the generative strength of language models and the evaluative strength of humans. Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns. Machine generated examples are then automatically filtered, and finally revised and labeled by human crowdworkers. The resulting dataset, WANLI, consists of 107,885 NLI examples and presents unique empirical strengths over existing NLI datasets. Remarkably, training a model on WANLI improves performance on eight out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI, compared to training on the 4x larger MultiNLI. Moreover, it continues to be more effective than MultiNLI augmented with other NLI datasets. Our results demonstrate the promise of leveraging natural language generation techniques and re-imagining the role of humans in the dataset creation process.

## introduction as much as large-scale crowdsourced datasets have expedited progress on various nlp problems, a growing body of research has revealed fundamental limitations in existing datasets: they are often flooded with repetitive and spurious patterns, rather than covering the broad range of linguistic phenomena required by the task @xcite . this leads to models that seem to achieve humanlevel performance on in-domain test sets, yet are brittle when given out-of-domain or adversarial examples @xcite . starting with a data map @xcite of an existing dataset relative to a trained model, @xcite we automatically identify pockets of data instances exemplifying challenging reasoning patterns. next, (2) we use gpt-3 to generate new instances with the same pattern. these generated examples are then (3) automatically filtered via a metric we introduce inspired by data maps, and (4) given to human annotators to assign a gold label and optionally revise. we attribute this problem to an inherent challenge in the crowdsourcing design-the prevalent paradigm for creating large-scale nlp datasetswhere a relatively small number of workers create a massive number of free text examples. while human annotators are generally reliable for writing correct examples, crafting diverse and creative examples at scale can be challenging. thus, crowdworkers often resort to a limited set of writing strategies for speed, at the expense of diversity @xcite . when models overfit to such repetitive patterns, they fail to generalize to out-of-domain examples where these patterns no longer hold @xcite . on the other hand, there has been remarkable progress in open-ended text generation based on massive language models @xcite . despite known deficiencies such as incoherence or repetition @xcite , these models often produce human-like text @xcite and show potential for creative writing tasks @xcite . importantly, these models are capable of replicating a pattern given just a few examples in context @xcite . in this paper, we introduce a novel approach for dataset creation which brings together the generative strength of language models and the evaluative strength of humans through human and machine collaboration ( §2). the key insight of our approach is that language models can create new examples by replicating linguistic patterns that are valuable for training, without necessarily "understanding" the task itself. illustrated in figure 1 , our pipeline starts with an existing dataset. we use dataset cartography from @xcite to automatically identify pockets of examples that demonstrate challenging reasoning patterns relative to a trained model. using each group as a set of in-context examples, we leverage a pretrained language model to generate new examples likely to have the same pattern (see we demonstrate the effectiveness of our approach on the task of natural language inference (nli), which determines whether a premise entails (i.e., implies the truth of) a hypothesis, both expressed in natural language. despite being one of the most resource-available tasks in nlp, analysis and challenge sets repeatedly demonstrate the limitations of existing datasets and the brittleness of nli models trained on them @xcite @xcite . using multinli @xcite as our original dataset, we use our pipeline to create a dataset of 107,885 examples, which we call worker-and-ai nli (wanli). 1 remarkably, empirical results demonstrate that replacing multinli supervision with wanli (which is 4 times smaller) improves performance on eight different out-of-domain test sets, including datasets that are converted to the nli format from downstream tasks such as question-answering and fact verification ( §3). this result holds even when augmenting multinli with other nli datasets and recently proposed augmentation sets. moreover, including wanli in the training data can help improve performance on certain in-domain test sets. we then analyze wanli and show that it has fewer previously documented spurious correlations than multinli ( §4), and provide insights into the collaborative framework ( §5). our approach contrasts with previous instructionbased generation of dataset examples @xcite , which require the model to understand the task from context, fundamentally limiting the complexity of generated output to what is accessible by the model. moreover, our human-in-the-loop approach is collaborative, rather than adversarial @xcite @xcite . overall, we leverage the best of both worlds: a powerful model's ability to efficiently generate diverse examples, and humans' ability to improve and ensure the quality of generations. our worker-ai collaborative approach is more scalable compared to the traditional crowdsourcing framework. our approach is generalizable, allowing for rejuvenating datasets on many different classification tasks, especially when performance seems to stagnate due to overfitting to popular benchmarks @xcite . our work shows the promise of leveraging language models in a controlled way to aid the dataset creation process, and we encourage the community to think of dataset curation as an ai challenge itself. ## worker-ai collaborative dataset creation for nli we describe our four-stage approach for dataset creation based on worker and ai collaboration. in this work, we apply it to the task of natural language inference (nli), which involves predicting whether a premise entails, contradicts or is neutral to a hypothesis. nli has broad applicability in nlp: it has proven useful for pretraining @xcite , and can be applied to verify candidate answers in question-answering p: the three schools have a number of students who are from families with no history of financial difficulties. h: families with no history of financial difficulties send their children to the three schools. ## training nli models with wanli we finetune different copies of roberta-large @xcite on different training sets, and evaluate each resulting model's performance on a large suite of nli challenge sets. given that the challenge sets were constructed independently of multinli or wanli, we consider them out-ofdistribution (ood) for both training datasets. ## artifacts in wanli we next investigate whether wanli contains similar artifacts to multinli. 6 we find that while wanli contains fewer previously known spurious correlations, it has a distinct set of lexical correlations that may reflect artifacts in gpt-3 output. ## related work crowdsourcing the scalability and flexibility of crowdsourcing has enabled the creation of foundational nlp benchmarks across a wide range of subproblems, and made it the dominant paradigm for data collection @xcite @xcite . nonetheless, a growing body of research shows that resulting datasets may not isolate the key linguistic phenomena @xcite @xcite . for crowdsourcing nli datasets, where the annotator is given a premise and asked to write a hypothesis of each label @xcite , the presence of annotation artifacts is especially well-studied @xcite @xcite . recent work attempted to remedy this through different data collection protocols but found negative results @xcite , showing this is a hard problem requiring greater innovation. ## conclusion at the heart of dataset creation is distilling human linguistic competence into data that models can learn from. the traditional crowdsourcing paradigm takes the view that the best approach for this is to solicit people to write free-form examples expressing their capabilities. in this work, we present a worker-and-ai collaborative approach and apply it to create wanli, whose empirical utility suggests that a better way of eliciting human intelligence at scale is to ask workers to revise and evaluate content. to this end, we hope to encourage more work in developing generative algorithms to aid the dataset creation process, and therefore re-imagining the role of human annotation.