A sking C rowdworkers to W rite E ntailment E xamples: T he B est of B ad Options (2020)

Abstract: Large-scale natural language inference (NLI) datasets such as SNLI or MNLI have been created by asking crowdworkers to read a premise and write three new hypotheses, one for each possible semantic relationships (entailment, contradiction, and neutral). While this protocol has been used to create useful benchmark data, it remains unclear whether the writing-based annotation protocol is optimal for any purpose, since it has not been evaluated directly. Furthermore, there is ample evidence that crowdworker writing can introduce artifacts in the data. We investigate two alternative protocols which automatically create candidate (premise, hypothesis) pairs for annotators to label. Using these protocols and a writing-based baseline, we collect several new English NLI datasets of over 3k examples each, each using a fixed amount of annotator time, but a varying number of examples to fit that time budget. Our experiments on NLI and transfer learning show negative results: None of the alternative protocols outperforms the baseline in evaluations of generalization within NLI or on transfer to outside target tasks. We conclude that crowdworker writing still the best known option for entailment data, highlighting the need for further data collection work to focus on improving writing-based annotation processes.

## introduction research on natural language understanding has benefited greatly from the availability of largescale, annotated data, especially for tasks like reading comprehension and natural language inference, which lend themselves to non-expert crowdsourcing. these datasets are useful in three settings: evaluation @xcite @xcite ; pretraining @xcite @xcite ; and as training data for downstream tasks @xcite . natural language inference (nli), also known as recognizing textual entailment (rte; @xcite is the problem of determining whether or not a hypothesis semantically entails a premise. the two largest nli corpora, snli @xcite and mnli @xcite are created by asking crowdworkers to write three labeled hypothesis sentences given a premise sentence taken from a preexisting text corpus. while these datasets have been widely used as benchmarks for nlu, there have been no studies evaluating writing-based annotation for collecting nli data. moreover, there is growing evidence that human writing can introduce annotation artifacts, which enable models to perform moderately well just by learning spurious statistical patterns in the data @xcite @xcite . this paper explores the possibility of collecting high-quality nli data without asking crowdworkers to write hypotheses. we introduce two alternative protocols (figure 1 ) which substitute crowdworker writing with fully-automated pipelines to generate premise-hypothesis sentence pairs, which annotators then simply label. the first protocol uses a sentence-similarity-based method to pair similar sentences from large unannotated corpora. the second protocol uses parallel sentences and uses machine translation systems to generate sentence pairs. using the mnli protocol as our baseline, we collect five datasets using premises taken from gigaword news text @xcite and wikipedia. we then compare models trained using these datasets for their generalization performance within nli and for transfer learning to other tasks. we start from the assumption that writing a new hypothesis takes more time and effort than simply labeling a presented hypothesis. as a result, it is plausible that our protocols could offer some value even if the quality of the data they produce is no better than a writing-based baseline. to study the cost trade-off, we collect each dataset under the same fixed annotation budget with a fixed (∼ us @xmath01,791 for base-news and $1,445 for base-wiki), allowing us to collect around twice as many examples for each protocol. 6 for each dataset, we randomly select 250 sentence pairs as the test set and use the rest as the training set. to ensure accurate labeling, we perform an additional round of annotation on the test sets. we ask four crowdworkers to label each pair using the same instructions that we use for data labeling, giving us a total of 5 annotations per example. we assign the majority vote as the gold label. ## experiments we aim to test whether our alternative protocols can produce high-quality data that yield models that generalize well within nli and in transfer learning. for the nli evaluation, we evaluate each model on nine test sets: (i) the five new individual test sets, each containing ∼250 examples; (ii) the mnli development set; and (iii) the three development sets of adversarial nli (anli; @xcite , collected from three rounds of annotation (a1, a2, a3). anli is collected using an iterative adversarial approach that follows mnli but encourages we run experiments using the jiant toolkit @xcite , which is the recommended baseline package for superglue, and is based on pytorch @xcite , huggingface transformers @xcite . cbow baselines obtain near chance performance. using roberta, the top performing models are all trained on datasets collected using base: base-news and mnli-3k. we find that models trained using translate-wiki obtain the worst performance. ## related work there is a large body of work on constructing data for natural language inference. the first test suite for entailment problems, fracas @xcite , is a very small set created manually by experts to isolate phenomena of interest. the rte challenge corpora @xcite , et seq) were built by asking human annotators to judge whether a text entails a hypothesis. the sick dataset @xcite is constructed by mining existing paraphrase sentence pairs from image and video captions, which annotators then label. some recent works also use automatic methods for generating sentence pairs for entailment data. @xcite propose a framework to generate hypotheses based on context from general world knowledge or neural sequence-to-sequence methods. the dnc corpus @xcite ) is an nli dataset with ordinal judgments constructed by recasting several nlp datasets to nli examples and labeling them using custom automatic procedures. qa-nli @xcite is an nli dataset derived from existing qa datasets. similar to ours, both dnc and qa-nli use automatic methods to generate sentence pairs. however, neither of them explicitly evaluates whether machinegenerated pairs are better than human-generated pairs. bowman et al. (2020) propose four potential modifications to the snli/mnli protocol, all still involving crowdworker writing, and show that none yields improvements in the resulting data. swag @xcite and hellaswag @xcite construct sentence pairs from specific data sources and use language models to generate challenging negative examples. ## conclusion in this paper, we introduce two data collection protocols which use fully-automatic pipelines to collect hypotheses, replacing crowdworker writing in the mnli baseline protocol. we find that switching to a writing-free process with the same source data and annotator pool yields poor-quality data. our main experiments show strong negative results both in nli generalization and transfer learning, and mixed results on annotation artifacts, suggesting that mnli-style crowdworker writing examples are broadly better than automatically paired ones. this finding dovetails with that of @xcite , who find that they are unable to improve upon a base mnli-style prompt when introducing aids meant to improve annotator speed or creativity. future work along this line might focus on crowdsourcing strategies (beyond the basic hit design) which encourage crowdworkers to produce high-quality data with reduced artifacts. while our fully-automatic methods to construct sentence pairs yield negative results, we have not exhausted all possible automatic techniques for collecting similar sentences. however, given that we use state-of-the-art tools including faiss, roberta, and opus, and refine our methods with several rounds of piloting and tuning, we are skeptical that there is low-hanging fruit in the two directions we explored. a more radically different direction might involve generating pairs from scratch, using a large language model like @xcite . however, this would still require training data from crowdworker-written dataset, and might add a major source of potentially difficult-todiagnose bias. finally, despite its known issues, we find that mnli-style data is still the most effective for both nli evaluation and transfer learning, and future efforts to create similar data should work from that starting point.