New Protocols and Negative Results for Textual Entailment Data Collection (2020)

Abstract: Natural language inference (NLI) data has proven useful in benchmarking and, especially, as pretraining data for tasks requiring language understanding. However, the crowdsourcing protocol that was used to collect this data has known issues and was not explicitly optimized for either of these purposes, so it is likely far from ideal. We propose four alternative protocols, each aimed at improving either the ease with which annotators can produce sound training examples or the quality and diversity of those examples. Using these alternatives and a fifth baseline protocol, we collect and compare five new 8.5k-example training sets. In evaluations focused on transfer learning applications, our results are solidly negative, with models trained on our baseline dataset yielding good transfer performance to downstream tasks, but none of our four new methods (nor the recent ANLI) showing any improvements over that baseline. In a small silver lining, we observe that all four new protocols, especially those where annotators edit *pre-filled* text boxes, reduce previously observed issues with annotation artifacts.

## introduction the task of natural language inference (nli; also known as textual entailment) has been widely used as an evaluation task when developing new methods for language understanding tasks, and it has recently become clear that high-quality nli data can be useful in transfer learning as well, driving much of the recent use of the task: several recent papers have shown that training large neural network models on natural language inference data, then fine-tuning them for other language understanding tasks often yields substantially better results on those target tasks @xcite . this result holds even when starting from large models like bert @xcite that have already been pretrained extensively on unlabeled data @xcite @xcite . the largest general-purpose corpus for nli, and the one that has proven most successful in this setting, is the multi-genre nli corpus (mnli @xcite . mnli was designed informally for use in a benchmark task (with no consideration of transfer learning), and in any case, no explicit experimental research went into its design. further, data collected under mnli's data collection protocol has known issues with annotation artifacts which make it possible to perform much better than chance using only one of the two sentences that make up each example @xcite @xcite . this work experimentally evaluates four potential changes to the original mnli data collection protocol that are designed to improve either the ease with which annotators can produce sound training examples or the quality and diversity of those examples. we collect a baseline dataset of 8.5k examples that follows the mnli protocol with our annotator pool, followed by four additional datasets of the same size which isolate each of our candidate changes. (see figure 1 for a schematic.) we then compare all five in a set of experiments, focused on transfer learning, that look at our ability to use each of these datasets to improve performance on the eight downstream language understanding tasks in the superglue @xcite benchmark. all five of our datasets are consistent with the task definition that was used in mnli, which is in turn based on the definition introduced by @xcite . in this task, each example consists of a pair of short texts: a premise and a hypothesis. the model is asked to read both texts and make a three-way classification decision: given the premise, would a reasonable person infer that hypothesis must be true @xcite , that it must be false (contradiction), or that there is not enough information to judge (neutral)? while it is certainly not clear that this design is optimal for any application, we leave a more broad-based exploration of task definitions for future work. our base data collection protocol follows mnli closely in asking annotators to read a premise sentence and then write three corresponding hypothesis sentences in empty text boxes corresponding to the three different labels (entailment, contradiction, and neutral). when an annotator follows this protocol, they produce three sentence pairs at once, all sharing a single premise. our paragraph protocol tests the effect of supplying annotators with complete paragraphs, rather than sentences, as premises. longer texts offer the potential for discourse-level inferences, the addition of which should yield a dataset that is more difficult, more diverse, and less likely to contain trivial artifacts. however, one might expect that asking annotators to read full paragraphs should increase the time required to create a single example; time which could potentially be better spent creating more examples. our editpremise and editother protocols test the effect of pre-filling a single seed text in each of the three text boxes that annotators are asked to fill out. by reducing the raw amount of typing required, this could allow annotators to produce good examples more quickly. by encouraging them to keep the three sentences similar, it could also indirectly facilitate the construction of minimalpair-like examples that minimize artifacts, in the style of @xcite . we test two variants of this idea: one uses a copy of the premise sentence as a seed text and the second retrieves a new sentence from an existing corpus that is similar to the premise sentence, and uses that. our contrast protocol tests the effect of adding artificial constraints on the kinds of hypothesis sentences annotators can write. giving annotators difficult and varying constraints could encourage creativity and prevent annotators from falling into patterns in their writing that lead to easier or more repetitive data. however, as with the use of longer contexts in paragraph, this protocol risks substantially slowing the annotation process. we experiment with a procedure inspired by that used to create the language-and-vision dataset nlvr2 @xcite , in which annotators must write sentences that show some specified relationship (entailment or contradiction) to a given premise, but do not show that relationship to a second similar distractor premise. because we see transfer learning as the primary application area for which it would be valuable to collect additional large-scale nli datasets, we focus our evaluation on this setting, and do not collect or designate test sets for the experimental datasets we collect. in transfer evaluations on the superglue benchmark @xcite , our base dataset and the datasets collected under our four new protocols offer substantial improvements in transfer ability over a plain roberta or xlnet model, comparable to the gains seen with an equally-sized sample of mnli. however, base reliably shows the strongest transfer results. this finding, combined with a low variance across runs, strongly suggests that none of these four interventions improves the suitability of nli data for transfer learning. we also observe that base, paragraph, editpremise, and editother all require very similar amounts of annotator time, reducing the potential downside of paragraph, but also invalidating the primary motivation behind ed-itpremise and editother. while our primary results are negative, we also observe that all four of these methods produce data of comparable subjective quality to base while significantly reducing the incidence of previously reported annotation artifacts. two recent papers have investigated methods that could augment the base mnli protocol we study here. anli @xcite collects new examples following this protocol, but adds an incentive for crowdworkers to produce sentence pairs on which a baseline system will perform poorly. @xcite introduce a method for expanding an already-collected dataset by making minimal edits to existing examples that change their labels, with the intent to better teach models to isolate the factors that are causally responsible for the label assignments. both of these papers offer in addition to nlvr2 (which motivated our contrast protocol), winogrande @xcite also showed promising results from the use of artificial constraints during the annotation process for another style of dataset. the observation that nli data can be effective in pretraining was first reported for snli and mnli by @xcite on models pretrained from scratch on nli data. this finding was replicated in the setting of multi-task pretraining by @xcite . this was later extended to the context of intermediate training-where a model is pretrained on unlabeled data, then on relatively abundant labeled data (mnli), and finally scarce task-specific labeled data-by phang et al. (2018), clark et al. (2019), liu et al. (2019a), yang et al. (2019), and liu et al. (2019b ) across a range of large pretrained models models and target language-understanding tasks. similar results have been observed with transfer from the other reasoning-oriented datasets @xcite , especially to target tasks centered on common sense. another related body of work @xcite @xcite has explored the broader empirical landscape of which supervised nlp tasks can offer effective pretraining for other supervised nlp tasks. ## data collection the annotation interface for our tasks is similar to that used for snli and mnli: we provide a premise from a preexisting text source and ask human annotators to provide three hypothesis sentences: one that says something true about the fact or situation in the prompt (entailment), one that says something that may or may not be true about the fact or situation in the prompt (neutral)-with the additional instruction that this sentence should discuss the same topic as the prompt but could be either true or false because the prompt does not provide enough information to be sure-and one that definitely does not say something true about the fact or situation in the prompt (contradiction). we evaluate five variants of this interface: base we show annotators a premise sentence and ask them to compose one new sentence for each label. paragraph we use the same instructions as base, but with full paragraphs, rather than single sentences, as the supplied premises. editpremise we pre-fill three text boxes with editable copies of the premise sentence, and ask annotators to edit each text field to compose sentences that match the three different labels. annotators may delete the pre-filled text. editother we follow the same procedure as editpremise, but rather than pre-filling the premise as a seed sentence, we instead use a similarity search method to retrieve a new sentence that is similar to the premise. contrast we again retrieve a second sentence that is similar to the premise, but we display it as a contrasting premise rather than using it to seed an editable text box. we then ask annotators to compose two new sentences: one sentence must be true only about the fact or situation in the first premise (that is, contradictory or neutral with respect to the contrasting premise). the other sentence must be false only about the fact or situation in the first premise (and true or neutral with respect to the contrasting premise). this yields an entailment pair and a contradiction pair, both of which use only the first premises, with the contrasting premise serving only as a constraint on the annotation process. we could not find a sufficiently intuitive way to collect neutral sentence pairs under this protocol and opted to use only two classes rather than increase the difficulty of an already unintuitive task. ## modeling experiments we run three types of machine learning experiments: sanity check experiments where we train and test on the nli task-both in a standard setting and in a hypothesis-only limited-input setting to measure relevant annotation artifacts-and our primary evaluation experiments in which we train models on nli before evaluating them on other tasks through transfer learning. these experiments generally compare models trained on ten nli datasets: each of the five 8.5kexample training sets introduced in this paper; the we run our experiments using jiant 1.2 @xcite , which implements the superglue tasks, mnli, and anli, and in turn builds on transformers @xcite , allennlp @xcite , and pytorch @xcite . to make it possible to train these large models on single consumer gpus, we use small-batch (@xmath0. 5 we train for up to 2 epochs for the very large record, 10 epochs for the very small cb, copa, and wsc, and 4 epochs for the remaining tasks. except where noted, all results reflect the median final performance from three random restarts of training. 6 direct nli evaluations as a preliminary sanity check, our base data yields a model that performs somewhat worse than a comparable mnli gov. the main conclusion we draw from these results is that none of the first three interventions improve performance on the out-of-domain glue diagnostic set, suggesting that they do not help in the collection of high-quality training data that is consistent with the mnli label definitions. we also observe that the newer anli data yields worse performance than mnli on the out-of-domain evaluation data when we control for dataset size. hypothesis-only models to further investigate the degree to which our hypotheses contain artifacts that reveal their labels, our first three interventions, especially edit-premise, show much lower hypothesis-only performance than base. this drop is much larger than the drop seen in our standard nli experiments in the self column of transfer evaluations for our primary evaluation, we use the training sets from our datasets in stilts-style intermediate training @xcite : we fine-tune a large pretrained model on our collected data using standard fine-tuning procedures, then fine-tune copies of the resulting model again on each of the target task datasets we use. we then measure the aggregate performance of the resulting models across those evaluation datasets. we evaluate on the target tasks in the super-glue benchmark @xcite : which consists of standardized splits and metrics for the question answering tasks boolq @xcite , multirc @xcite , record @xcite ; the entailment and reasoning tasks commitmentbank (cb; de marneffe et al., 2019), choice of plausible alternatives (copa; @xcite , recognizing textual entailment (rte; @xcite @xcite , and the winograd schema challenge (wsc; @xcite ; and the word sense disambiguation task wic (pilehvar and camacho-collados, 2019). these tasks were selected to be difficult for bert but relatively easy for crowdworkers, and are meant to replace the largely-solved glue benchmark @xcite . superglue does not include labeled test data, and does not allow for substantial ablation analyses on its test sets. since we have no single final model whose performance we aim to show off, we do not use the test sets. we train our wsc model in the standard way without adding data or modifying the format (as in @xcite . without these modifications, few of our models exceed chance accuracy. results are shown in we chose to collect 8,500-example samples because of the prior observation that this approximate amount was sufficient to show clear results on transfer learning, and we reproduce that finding here: both mnli 8.5k and the base dataset yield large improvements over plain roberta or xlnet through transfer learning. if any of our interventions were to be helpful in general, we would expect them to be harmless or helpful in our regime relative to base. this is not what we observe. we believe that this is the first study to evaluate anli as a pretraining task in transfer learning, and we observe that the large combined anli training set yields consistently better transfer than the original mnli dataset. however, we observe (to our surprise) that this result reverses when we control for anli's larger size, with an 8.5k-example sample of mnli yielding consistently better performance than an equivalently small sample of anli. our best overall result uses only 8.5k nli training examples, suggesting either that this size is enough to maximize the gains available through nli pretraining, or that the potential for models to forget skills learned in pretraining makes using larger intermediate datasets more challenging. ## conclusion our chief results on transfer learning are conclusively negative: all four interventions yield substantially worse transfer performance than our base mnli data collection protocol. however, we also observe promising signs that all four of our interventions help to reduce the prevalence of artifacts in the generated hypotheses that reveal the label. while these interventions may be helpful for future evaluation data, it appears that the type of creativity induced by our relatively open-ended base prompt works well for pretraining, and the resulting artifacts are a tolerable side-effect of that creativity. the need and opportunity that motivated this work remains compelling: human-annotated data like mnli has already proven itself as a valuable tool in teaching machines general-purpose skills for language understanding, and discovering ways to more effectively build and use such data could further accelerate the field's already fast progress toward robust, general-purpose language understanding technologies. on another note, most available text corpora, including our wikipedia source text and comparable past nli datasets, contain evidence of social inequalities and stereotypes, which models can easily learn to reproduce @xcite . our interventions are not meant to address this, and are likely orthogonal. bias mitigation in models and datasets remains a crucial direction for future work if systems based on datasets like the ones we study are to be widely deployed. beyond this: work on incentive structures and task design could facilitate the creation of crowdsourced datasets that are both creative and consistently labeled. machine learning methods work on transfer learning could help to better understand and exploit the effects that drive the successes we have seen with nli data so far. finally, there remains room for further empirical work investigating the kinds of task definitions and data collection protocols most likely to yield training data that teaches models transferrable skills.