CONDAQA : A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022)

Abstract: The full power of human language-based communication cannot be realized without negation. All human languages have some form of negation. Despite this, negation remains a challenging phenomenon for current natural language understanding systems. To facilitate the future development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs. We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage. We also have workers make three kinds of edits to the passage—paraphrasing the negated statement, changing the scope of the negation, and reversing the negation—resulting in clusters of question-answer pairs that are difficult for models to answer with spurious shortcuts. CONDAQA features 14,182 question-answer pairs with over 200 unique negation cues and is challenging for current state-of-the-art models. The best performing model on CONDAQA (UnifiedQA-v2-3b) achieves only 42% on our consistency metric, well below human performance which is 81%. We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facilitate the development of future NLP methods that work on negated language.

## introduction negation is fundamental to human communication. it is a phenomenon of semantic opposition, relating one expression to another whose meaning is in some way opposed. negation supports key properties of human linguistic systems such as contradiction and denial @xcite . despite the prevalence of negation, processing it effectively continues to elude models. here are just a few of the many recently reported failures: "the model [bert-large trained on squad] does not seem capable of handling...simple examples of negation" @xcite . "we find that indeed the presence of negation can significantly impact downstream quality [of machine translation systems]" @xcite . "state-of-the-art models answer questions from the vqa...correctly, but struggle when asked a logical composition including negation" @xcite . how can nlu systems meet this long-standing challenge? to facilitate systems that can process negation effectively, it is crucial to have high-quality evaluations that accurately measure models' competency at processing and understanding negation. in this work, we take a step toward this goal by contributing the first large-scale reading comprehension dataset, condaqa, focused on reasoning about negated statements in language. 1 the three-stage annotation process we develop to construct condaqa is illustrated in fig. 1 . we first collect passages from english wikipedia that contain negation cues, including single-and multiword negation phrases, as well as affixal negation. in the first stage, crowdworkers make three types of modifications to the original passage: (1) they paraphrase the negated statement, (2) they modify the scope of the negated statement (while retaining the negation cue), and (3) they undo the negation. in the second stage, we instruct crowdworkers to ask challenging questions about the implications of the negated statement. the crowdworkers then answer the questions they wrote previously for the original and edited passages. this process resulted in a dataset of 14,182 questions, covering a variety of negation cue types and over 200 unique negation cues, as well as a congold answers given by crowdworkers; answers predicted by instructgpt (text-davinci-002) prompted with 8 shots. see §2 for more details about each stage. we extensively benchmark baseline models on condaqa in three training data regimes: using all training examples, using only a small fraction (few-shot), or not using any examples (zeroshot). we show that condaqa is challenging for current models. finetuning unified-qa-3b @xcite -which was trained on 20 qa datasets-on condaqa, achieves the best result of 73.26% compared to human accuracy of 91.49%. further, we find that models are largely inconsistent; the best model achieves a consistency score of only 42.18% (40% below human consistency). this very low consistency score demonstrates that handling negation phenomena is still a major unresolved issue in nlp, along with sensitivity to contrasting data more generally. the dataset and baselines are available at https:// github.com/abhilasharavichander/condaqa . ## condaqa data collection this section describes our goals in constructing condaqa and our data collection procedure. design considerations our goal is to evaluate models on their ability to process the contextual implications of negation. we have the following four desiderata for our question-answering dataset: ## condaqa data analysis in this section, we provide an analysis of the passages, questions, edits, and answers in condaqa. descriptive statistics are provided in negation cues negation is expressed in many complex and varied ways in language @xcite . to characterize the distribution of types of negated statements in condaqa, we analyze the negation cues in wikipedia passages that annotators could select. figures 2 and 4 (appendix) show that the distribution over these cues and their grammatical roles is considerably diverse. moreover, there are 219 unique cues in condaqa and 75 novel cues in the test set that are unseen in the training data. this is a substantially wider range of negation cues than what is included in prior work; see appendix a for a detailed comparison. reasoning type passage snippet question answer explanation social norms (10%) on october 8, 1883, the us patent office ruled that edison's patent was based on the work of william e. sawyer and was, therefore, invalid . litigation continued for nearly six years. in 1885, latimer switched camps and started working with edison. from the information given in the passage, would you say that coincidence is the most charitable explanation for what was essentially the same innovation, in much the same way that newton and leibniz seemingly discovered calculus independently, without knowing of the other's progress? yes plagarism is frowned upon in society, more so than accidentally reaching the same conclusions as someone else. psychology (9%) [...] disraeli later romanticised his origins, claiming his father's family was of grand iberian and venetian descent; in fact isaac's family was of no great distinction [...] historians differ on disraeli's motives for rewriting his family history: [...] sarah bradford believes "his dislike of the commonplace would not allow him to accept the facts of his birth as being as middle-class and undramatic as they really were". would disraeli have been flattered by a biography that explored his middle class upbringing, according to bradford? no a person such as disraeli who wants to project a grandiose image of themselves is likely to be unhappy when people discuss mundane aspects about his upbringing. cause and effect (7%) oil produced from palm fruit is called 'red palm oil' or just 'palm oil'... in its unprocessed state, red palm oil has an intense deep red color because of its abundant carotene content. [...] would a consumer who was primarily interested in the eyehealth benefits of carotenes and lycopene want to shop for palm oils by their color, rather than listening to marketing slogans such as "extra virgin" or "minimally processed"? yes a high carotene content causes a deep red color, so a person searching for things with high carotene content can look at their color. ## baseline performance on condaqa we now evaluate state-of-the-art models' abilities to solve instances of condaqa. we evaluate models that we train either on the entire condaqa training data or few examples, as well as zero-shot models. we use two classes of metrics: accuracy the percentage of predictions which match the ground truth answer. if the answer is a span, this metric measures whether the prediction ## results model performance on condaqa is given in we create a contrastive dataset to be able to measure consistency as measuring models' ability to robustly predict answers across small input perturbations can provide a more accurate view of linguistic capabilities @xcite . here, there is a gap of ∼40% in consistency between humans and the best model. models are most robust to paraphrase edits: if a model answers a question correctly for the original passage, it is likely to be robust to changes in how that negation is expressed. we observe that the heuristic-based baselines exhibit low consistency, suggesting the consistency metric may be a more reliable measure than accuracy to evaluate models' ability to process negation. thus, mainstream benchmarks should consider including consistency as a metric to more reliably measure progress on language understanding. few-and zero-shot baselines do not match fully finetuned models' performance, but considerably improve over the majority baseline. for unifiedqa-v2 in particular, this suggests that some reasoning about implications of negation is acquired during pretraining. surprisingly, unifiedqa-v2 few-shot performance is worse than zero-shot. while this behavior has been reported for in-context learning with gpt-3 @xcite , we did not expect to observe this for a finetuned model. 2 unifiedqa-v2-3b finetuned with a few examples is comparable to instructgpt (text-davinci-002; at least 175b parameters) with in-context learning. chain-ofthought prompting (cot) notably improves the performance of instructgpt, especially in terms of the most challenging metrics: scope and affirmative consistency. in the zero-shot setting, the 11b version of unifiedqa-v2 performs the best, while the base version of only 220m parameters is comparable to instructgpt. unifiedqa-v2-11b is also better than flan-t5-xxl (a 11bparameter model as well). given that unifiedqa-v1 @xcite has been effective for tasks beyond qa @xcite , this result suggests that unifiedqa ## analysis while examining model errors, we find unifiedqa-v2-large has a negative correlation with question length (figure 7 in appendix d). humans can still reliably answer such long questions that are frequent in condaqa. we also analyze the performance of unifiedqa-v2-large across answer types, finding that: (i) the model performs best when the answer is "no", (ii) it almost never predicts "don't know", and (iii) its performance on span extraction questions is in-between those two extremes (figure 8 in appendix d). unifiedqa-v2-3b exhibits similar behavior, with improved performance on questions which admit "don't know" as an answer. we analyze questions across the wikipedia pas-sages and the passages with edited scopes, with the focus on: (i) instances where the true answer does not change with the edited scope and the model should be stable, and (ii) instances where the true answer does change and the model should be sensitive to the edit. we find that when the fully-finetuned unifiedqa-v2-3b (the bestperforming model) answers the question correctly for the wikipedia passage, it only produces the answer correctly for 63.23% of questions where the scope edit induces a different answer. in contrast, the model answers questions correctly for 91.03% of the instances where the answer does not change with the scope edit. 3 this suggests the model is not sensitive to changes of the scope of negated statements. we also analyze to what extent unifiedqa-v2-3b distinguishes between negated statements and their affirmative counterparts. we examine model predictions for 1080 sample pairs where the answer changes when the negation is undone. for 43.52% of these, the model changes its predictions. this suggests, in contrast to previous work @xcite , that models are sensitive to negated contexts to some extent. ## related work in aristotle's de interpretatione, all declarative statements are classified as either affirmations or negations used to affirm or contradict the occurrence of events @xcite . negation is expressed through a variety of formulations @xcite and is prevalent in english corpora @xcite . despite that, evidence from multiple tasks that require language understanding capabilities-such as nli @xcite , sentiment analysis @xcite @xcite , paraphrase identification @xcite , machine translation @xcite , and qa (ribeiro et al., 2020; sen and saffari, 2020)-identify negation as a challenging semantic phenomenon for models. @xcite analyze negation in 8 nlu datasets and conclude: "new corpora accounting for negation are needed to solve nlu tasks when negation is present". we expect condaqa will help. ## conclusion negation supports key properties of human linguistic systems such as the ability to distinguish between truth and falsity. we present condaqa, a qa dataset that contains 14,182 examples to evaluate models' ability to reason about the implication of negated statements. we describe a procedure for contrastive dataset collection that results in challenging questions, present a detailed analysis of the dataset, and evaluate a suite of strong baselines in fully-finetuned, few-shot, and zero-shot settings. we evaluate models on both their accuracy and consistency, and find that this dataset is highly challenging-even the best-performing model is 18 points lower in accuracy than our human baseline, and about 40 points lower in consistency. we expect that condaqa will facilitate nlu systems that can handle negation.