C onj NLI : Natural Language Inference Over Conjunctive Sentences (2020)

Abstract: Reasoning about conjuncts in conjunctive sentences is important for a deeper understanding of conjunctions in English and also how their usages and semantics differ from conjunctive and disjunctive boolean logic. Existing NLI stress tests do not consider non-boolean usages of conjunctions and use templates for testing such model knowledge. Hence, we introduce ConjNLI, a challenge stress-test for natural language inference over conjunctive sentences, where the premise differs from the hypothesis by conjuncts removed, added, or replaced. These sentences contain single and multiple instances of coordinating conjunctions (“and”, “or”, “but”, “nor”) with quantifiers, negations, and requiring diverse boolean and non-boolean inferences over conjuncts. We find that large-scale pre-trained language models like RoBERTa do not understand conjunctive semantics well and resort to shallow heuristics to make inferences over such sentences. As some initial solutions, we first present an iterative adversarial fine-tuning method that uses synthetically created training data based on boolean and non-boolean heuristics. We also propose a direct model advancement by making RoBERTa aware of predicate semantic roles. While we observe some performance gains, ConjNLI is still challenging for current methods, thus encouraging interesting future work for better understanding of conjunctions.

## introduction coordinating conjunctions are a common syntactic phenomenon in english: 38.8% of sentences in the penn tree bank have at least one coordinating word between "and", "or", and "but" @xcite . conjunctions add complexity to the sentences, thereby making inferences over such sen-tences more realistic and challenging. a sentence can have many conjunctions, each conjoining two or more conjuncts of varied syntactic categories such as noun phrases, verb phrases, prepositional phrases, clauses, etc. besides syntax, conjunctions in english have a lot of semantics associated to them and different conjunctions ("and" vs "or") affect the meaning of a sentence differently. recent years have seen significant progress in the task of natural language inference (nli) through the development of large-scale datasets like snli @xcite and mnli @xcite . although large-scale pre-trained language models like bert @xcite and roberta @xcite have achieved super-human performances on these datasets, there have been concerns raised about these models exploiting idiosyncrasies in the data using tricks like pattern matching @xcite . thus, various stress-testing datasets have been proposed that probe nli models for simple lexical inferences @xcite , quantifiers @xcite , numerical reasoning, antonymy and negation @xcite . however, despite the heavy usage of conjunctions in english, there is no specific nli dataset that tests their understanding in detail. although snli has 30% of samples with conjunctions, most of these examples do not require inferences over the conjuncts that are connected by the coordinating word. on a random sample of 100 conjunctive examples from snli, we find that 72% of them have the conjuncts unchanged between the premise and the hypothesis (e.g., "man and woman sitting on the sidewalk" → "man and woman are sitting") and there are almost no examples with non-boolean conjunctions (e.g., "a total of five men and women are sitting." → "a total of 5 men are sitting." (contradiction)). as discussed below, inference over conjuncts directly translates to boolean and non- he is a worcester resident and a member of the republican party. contradiction 4 a total of 793880 acre, or 36 percent of the park was affected by the wildfires. a total of 793880 acre, was affected by the wildfires. entailment 5 its total running time is 9 minutes and 9 seconds, spanning seven tracks. its total running time is 9 minutes, spanning seven tracks. contradiction † 6 he began recording for the columbia phonograph company, in 1889 or 1890. he began recording for the columbia phonograph company, in 1890. neutral † 7 fowler wrote or co-wrote all but one of the songs on album. fowler wrote or co-wrote all of the songs on album. in our work, we introduce conjnli, a new stress-test for nli over diverse and challenging conjunctive sentences. our dataset contains annotated examples where the hypothesis differs from the premise by either a conjunct removed, added or replaced. these sentences contain single and multiple instances of coordinating conjunctions @xcite with quantifiers, negations, and requiring diverse boolean and non-boolean inferences over conjuncts. we observe that state-of-the-art models such as bert and roberta, trained on existing datasets like snli and mnli, often fail to make these inferences for our dataset. for example, bert predicts entailment for the non-boolean "and" example #5 in we also present some initial model advancements that attempt to alleviate some of these challenges in our new dataset. first, we create synthetic training data using boolean and non-boolean heuristics. we use this data to adversarially train roberta-style models by an iterative adversarial fine-tuning method. second, we make roberta aware of predicate semantic roles by augmenting the nli model with the predicate-aware embeddings of the premise and the hypothesis. predicate arguments in sentences can help distinguish between two syntactically similar inference pairs with different target labels ( our work is positioned at the intersection of understanding the semantics of conjunctions in english and its association to nli. ## data creation creation of conjnli involves four stages, as shown in figure 1 . the (premise, hypothesis) pairs are created automatically, followed by manual verification and expert annotation. ## data analysis post-annotation, we arrive at a consolidated set of 1623 examples, which is a reasonably large size compared to previous nli stress-tests with expert annotations. we randomly split these into 623 validation and 1000 test examples, as shown in ## methods in this section, we first describe our iterative adversarial fine-tuning method (including the creation of adversarial training data), followed by some initial predicate-aware models to try to tackle conjnli. ## experiments and results we perform experiments on three datasets -(1) conjnli, (2) snli @xcite and ( 3 ) mnli @xcite . the appendix contains details about our experimental setup. ## conclusion we presented conjnli, a new stress-test dataset for nli in conjunctive sentences ("and", "or", "but", "nor") in the presence of negations and quantifiers and requiring diverse "boolean" and "nonboolean" inferences over conjuncts. large-scale pre-trained lms like roberta are not able to optimally understand the conjunctive semantics in our dataset. we presented some initial solutions via adversarial training and a predicate-aware roberta model, and achieved some reasonable performance gains on conjnli. however, we also show limitations of our proposed methods, thereby encouraging future work on conjnli for better understanding of conjunctive semantics.