Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference (2023)

Abstract: We introduce a synthetic dataset called Sentences Involving Complex Compositional Knowledge (SICCK) and a novel analysis that investigates the performance of Natural Language Inference (NLI) models to understand compositionality in logic. We produce 1,304 sentence pairs by modifying 15 examples from the SICK dataset (Marelli et al., 2014). To this end, we modify the original texts using a set of phrases modifiers that correspond to universal quantifiers, existential quantifiers, negation, and other concept modifiers in Natural Logic (NL) (MacCartney, 2009). We use these phrases to modify the subject, verb, and object parts of the premise and hypothesis. Lastly, we annotate these modified texts with the corresponding entailment labels following NL rules. We conduct a preliminary verification of how well the change in the structural and semantic composition is captured by neural NLI models, in both zero-shot and fine-tuned scenarios. We found that the performance of NLI models under the zero-shot setting is poor, especially for modified sentences with negation and existential quantifiers. After fine-tuning this dataset, we observe that models continue to perform poorly over negation, existential and universal modifiers.

## introduction natural language inference (nli) has made tremendous progress in recent years, both in terms of datasets, e.g., @xcite , multinli @xcite , adversarial nli @xcite , nli_xy @xcite , monalog @xcite , and methods @xcite @xcite . however, many of these directions lack explainability, a critical drawback that limits their applicability to critical domains such as medical, legal, or financial. in contrast, natural logic (nl) @xcite provides the necessary explainability through explicit compositionality that is driven by several relations that serve as building blocks (forward entailment (fe), reverse entailment (re), negation, cover, alternation, equivalence, and independence) as well as rules to combine them, which model changes in monotonicity. in this work, we analyze how well transformer networks trained for nli understand the atomic reasoning blocks defined in nl, and how well they can compose them to detect changes in monotonicity @xcite . to this end, we create a dataset containing 1304 sentences by modifying 15 premise/hypothesis pairs from the sick dataset @xcite . the dataset is generated by modifying the premise and hypothesis sentences selected, as follows: using this dataset, we analyzed the capacity of three different nli methods to correctly capture the change in entailment given the modified texts. in particular, the contributions of this work are as follows: the resulting dataset is freely available at https://github.com/clulab/releases/ tree/sushma/acl2023-nlrse-sicck . natural logic (nl) is a formal reasoning approach that makes use of syntactic structure and semantic properties of lexical items to understand compositionally @xcite . logical reasoning is a known challenge for neural nli models @xcite . in particular, nli models struggle to understand quantifiers, which is highlighted by the fact that these models do not generalize well over quantifierdriven inference tasks @xcite . the monotonicity calculus over quantifiers with tokenlevel polarity has been explored using the ccg parser over the sick dataset to generate a synthetic dataset that considers compositional data augmentation @xcite and monotonicity calculus @xcite . other recent research focused on language structures to highlight the importance of compositionality, i.e., the premise and hypothesis differ only in the order of the words, or the presence of antonyms, synonyms, or negation @xcite . having such data augmentation can help move closer to the compositional encoding of the language @xcite . our work extends this direction: our dataset captures both phrasal changes (e.g., synonyms, hypernyms), which we inherit from the sick dataset @xcite , as well as multiple types of modifiers that are critical for nli such as universal, existential, negation, and adjectives/adverbs. the fracas test suite @xcite contains 346 examples that explore aspects of natural logic applied to nli @xcite . the help dataset @xcite modifies phrases in premise/hypothesis sentences based on monotonicity reasoning from combinatorial categorical grammar @xcite and semantic tagging @xcite . as mentioned above, our work is complementary to such datasets, as we cover other types of text modifications. the med dataset @xcite is another manually-labeled dataset where hypotheses were also modified by the human labelers given the monotonicity information for the premises. similarly, we manually labeled nli information, but our work focuses mainly on compositional information in a sentential context. enhancing the dataset with data augmentation is another recent method to test the generalizability of nli models @xcite . lexical entailment acquired from the distributional behavior of word pairs @xcite led to the subsequent work of @xcite , who produced a 3-way classification task for nli dataset that serves as a benchmark for evaluating natural language understanding. using natural logic as a means to learn and reason about the semantic and lexical relations is a common method used to improve the reasoning capabilities of the nli models @xcite . the nli_xy dataset @xcite conducts structural investigation over the transformer-based nli models. in particular, the authors investigate how monotonicity (upwards or downwards) changes when the premises and hypotheses are modified through the insertion of hypernym/hyponym phrases. this work is complementary to ours: while they focus on monotonicity in lexicalization (e.g., changing from a hypernym to a hyponym), we focus on changes in monotonicity due to explicit modifiers applied on top of such lexical modifications. the monalog system @xcite introduces a simple yet explainable nli method that relies on a simplified natural logic implementation. the proposed method operates by implementing monotonicity calculus over ccg syntactic trees using "a small inventory of monotonicity facts about quantifiers, lexical items and token-level polarity." despite its simplicity, the authors report excellent performance on the sick dataset. more closely related to our work, they use monalog to generate additional training data for nli from the generated proofs. ## dataset we introduce a synthetic dataset to facilitate the analysis of compositionality in logic. the dataset contains 1,304 sentences that were created by modifying 15 examples from the sick dataset @xcite with a variety of modifiers. to this end, we used a set of phrases that correspond to universal quantifiers, existential quantifiers, negation, and other concept modifiers in natural logic (nl) @xcite . these modifiers were applied to syntactic constructs in both premise and hypothesis and the entailment labels are adjusted, as detailed below. ## evaluation we conducted an evaluation of how nli methods capture the explicit compositionality in our dataset using two configurations: a zero-shot setting, in which we used nli systems trained externally, and a fine-tuned setting, in which the same models were fine-tuned using our dataset. ## error analysis we analyze the incorrect predictions of the nli models over sicck dataset in this section. we observed that nli models performed better over adjectives and adverbs, and relatively well over universal quantifiers in comparison to sentences modified with negation and existential quantifiers under both fine-tuned as well as zero-shot settings. we also observed that models seem to be confused when the adjusted label was neutral and the modifier types were negations. svo # count neutral subject 86 65 verb 41 35 object 40 22 ## conclusion this paper introduced a new, synthetic dataset that facilitates analyses of how nli models capture compositionality. the dataset contains 1,304 sentence pairs that were created by modifying 15 examples from the sick dataset @xcite with a variety of modifiers that correspond to universal quantifiers, existential quantifiers, negation, and other concept modifiers in natural logic (nl) @xcite . we used these phrases to modify the subject, verb, and object parts of the premise and hypothesis. lastly, we annotated these modified texts with the corresponding entailment labels following nl rules. we conducted a preliminary analysis of how well the change in the structural and semantic composi-tion is captured and detected by neural nli models, in both zero-shot and fine-tuned scenarios. we found that the performance of nli models is poor in both settings, especially for modified sentences with negation and existential quantifiers, and when verbs are modified.