Decomposing Natural Logic Inferences for Neural NLI (2022)

Abstract: In the interest of interpreting neural NLI models and their reasoning strategies, we carry out a systematic probing study which investigates whether these modelscapture the crucial semantic features central to natural logic: monotonicity and concept inclusion. Correctly identifying valid inferences in downward-monotone contexts is a known stumbling block for NLI performance,subsuming linguistic phenomena such as negation scope and generalized quantifiers. To understand this difficulty, we emphasize monotonicity as a property of a context and examine the extent to which models capture relevant monotonicity information in the vector representations which are intermediate to their decision making process. Drawing on the recent advancement of the probing paradigm,we compare the presence of monotonicity features across various models. We find that monotonicity information is notably weak in the representations of popularNLI models which achieve high scores on benchmarks, and observe that previous improvements to these models based on fine-tuning strategies have introduced stronger monotonicity features together with their improved performance on challenge sets.

## introduction large, black box neural models which achieve high scores on benchmark datasets designed for testing natural language understanding are the subject of much scrutiny and investigation. it is often investigated whether models are able to capture specific semantic phenomena which mimic human reasoning and/or logical formalism, as there is evidence that they sometimes exploit simple heuristics and dataset artifacts instead @xcite . in this work, we consider the rigorous setting of natural logic @xcite . this is a highly systematic reasoning principle relying on only two abstract features, each of which is in itself linguistically complex: monotonicity and concept inclusion word-pair relations. it underlies the majority of symbolic/rule-based and hybrid approaches to nli and is an important baseline reasoning phenomenon to look for in a robust and principled nli model. downward monotone operators such as negation markers and generalized quantifiers result in the kinds of natural logic inferences which are often known to stump neural nli models that demonstrate high performance on large benchmark sets such as mnli @xcite : this has been identified in behavioural studies based on targeted challenge test sets, such as in @xcite and @xcite . in this work, however, we present a structural study: we investigate the extent to which the features relevant for identifying natural logic inferences, especially context monotonicty itself, are captured in the model's internal representations. to this end, we carry out a systematic probing study. our contributions are may be summarized as follows: ## related work natural logic dates back to the formalisms of @xcite , but has been received more recent treatments and reformulations in @xcite and @xcite . symbolic and hybrid neuro-symbolic implementations of the natural logic paradigm have been explored in @xcite ; @xcite ; abzianidze (2017) and @xcite . the shortcomings of natural logic handling in various neural nli models have been shown with several behavioural studies, where nli challenge sets exhibiting examples of downward monotone reasoning are used to evaluate performance of models with respect to these reasoning patterns @xcite @xcite . in an attempt to better identify linguistic features that neural models manage or fail to capture, researchers have employed probing strategies: namely, the diagnostic classification @xcite of auxiliary feature labels from internal model representations. most probing studies in natural language processing focus on the syntactic features captured in transformer-based language models @xcite , but calls have been made for more sophisticated probing tasks which rely more on contextual information @xcite . in the realm of semantics, probing studies have focused more on lexical semantics @xcite : word pair relations are central to monotonicity reasoning, and thus form part of our probing study as well, but the novelty of our work is the task of classifying context monotonicity from intermediate contextual embeddings. ## problem formulation: decomposing natural logic natural logic inferences (as formalized in sanchez 1991; maccartney and manning 2007) are usually described with respect to substitution operations. certain word substitutions result in either forward or reverse entailment, while others result in neither. this is the basis for a calculus of determining entailment from substitution sequences @xcite @xcite . broadly speaking, we wish to determine whether well-known transformer-based nli models mimic the reasoning strategies of natural logic. however, as neural nli models are black box classifiers that only see a premise/hypothesis sentence pair as its input, it is not immediate how to compare its process to a rule-based system. to this end, we consider a formulation of natural logic which describes its rules in terms of concept pair relations and context monotonicity (similar to @xcite . ## nli-xy dataset we follow this formalism as the basis for the nli-xy dataset. this is the first probing dataset in nlp where the auxiliary labels for intermediate semantic features influence the final task label in a rigid and deterministic (yet simple) way, with these features being themselves linguistically complex. as such, it is a "decomposed" natural logic dataset, where the positive entailment labels are further enriched with labels for the monotonicity and relational properties which gave rise to them. this allows for informative qualitative and structural analyses into natural logic handling strategies in neural nli models. the nli-xy dataset is comprised of the following: auxilliary label context f i did not eat any for breakfast. ↓ (downward monotone) insertion pair (x,y) (fruit, raspberries) = (reverse concept inclusion) nli label premise f (x) i did not eat any fruit for breakfast. entailment hypothesis f (y ) i did not eat any raspberries for breakfast. we present examples of the component parts and their composition in context extraction we extract context examples from two nli datasets which were designed for the behavioural analysis of nli model performance on monotonicity reasoning. in particular, we use the manually curated evaluation set med @xcite and the automatically generated help training set @xcite . by design, as they are collections of nli examples exhibiting monotonicity reasoning, these datasets mostly follow our required (f (x), f (y )) structure, and are labeled as instances of upward or downward monotonicity reasoning (although the contexts are not explicitly identified). we extract the common context f from these examples after manually removing a few which do not follow this structure (differing, for example, in pronoun number agreement or prepositional phrases). we choose to treat determiners and quantifiers as part of the context, as these are the kinds of closed-class linguistic operators whose monotonicity profiles we are interested in. to ensure grammatically valid insertions, we manually identify whether each context as suitable either for a singular noun, mass noun or plural noun in the blank/"x" position. insertion pairs our (x, y ) insertion phrase pairs come from two sources: firstly, the labeled word pairs from the monli dataset @xcite , which features only single-word noun phrases. secondly, we include an additional hand-curated dataset which has a small number of phrase-pair examples, which includes intersective modifiers (e.g. ("brown sugar", "sugar")) and prepositional phrases (e.g. ("sentence", "sentence about oranges")). several of these examples were drawn from the med dataset. each word in the pair is labelled as a singular, plural or mass noun, so that they may be permuted through the contexts in a reasonably grammatical way. premise/hypothesis pairs premise/hypothesis pairs are constructed by permuting insertion pairs through the set of contexts within the grammatical constraints. such a permutation strategy may generate examples which are not consistently meaningful, but we see the monotonicity reasoning pattern as sufficiently rigid and syntactic that it is of interest to observe how models treat less "meaningful" entailment examples that still hold with respect to the natural logic formalism: for example, "i did not swim in a person" entails "i did not swim in an irishman" at a systematic level. this does raise a question of whether we do (or even should) observe certain systematic behaviours on out-of-distribution examples: we leave the further investigation of this matter for future work. lastly, we note that the data is split into train, dev and test partitions before this permutation occurs, so that there are no shared contexts or insertion pairs between the different data partitions, in an attempt to avoid overlap issues such as those discussed in @xcite . the full dataset statistics are reported in ## experimental setup our experiments are designed to investigate the following questions: firstly, how do nli models compare in their learned encoding of context monotonicity and lexical relational features? secondly, if a model successfully captures these features, to what extent do they correspond with the model's predicted entailment label? we investigate these questions with a detailed probing study and a supporting qualitative analysis, using decomposed error break-downs and representation visualization. ## conclusion in summary, the nli-xy has enabled us to present evidence that explicit context monotonicity feature clustering in neural model representations seems to correspond to better performance on natural logic challenge sets which test downward-monotone reasoning. in particular, many popular models trained on mnli seem to lack this behaviour, accounting for previous observations that they systematically fail in downward-monotone contexts. furthermore, the probes' labels also have some explanatory value: both entailment and nonentailment labels can each further be broken down into sub-regions. this qualifies the classification with the observations that the data point occurs in a cluster of examples with a) upward (resp. downward) contexts and b) a forward (resp. backward) containment relation between the substituted noun phrases. in this sense, the analyses in this work can thus be interpreted as an explainable "decomposition" of the treatment natural logic examples in neural models.