Supporting Context Monotonicity Abstractions in Neural NLI Models (2021)

Abstract: Natural language contexts display logical regularities with respect to substitutions of related concepts: these are captured in a functional order-theoretic property called monotonicity. For a certain class of NLI problems where the resulting entailment label depends only on the context monotonicity and the relation between the substituted concepts, we build on previous techniques that aim to improve the performance of NLI models for these problems, as consistent performance across both upward and downward monotone contexts still seems difficult to attain even for state of the art models. To this end, we reframe the problem of context monotonicity classification to make it compatible with transformer-based pre-trained NLI models and add this task to the training pipeline. Furthermore, we introduce a sound and complete simplified monotonicity logic formalism which describes our treatment of contexts as abstract units. Using the notions in our formalism, we adapt targeted challenge sets to investigate whether an intermediate context monotonicity classification task can aid NLI models’ performance on examples exhibiting monotonicity reasoning.

## introduction nli has seen much success in terms of performance on large benchmark datasets, but there are still expected systematic reasoning patterns that we fail to observe in the state of the art nli models. we focus in particular on monotonicity reasoning: a large class of nli problems that can be described as a form of substitutional reasoning which displays logical regularities with respect to substitution of related concepts. in this setting, a subphrase a of a premise p(a) is replaced with a phrase b, yielding a hypothesis p(b). usually, the resulting entailment label relies on exactly two properties: the inclusion relation between concepts a and b, and the systematic behaviour of the context p with respect to such relations. in formal semantics, this is referred to as the monotonicity of p (where p is either upward or downward monotone), and this reasoning pattern is referred to as monotonicity reasoning. monotonicity reasoning is incredibly systematic, and thus is a much-probed behaviour in enquiries into the systematicity and generalization capability of neural nli models @xcite @xcite @xcite . determining both the concept relation and the context monotonicity requires significant linguistic understanding of syntactic structure and scope of operators, but in terms of reasoning, this is a very systematic pattern that nevertheless has a history of causing problems for neural models. it has been observed @xcite that current state of the art transformer-based nli models tend to routinely fail in downward monotone contexts, such as those arising in the presence of negation or generalized quantifiers. recent strategies @xcite to address the shortcomings of nli models in downwardmonotone contexts have followed the inoculation method @xcite : additional training data which exhibits the target phenomenon (in this case, downward-monotone reasoning) is used to fine-tune existing models. this is done with some success in @xcite and @xcite . in contrast, we wish to investigate a transfer learning strategy that directly targets the monotonicity question as an additional training task to see if this can further improve the monotonicity reasoning performance of popular transformer-based nli models. our contributions are as follows: ## related work the study of monotonicity in natural language has a strongly developed linguistic and mathematical theoretical groundwork, dating back to the monotonicity calculus of @xcite and in semantic studies such as @xcite . its formal treatments have led to the expansion of typed lambda calculus with an order relation so as to model this order-theoretic behaviour, resulting in a corresponding deduction system and completeness theorem in @xcite . there are varying presentations and some variation in terminology, but for the most part monotonicity refers to the order-theoretic property of the context as a function, while the term polarity usually refers to the tag assigned to the node in a ccg parse tree or a word in a sentence. the inferential mechanism based on monotonicity properties of quantifiers, determiners and contexts in general is sometimes referred to as natural logic, and the underlying principles of natural logic applying to set-theoretic concept relations has led to work on generalized monotonicity @xcite . however, the additional relations such as negation, alternation and cover are no longer order-theoretic notions. ## experiments building on the observations in the abovementioned previous papers, we ask the following questions: our investigation proceeds in three parts: firstly, we attempt to fine-tune a sota nli model for a context monotonicity classification task. secondly, we retrain the above model for nli and evaluate the performance on several evaluation datasets which specifically target examples of both upward and downward monotonicity reasoning. we examine whether there is any improvement over a previously suggested approach on finetuning on a large, automatically generated dataset (help) from @xcite . ## discussion average performance firstly, we confirm previous observations that the starting pretrained transformer model roberta-large-mnli (which is considered a high-performing nli model, achieving over 93% accuracy on the large mnli development set) has a dramatic performance imbalance with respect to context monotonicity. the fact that performance on downward monotone contexts is as low as 5% suggests that this model perhaps routinely assumes upward monotone contexts. it was noted in @xcite that the mnli benchmark dataset is strongly skewed in favor of upward monotone examples, which may account for this. our approach outperforms or matches the baseline models in three of the summary accuracy scores, and is competitive in the fourth. furthermore, in most cases we observe less performance loss on the benchmark sets. ## conclusion and future work introducing context monotonicity classification into the training pipeline of nli models provides some performance gains on challenge datasets designed to test monotonicity reasoning. we see contexts as crucial objects of study in future approaches to natural language inference. the ability to detect their logical properties (such as monotonicity) opens the door for hybrid neuro-symbolic nli models and reasoning systems, especially in so far as dealing with out of domain insertions that may confuse out-of-the-box nli models. the linguistic flexibility that transformer-based language models bring is too good to lose; leveraging their power in situations where only part of our sentence is in a model's distribution would be helpful for domain-specific use cases with many out-ofdistribution nouns. overall, we are interested in furthering both the analysis and improvement of emergent modelling of abstract logical features in neural natural language processing models.