Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? (2020)

Abstract: Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. In this paper, we introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language, namely, the regularity for performing arbitrary inferences with generalization on composition. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits. A series of experiments show that three neural models systematically draw inferences on unseen combinations of lexical and logical phenomena when the syntactic structures of the sentences are similar between the training and test sets. However, the performance of the models significantly decreases when the structures are slightly changed in the test set while retaining all vocabularies and constituents already appearing in the training set. This indicates that the generalization ability of neural models is limited to cases where the syntactic structures are nearly the same as those in the training set.

## introduction natural language inference (nli), a task whereby a system judges whether given a set of premises p semantically entails a hypothesis h @xcite , is a fundamental task for natural language understanding. as with other nlp tasks, recent studies have shown a remarkable impact of deep neural networks in nli @xcite @xcite . however, it remains unclear to what extent dnn-based models are capable of learning the compositional generalization underlying nli from given labeled training instances. systematicity of inference (or inferential systematicity) @xcite in natural language has been intensively studied in the field of formal semantics. from among the various aspects of inferential systematicity, in the context of nli, we focus on monotonicity @xcite and its productivity. consider the following premise-hypothesis pairs (1)-(3), which have the target label entailment: as in (1), for example, quantifiers such as some exhibit upward monotone (shown as [... ↑]), and replacing a phrase in an upward-entailing context in a sentence with a more general phrase (replacing puppies in p with dogs as in h) yields a sentence inferable from the original sentence. in contrast, as in (2), quantifiers such as no exhibit downward monotone (shown as [... ↓]), and replacing a phrase in a downward-entailing context with a more specific phrase (replacing cats in p with small cats as in h) yields a sentence inferable from the original sentence. such primitive inference patterns combine recursively as in (3). this manner of monotonicity and its productivity produces a potentially infinite number of inferential patterns. therefore, nli models must be capable of systematically interpreting such primitive patterns and reasoning over unseen combinations of patterns. although many studies have addressed this issue by modeling logical reasoning in formal semantics @xcite @xcite and testing dnn-based models on monotonicity inference @xcite , 2020), the ability of dnn-based models to generalize to unseen combinations of patterns is still underexplored. given this background, we investigate the systematic generalization ability of dnn-based models on four aspects of monotonicity: (i) systematicity of predicate replacements (i.e., replacements with a more general or specific phrase), (ii) systematicity of embedding quantifiers, (iii) productivity, and (iv) localism (see section 2.2). to this aim, we introduce a new evaluation protocol where we (i) synthesize training instances from sampled sentences and (ii) systematically control which patterns are shown to the models in the training phase and which are left unseen. the rationale behind this protocol is twofold. first, patterns of monotonicity inference are highly systematic, so we can create training data with arbitrary combinations of patterns, as in examples (1)-(3). second, evaluating the performance of the models trained with well-known nli datasets such as multinli @xcite might severely underestimate the ability of the models because such datasets tend to contain only a limited number of training instances that exhibit the inferential patterns of interest. furthermore, using such datasets would prevent us from identifying which combinations of patterns the models can infer from which patterns in the training data. this paper makes two primary contributions. first, we introduce an evaluation protocol 1 using 1 the evaluation code will be publicly available at https://github.com/verypluming/systematicity . the systematic control of the training/test split under various combinations of semantic properties to evaluate whether models learn inferential systematicity in natural language. second, we apply our evaluation protocol to three nli models and present evidence suggesting that, while all models generalize to unseen combinations of lexical and logical phenomena, their generalization ability is limited to cases where sentence structures are nearly the same as those in the training set. ## related work the question of whether neural networks are capable of processing compositionality has been widely discussed @xcite . recent empirical studies illustrate the importance and difficulty of evaluating the capability of neural models. generation tasks using artificial datasets have been proposed for testing whether models compositionally interpret training data from the underlying grammar of the data (lake and @xcite @xcite @xcite . however, these conclusions are controversial, and it remains unclear whether the failure of models on these tasks stems from their inability to deal with compositionality. previous studies using logical inference tasks have also reported both positive and negative results. assessment results on propositional logic @xcite , first-order logic @xcite , and natural logic @xcite show that neural networks can generalize to unseen words and lengths. in contrast, @xcite obtained negative results by testing models under fair conditions of natural logic. our study suggests that these conflicting results come from an absence of perspective on combinations of semantic properties. regarding assessment of the behavior of modern language models, @xcite @xcite investigated their syntactic capabilities by testing such models on subject-verb agreement tasks. many studies of nli tasks @xcite @xcite @xcite @xcite have provided evaluation methodologies and found that current nli models often fail on particular inference types, or that they learn undesired heuristics from the training set. in particular, recent works @xcite have evaluated models on monotonicity, but did not focus on the ability to generalize to unseen combinations of patterns. monotonicity covers various systematic inferential patterns, and thus is an adequate semantic phenomenon for assessing inferential systematicity in natural language. another benefit of focusing on monotonicity is that it provides hard problem settings against heuristics @xcite , which fail to perform downward-entailing inferences where the hypothesis is longer than the premise. ## conclusion we introduced a method for evaluating whether dnn-based models can learn systematicity of monotonicity inference under four aspects. a series of experiments showed that the capability of three models to capture systematicity of predicate replacements was limited to cases where the positions of the constituents were similar between the training and test sets. for embedding monotonicity, no models consistently drew inferences involving embedded clauses whose depths were two levels deeper than those in the training set. this suggests that models fail to capture inferential systematicity of monotonicity and its productivity. we also found that bert trained with our synthetic dataset mixed with multinli maintained performance on multinli while improving the performance on monotonicity. this indicates that though current dnn-based models do not systematically interpret monotonicity inference, some models might have sufficient ability to memorize different types of reasoning. we hope that our work will be useful in future research for realizing more advanced models that are capable of appropriately performing arbitrary inferences.