N eural L og: Natural Language Inference with Joint Neural and Logical Reasoning (2021)

Abstract: Deep learning (DL) based language models achieve high performance on various benchmarks for Natural Language Inference (NLI). And at this time, symbolic approaches to NLI are receiving less attention. Both approaches (symbolic and DL) have their advantages and weaknesses. However, currently, no method combines them in a system to solve the task of NLI. To merge symbolic and deep learning methods, we propose an inference framework called NeuralLog, which utilizes both a monotonicity-based logical inference engine and a neural network language model for phrase alignment. Our framework models the NLI task as a classic search problem and uses the beam search algorithm to search for optimal inference paths. Experiments show that our joint logic and neural inference system improves accuracy on the NLI task and can achieve state-of-art accuracy on the SICK and MED datasets.

## introduction currently, many nli benchmarks' state-of-the-art systems are exclusively deep learning (dl) based language models @xcite @xcite . these models often contain a large number of parameters, use high-quality pre-trained embeddings, and are trained on large-scale datasets, which enable them to handle diverse and large test data robustly. however, several experiments show that dl models lack generalization ability, adopt fallible syntactic heuristics, and show exploitation of annotation artifacts @xcite @xcite . on the other hand, there are logic-based systems that use symbolic reasoning and semantic formalism to solve nli @xcite @xcite * the first two authors have equal contribution figure 1 : analogy between path planning and an entailment inference path from the premise a motorcyclist with a red helmet is riding a blue motorcycle down the road to the hypothesis a motorcyclist is riding a motorbike along a roadway. @xcite . these systems show high precision on complex inferences involving difficult linguistic phenomena and present logical and explainable reasoning processes. however, these systems lack background knowledge and do not handle sentences with syntactic variations well, which makes them poor competitors with state-ofthe-art dl models. both dl and logic-based systems show a major issue with nli models: they are too one-dimensional (either purely dl or purely logic), and no method has combined these two ap-proaches together for solving nli. this paper makes several contributions, as follows: first, we propose a new framework in section 3 for combining logic-based inference with deeplearning-based network inference for better performance on conducting natural language inference. we model an nli task as a path-searching problem between the premises and the hypothesis. we use beam-search to find an optimal path that can transform a premise to a hypothesis through a series of inference steps. this way, different inference modules can be inserted into the system. for example, dl inference modules will handle inferences with diverse syntactic changes and logic inference modules will handle inferences that require complex reasoning. second, we introduce a new method in section 4.3 to handle syntactic variations in natural language through sequence chunking and dl based paraphrase detection. we evaluate our system in section 6 by conducting experiments on the sick and med datasets. experiments show that joint logical and neural reasoning show state-of-art accuracy and recall on these datasets. ## related work perhaps the closest systems to neurallog are @xcite , monalog @xcite , and hy-nli @xcite . using @xcite to work with logic representations derived from ccg trees, @xcite proposed a framework that can detect phrase correspondences for a sentence pair, using natural deduction on semantic relations and can thus extract various paraphrases automatically. their experiments show that assessing phrase correspondences helps improve nli accuracy. our system uses a similar methodology to solve syntactic variation inferences, where we determine if two phrases are paraphrases. our method is rather different on this point, since we call on neural language models to detect paraphrases between two sentences. we feel that it would be interesting to compare the systems on a more theoretical level, but we have not done the comparison in this paper. neurallog inherits the use of polarity marking found in monalog @xcite . (however, we use the dependency-based system of @xcite instead of the ccg-based system of @xcite .) monalog did propose some integration with neural models, using bert when logic failed to find entailment or contradiction. we are doing something very different, using neural models to detect paraphrases at several levels of "chunking". in addition, the exact algorithms found in sections 3 and 4 are new here. in a sense, our work on alignment in nli goes back to @xcite where alignment was used to find a chain of edits that changes a premise to a hypothesis, but our work uses much that simply was not available in 2009. hy-nli is a hybrid system that makes inferences using either symbolic or deep learning models based on how linguistically challenging a pair of sentences is. the principle hy-nli followed is that deep learning models are better at handling sentences that are linguistically less complex, and symbolic models are better for sentences containing hard linguistic phenomena. although the system integrates both symbolic and neural methods, its decision process is still separate, in which the symbolic and deep learning sides make decisions without relying on the other side. differently, our system incorporates logical inferences and neural inferences as part of the decision process, in which the two inference methods rely on each other to make a final decision. ## evaluation 6.1 experiment setup for universal dependency parsing, we follow chen and gao (2021)'s framework and use a parser model p r acc. ml/dl-based systems bert (base, uncased) 86.8 85.4 86.7 yin and schütze (2017) --87.1 beltagy et al. (2016) --85.1 logic-based systems abzianidze (2017) 98.0 58.1 81.4 martínez-gómez et al. (2017) 97.0 63.6 83.1 yanaka et al. (2018) 84.2 77.3 84.3 hu et al. (2020) 83.8 70.7 77.2 abzianidze (2020) 94.3 67.9 84.4 hybrid system hu et al. (2020)+bert 83.2 85.5 85.4 kalouli et al. (2020) --86.5 our system neurallog (full system) 88.0 87.6 90.3 -albert-sv 68.9 79.3 71.4 -monotonicity 74.5 75.1 74.7 ## conclusion and future work in this paper, we presented a framework to combine logic-based inference with deep-learning based inference for improved natural language inference performance. the main method is using a search engine and an alignment based controller to dispatch the two inference methods (logic and deeplearning) to their area of expertise. this way, logicbased modules can solve inference that requires logical rules and deep-learning based modules can solve inferences that contain syntactic variations which are easier for neural networks. our system uses a beam search algorithm and three inference modules (lexical, phrasal, and syntactic variation) to find an optimal path that can transform a premise to a hypothesis. our system handles syntactic variations in natural sentences using the neural network on phrase chunks, and our system determines contradictions by searching for contradiction signatures (evidence for contradiction). evaluations on sick and med show that our proposed framework for joint logical and neural reasoning can achieve state-of-art accuracy on these datasets. our experiments on ablation tests show that neither logic nor neural reasoning alone fully solve natural language inference, but a joint operation between them can bring improved performance. for future work, one plan is to extend our system with more logic inference methods such as those using dynamic semantics @xcite and more neural inference methods such as those for commonsense reasoning @xcite . we also plan to implement a learning method that allows the system to learn from mistakes on a training dataset and automatically expand or correct its rules and knowledge bases, which is similar to abzianidze (2020)'s work.