Exploring End-to-End Differentiable Natural Logic Modeling (2020)

Abstract: We explore end-to-end trained differentiable models that integrate natural logic with neural networks, aiming to keep the backbone of natural language reasoning based on the natural logic formalism while introducing subsymbolic vector representations and neural components. The proposed model adapts module networks to model natural logic operations, which is enhanced with a memory component to model contextual information. Experiments show that the proposed framework can effectively model monotonicity-based reasoning, compared to the baseline neural network models without built-in inductive bias for monotonicity-based reasoning. Our proposed model shows to be robust when transferred from upward to downward inference. We perform further analyses on the performance of the proposed model on aggregation, showing the effectiveness of the proposed subcomponents on helping achieve better intermediate aggregation performance.

## introduction a recent research trend has attempted to further advance the long-standing problem of bringing together the complementary strengths of neural networks and symbolic models, e.g., the research performed in @xcite @xcite @xcite @xcite , among others. it is known that neural models can approximate complex functions and are robust to noise and ambiguity, while symbolic models often render superior explainability and interpretability but are brittle and prone to fail in the presence of noise and uncertainty. the majority of research efforts are based on some abstract logical forms such as the first-order logic (fol) or its fragments. for natural language, obtaining such a representation is known to face many thorny challenges. natural logic instead aims to sidestep some of the challenges by performing inferences over surface forms of text based on monotonicity or projectivity @xcite @xcite , and has been applied to tasks such as natural language inference @xcite and question answering @xcite . in this work we explore differentiable natural logic models that integrate natural logic with neural networks, with the aim to keep the backbone of inference based on the natural logic formalism, while introducing subsymbolic vector representations and neural components into the framework. combining the advantages of neural networks with natural logic needs to take several basic problems into consideration. two problems flow directly from this objective: 1) how (and where) to leverage the strength of neural networks in the natural logic formalism, and; 2) how to alleviate the issue of a lack of intermediate supervision for training sub-components, which may lead to the spurious problem @xcite in the end-to-end training. we explore a framework in which module networks @xcite ) are leveraged to model the natural logic operations, which is enhanced with a memory module component to @xcite . ## background this section briefly reviews the natural logic formalism (maccartney and manning, 2009) that our work is based on. for more details, we refer readers to @xcite @xcite . monotonicity is a pervasive feature of natural language and an essential concept in natural logic @xcite @xcite . similar to the monotone functions in calculus, in natural language upward monotone keeps the entailment relation when the argument "increases" (e.g., some cats are playing some animals are playing, where cats is replaced by its hypernym animals). downward monotone keeps the entailment relation when the argument "decreases" (e.g., all animals are playing all cats are playing, where animals is replaced by its hyponym cats). to extend the monotonicity to consider exclusion, maccartney and manning (2009) investigate all sixteen equivalence classes of set relations and remove nine degenerate, semantically vacuous relations, thereby defining a seven-relation set @xmath0, , , ∧, | , , # } for natural logic, as shown in from a high-level perspective, the natural logic proof system proposed by maccartney and manning (2009) consists of the following steps. first, the alignment between two text spans (often two sentences) is obtained and then lexical relation recognition is performed for aligned pairs of words. consider a simplified example: a premise all animals outside are eating and a corresponding hypothesis all cats outside are playing, as shown in figure 1 . each pair of aligned words is assigned one of the relations in projection ρ : @xmath1. the projection operation has been implemented in the stanford natlog parser foot_1 . for a given sentence, natlog can output the projections at each word position. for example, built on this, relation aggregation is performed to aggregate multiple projected local relations, according to ## neural natural logic model we present a differentiable framework in which natural logic is integrated with neural networks. the overall architecture of the model is shown in figure 1 . at the core of the framework are natural logic operations modeled with memory-enhanced module networks, which are trained end-to-end to optimize the following objective: ## conclusions this paper studies end-to-end trained differentiable models that integrate natural logic with neural networks. the proposed model adapts module networks to model natural logic operations, which is a the 2-hop dataset figure 4 shows an example of the 2-hop dataset. the premise and hypothesis differ by two edits of word/phrase insertion, deletion, or substitution. the dataset provides the ground truth of aggregation at each time step (the equivalence relation is the default relation and is hence not included in the "ground truth of aggregation" section) and the word locations/indices associated with each edit. the 2-hop dataset is developed with the following three steps: