Attentive Tree-structured Network for Monotonicity Reasoning (2021)

Abstract: Many state-of-art neural models designed for monotonicity reasoning perform poorly on downward inference. To address this shortcoming, we developed an attentive tree-structured neural network. It consists of a tree-based long-short-term-memory network (Tree-LSTM) with soft attention. It is designed to model the syntactic parse tree information from the sentence pair of a reasoning task. A self-attentive aggregator is used for aligning the representations of the premise and the hypothesis. We present our model and evaluate it using the Monotonicity Entailment Dataset (MED). We show and attempt to explain that our model outperforms existing models on MED.

## introduction in this paper, we present and evaluate a treestructured long-short-term-memory (lstm) network in which the syntactic information of a sentence is encoded and the alignment between the premise-hypothesis pair is calculated through a self-attention mechanism. our work builds on the child-sum tree-lstm from @xcite . we evaluate our model on several datasets to show that it performs well on both upward and downward inference. particularly, our model demonstrated good performance on downward inference, which is a difficult task for most nli models. natural language inference (nli), also known as recognizing textual entailment (rte) is one of the important benchmark tasks for natural language understanding. many other language tasks can benefit from nli, such as question answering, text summarization, and machine reading comprehension. the goal of nli is to determine whether a given premise p semantically entails a given hypothesis h @xcite . consider the example below: the hypothesis can be inferred from the premise and therefore the premise entails the hypothesis. to arrive at a correct determination, an nli model often needs to perform different inferences including various types of lexical and logical inferences. in this paper, we are concerned with monotonicity reasoning, a type of logical inference that is based on word replacement. below is an example of monotonicity reasoning: an upward entailing phrase (↑) can allow inference from (1a) to ( 1b ), where a more general concept laptop replaces the more specific macbook. a downward entailing phrase (↓) allows an inference from (1a) to ( 1c ), where a more specific context new students replaces the word students. the direction of the monotonicity can be reversed by adding a downward entailing phrase like "not"; thus (2a) entails (2b). recently, @xcite constructed a new dataset called the monotonicity entailment dataset (med). the purpose of that dataset is to evaluate the ability of a neural inference model to perform monotonicity reasoning. it is the first dataset ever created for such purpose. while many neural language models have shown state-of-art performance on large annotated nli dataset such as the stanford natural language inference (snli) dataset @xcite @xcite , many of these models did not perform well on monotonicity reasoning. in particular, they had low accuracy when performing downward monotonicity inference. additionally, most of the state-of-art inference models that do well on upward monotonicity inference perform poorly on downward inference @xcite . ## related work existing work in this area has adopted a recursive tree-structured neural network for natural language inference. bowman et al. (2015b) proposed a treestructured neural tensor network (treerntns) that can learn representations to correctly identify logical relationships such as entailment. @xcite extended the recursive neural tensor networks to a recursive long-short term memory network, a tree-lstm, which combines the advantages of both the recursive neural network structure and the sequential recurrent neural network structure. the tree-lstm can learn memory cells that reflect the historical memories of the descendant cells and thus improved the model's ability to process long-distance interaction over hierarchies, such as the language parse information. @xcite proposed a simple decompose attention model for natural language inference. their model relies on the attention to decompose the problem into sub-problems so that the smaller problems can be solved separately and in parallel. @xcite , proposed the enhanced sequential inference model (esim) for natural language inference task. it incorporated the sequential lstm encoder with the syntactic parsing information from the tree-lstm structure to form a hybrid neural inference mode. they found that incorporating the parsing information can improve the performance of the model. a new type of inference model that relies on external knowledge called the knowledge-based inference model (kim) was introduced by @xcite . they incorporated neural nli models with external knowledge in co-attention, local inference collection, and inference composition components. the kim model achieved state-of-art performance on the snli and mnli datasets. ## our model in this section we present an attentive tree structured network (attentivetreenet) with selfattention based aggregation. this model is composed of the following main components: input sentence embedding, attentive tree-lstm encoder, self-attention aggregator and a multi-layer perceptron (mlp) classifier. figure 1 shows the architecture of our model. given an input sentence pair, consisting of a premise p and a hypothesis h, the objective of the model is to determine whether p entails h. our model takes in four inputs: the word embeddings of the premise and hypothesis and the dependency parse trees of the premise and hypothesis. the model initializes the embedding of p and h with some pre-trained word embedding; the parse trees are produced by a dependency parser. our model forms a siamese neural network structure @xcite , in which the premise and the hypothesis are passed into a pair of identical tree-lstms that share the same parameters and weights. the main idea is to find a function that can map the input sentences into a target space such that we can approximate the semantic distance in the input space. ## conclusions in this paper, we explained our attentive treestructured network to perform monotonicity reasoning. our model combines a tree-structured lstm network and a self-attention mechanism, which is a potential mechanism for future natural language inference models, to incorporate syntactic structures of the sentence to improve sentence-level monotonicity reasoning. we evaluated our model and showed that it achieves better accuracy on monotonicity reasoning than other inference models. in particular, our model is performing significantly better on downward inference than others. we interpret the results of the experiments as supporting the thesis that using parse trees of a sentence are helpful in inferring the entailment relation. future research on the attentive tree network might extend a tree-lstm architecture by replacing the lstm cell with newer language models that have much better performance on various number of natural language processing tasks. one such model is the transformer model. furthermore, future work might want to investigate how different attention mechanism affect a model's performance.