Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inference (2022)

Abstract: We introduce a neuro-symbolic natural logic framework based on reinforcement learning with introspective revision. The model samples and rewards specific reasoning paths through policy gradient, in which the introspective revision algorithm modifies intermediate symbolic reasoning steps to discover reward-earning operations as well as leverages external knowledge to alleviate spurious reasoning and training inefficiency. The framework is supported by properly designed local relation models to avoid input entangling, which helps ensure the interpretability of the proof paths. The proposed model has built-in interpretability and shows superior capability in monotonicity inference, systematic generalization, and interpretability, compared with previous models on the existing datasets.

## introduction in the past decade, deep neural networks have achieved impressive performance on modeling natural language inference (nli) @xcite @xcite , which aims to determine the entailment relations between a premise sentence and its corresponding hypothesis. progress in nli has greatly benefited from the models' capabilities at approximating complex underlying functions, discovering and utilizing rich (true and/or spurious) patterns, and exhibiting robustness to noise and ambiguity. however, the black-box models inherently lack interpretability, and still fail to capture many aspects of human reasoning, including monotonicity inference @xcite , systematic * equal contribution. a recent research trend has attempted to advance the long-standing problem of bringing together the complementary strengths of neural networks and symbolic models @xcite @xcite @xcite @xcite . specifically for natural language, natural logic has long been studied to model reasoning in human language @xcite @xcite @xcite @xcite @xcite . however, the work of investigating the joint advantage of neural networks and natural logic is sparse @xcite (see section 2 for more details) and understudied. in this paper, we present a neuro-symbolic framework that integrates natural logic with neural networks for natural language inference. at the local level, we explore appropriate transformer networks to model the local relations between the constituents of a premise and hypothesis, in order to prevent attention from fully entangling the input, which otherwise can seriously impair the interpretability of proof paths built on local relations. we then construct natural logic programs and use reinforcement learning to reward the aggregation of the local relations. when reinforcement learning passes the final reward signals (nli labels) through the neural natural logic composition network, it faces the challenges of excessive spurious programs (incorrect programs that lead to correct final nli labels) as well as training inefficiency; the former is particularly harmful to interpretability. our framework leverages the proposed introspective revision method to discover better reward-earning operations and leverage external knowledge to reduce spurious proofs. we conducted experiments on six datasets: snli @xcite , help @xcite , med @xcite , monli @xcite , natlog-2hop @xcite , and a compositional generalization dataset @xcite . the results show the model's superior capability in monotonicity inferences, systematic generalization, and interpretability, compared with previous models on these existing datasets, while the model remains a competitive performance on the generic snli test set. natural language inference: natural language inference (nli) @xcite @xcite aims to identify the entailment relations between the premise-hypothesis sentence pairs. benefiting from pre-training on large-scale unlabeled corpora and then fine-tuning on large crowd-sourced datasets like snli @xcite and multinli @xcite , the pretrained language models @xcite @xcite have achieved the state-of-the-art performance. however, recent work revealed several drawbacks of the current deep nli systems. the research in @xcite and @xcite has shown that deep nli models learn to utilize dataset biases and label-relevant artifacts for prediction. @xcite and @xcite showed that a dominating proportion of samples in snli and multinli are in upward monotone, and models trained on these datasets have limited ability to generalize to downward monotone. more recently, systematically generated datasets have been proposed to evaluate the current models' ability on compositional generalization and showed that pretrained transformers generalize poorly to unseen combinations of the semantic fragments @xcite @xcite . neural network with logic components for nli: recent work @xcite @xcite has started to combine neural networks with logic-based components. the work most related to ours is @xcite , which adapts esim @xcite to predict relations between tokens in a premise and hypothesis, and composes them to predict final inferential labels. rather than optimizing the likelihood of specific reasoning paths, the model maximizes the sum of the likelihood of all possible paths (i.e., marginal likelihood) that reach the correct final nli labels. as a result, the model potentially encourages a large set of spurious reasoning paths and has to rely on external prior and strong constraints to predict meaningful intermediate local relations. this paper, instead, proposes a reinforcement learning with introspective revision framework to sample and reward specific reasoning paths through the policy gradient method. the introspective revision leverages external commonsense knowledge to tackle spurious proof paths and training inefficiency, key issues in developing interpretable neuro-symbolic models. to support that, local relation components need to be carefully designed. we will demonstrate that the proposed model substantially outperforms that proposed in @xcite on five datasets. policy gradient: policy gradient algorithms like reinforce @xcite have been used in neuro-symbolic models to connect neural representation learning and symbolic reasoning @xcite @xcite @xcite . the original reinforce algorithm suffers from sparse rewards and high variance in the gradient. to overcome these issues, the research presented in popov et al. (2017), goyal et al. (2019), and trott et al. (2019) proposes reward shaping, which leverages domain-specific knowledge to carefully design the reward functions. instead of learning only from the desired outcomes, some approaches also learn from failed attempts. hindsight experience replay (her) (andrychowicz et al., 2017) and scheduled auxiliary control (sac-x) (riedmiller et al., 2018) can replay the failed episodes and provide the agent with auxiliary learning goals to enable sample-efficient learning. li et al. (2020) propose a back-search algorithm, which diagnoses the failed reasoning processes and corrects potential errors to facilitate model training. based on li et al. (2020), we propose the introspective revision method, which leverages external knowledge to effectively discover reward-earning reasoning programs and to alleviate spurious reasoning. 3 background our model's backbone logic framework is based on the maccartney and manning (2009) variant of the natural logic formalism. the inference system operates by mutating spans of text in a premise to obtain the corresponding hypothesis sentence, and generates proofs based on the natural logic relations of the mutations. to extend the entailment relations to consider semantic exclusion, maccartney and manning (2009) introduced seven set-theoretic relations b for modeling entailment relations between two spans of texts (see assuming the availability of the alignment between a premise and hypothesis, the system first infers the relations between aligned pairs of words or phrases. consider the top-left example in figure 1 : the relation between ''the child'' and ''the kid'' is equivalence (≡), same as the relation between ''does not love'' and ''doesn't like'', while ''sports'' reversely entails ( ) ''table-tennis''. the next step is monotonicity inference. monotonicity is a pervasive feature of natural language that explains the impact of semantic composition on entailment relations @xcite @xcite . similar to the monotone functions in calculus, upward monotone keeps the entailment relation when the argument ''increases'' (e.g., cat animal). downward monotone keeps the entailment relation when the argument ''decreases'' (e.g., in all animals all cats). the system performs monotonicity inference through a projection function ρ : @xmath0, which is determined by the context and projection rules. built on that, the system aggregates/composes the projected local relations to obtain the inferential relation between a premise and hypothesis. specifically, ## experiments we evaluate the performance of the proposed model on six nli tasks from various perspectives: the ability of performing monotonicity inference (section 5.2), reasoning systematicity (section 5.3), and model interpretability (section 5.4). our model is trained on stanford natural language inference (snli) @xcite , in which the relation between a premise and hypothesis is classified to either entailment, contradiction, or neutral. we set the unit reward μ = 1.0, and optimize our model with adam gradient descent for six epochs with a learning rate of 2e-5. we compare the models with discount factor γ ∈ {0.25, 0.50, 0.75, 1.00} @xmath1.05, 0.10, 0.20}. we found that the test accuracies are not sensitive to γ when γ ≥ 0.50, and we select γ = 0.50, = 0.20, which achieved the best validation accuracy on snli. for the introspective revision algorithm we set phase revision (knowl. / answ. / both) success rate of revision start 80.4% ( 85.2% / 8.1% / 6.7%) 59.4% end 81.7% ( 80.3% / 5.9% / 13.8%) 98.4% ## summary the proposed neuro-symbolic framework integrates the long-studied natural logic with reinforcement learning and introspective revision, effectively rewarding the intermediate proof paths and leveraging external knowledge to alleviate spurious reasoning. the model has built-in interpretability following natural logic, which allows for a wide range of intuitive inferences easily understandable by humans. experimental results show the model's superior capability in monotonicity-based inferences and systematic generalization, compared to previous models on the existing datasets, while the model keeps competitive performance on the generic snli test set.