MERI t: M eta- P ath G uided C ontrastive L earning for L ogical R easoning (2022)

Abstract: Logical reasoning is of vital importance to natural language understanding. Previous studies either employ graph-based models to incorporate prior knowledge about logical relations, or introduce symbolic logic into neural models through data augmentation. These methods, however, heavily depend on annotated training data, and thus suffer from over-fitting and poor generalization problems due to the dataset sparsity. To address these two problems, in this paper, we propose MERIt, a MEta-path guided contrastive learning method for logical ReasonIng of text, to perform self-supervised pre-training on abundant unlabeled text data. Two novel strategies serve as indispensable components of our method. In particular, a strategy based on meta-path is devised to discover the logical structure in natural texts, followed by a counterfactual data augmentation strategy to eliminate the information shortcut induced by pre-training. The experimental results on two challenging logical reasoning benchmarks, i.e., ReClor and LogiQA, demonstrate that our method outperforms the SOTA baselines with significant improvements.

## introduction logical reasoning has long been recognized as one key critical thinking ability of human being. until very recently, some pioneer researchers have crystallized this for the nlp community, and built several public challenging benchmarks, such as recolor @xcite and logiqa @xcite . logical reasoning 2 requires to correctly infer the semantic relations with respect to the constituents among different sentences. a typical formulation of logical reasoning is illustrated figure 1 : an instance of logical reasoning from the re-clor dataset. to infer the right answer, we should uncover the underlying logical structure, as shown in the bottom. (x) represents the logical variable (e.g., entity or phrase) and r j denotes the relation (e.g., predicate) between two logical variables. rj is the passive relation of r j . in figure 1 , namely, a real-world examination instance from reclor. as can be seen, to find the correct answer for the given question, one needs to extract the logical structures residing in a pair of each option and the whole context, and justify its reasonableness. as a matter of fact, logical reasoning is still at its initial stage, thence, existing studies are somewhat rare in literature. some efforts have been devoted to designing specific model architectures or integrating symbolic logic as the hints attached to the potential logical structure. for instance, @xcite and @xcite first constructed a graph of different constituents and then performed implicit reasoning with graph neural networks (gnns). @xcite proposed lreasoner, a unified context extension and data augmentation framework based on the parsed logical expressions. these approaches have achieved some progress on benchmark datasets. however, though equipped with pre-trained language models, they still suffer from problems like overfitting and poor generalization. we attribute these drawbacks to the difficulty of building a model aware of the logical relations beneath natural language, which is revealed from two sides: 1) the high sparsity of the existing datasets, and 2) the goal of general pre-training, i.e., masked language modeling @xcite , which however, deviates largely from that of the logical reasoning. to tackle this issue, we aim to build a bridge between logical reasoning and self-supervised pre-training, and accordingly inherit the strong generalization power from pre-trained language models. our proposed method is inspired by the recent progress of contrastive learning based pre-training. it mainly consists of two novel components: metapath guided data construction and counterfactual data augmentation. both components are leveraged to perform automatic instance composition from unlabeled corpus (e.g., wikipedia) for contrastive learning. regarding the first component, we propose to employ the meta-path to define a symbolic form of logical structure. the intuition behind this is that the logical structure can be expressed as a reasoning path composed of a series of relation triplets, and a meta-path inherently offers such a means of consistency @xcite . specifically, given an arbitrary document and a pair of entities in it, we try to find a positive instance pair in the document according to the logical structure. and the negative ones can thus be generated by modifying the relations involved in the structure, which explicitly break the logical consistency. nevertheless, the contrastive learning often fails when models easily locate trivial solutions @xcite . in this context, the pre-trained language model may exclude the negative options through their conflicts with the world knowledge. to eliminate this information shortcut, in our second novel component, we devise a strong counterfactual data augmentation @xcite ) strategy. by mixing counterfactual data during pre-training, of which the positive instance pair is also against the world knowledge, this component shows more ad-vantage in reasoning over logical relations. we integrate this method with both al-bert @xcite and roberta @xcite foot_0 for further pre-training, and then fine-tune them on two downstream logical reasoning benchmarks, i.e., reclor and logiqa. the experimental results demonstrate that our method can outperform all the existing strong baselines, yet without any augmentation from the original training data. besides, the ablation studies also show the effectiveness of the two essential strategies in our method. the contribution of this paper is summarized as follows: ## method in this paper, we study the problem of logical reasoning on the task of multiple choice question answering (mcqa). specifically, given a passage p , a question q and a set of k options @xmath0, • • • , o k }, the goal is to select the correct option o y , where @xmath1, k]. notably, to tackle this task, we devise a novel pre-training method equipped with contrastive learning, where the abundant knowledge contained in the largescale wikipedia documents is explored. we then transfer the learned knowledge to the downstream logical reasoning task. ## conclusion and future work in this paper, we present merit, a meta-path guided contrastive learning method to facilitate logical reasoning via self-supervised pre-training. merit is built upon the meta-path strategy for automatic data construction and the counterfactual data augmentation to eliminate the information shortcut during pre-training. with the evaluation on two logical reasoning benchmarks, our method has obtained significant improvements over strong baselines relying on task-specific model architecture or augmentation of original dataset. pertaining to the further work, we plan to strengthen our method from both data construction and model architecture design angles. more challenging instances are expected to be constructed if multiple meta-paths can be considered at the same time. besides, leveraging gnns may bring better interpretability and generalization since the graph structure can be integrated into both pre-training and fine-tuning stages.