P ath R easoner: Modeling Reasoning Path with Equivalent Extension for Logical Question Answering (2024)

Abstract: Logical reasoning task has attracted great interest since it was proposed. Faced with such a task, current competitive models, even large language models (e.g., ChatGPT and PaLM 2), still perform badly. Previous promising LMs struggle in logical consistency modeling and logical structure perception. To this end, we model the logical reasoning task by transforming each logical sample into reasoning paths and propose an architecture PathReasoner. It addresses the task from the views of both data and model. To expand the diversity of the logical samples, we propose an atom extension strategy supported by equivalent logical formulas, to form new reasoning paths. From the model perspective, we design a stack of transformer-style blocks. In particular, we propose a path-attention module to joint model in-atom and cross-atom relations with the high-order diffusion strategy. Experiments show that PathReasoner achieves competitive performances on two logical reasoning benchmarks and great generalization abilities.

## introduction with the emergence of pre-trained language models (plms) @xcite , recent years have witnessed remarkable progress in the task of machine reading comprehension (mrc) @xcite . to tackle more complex scenarios in reality, the challenging logical reasoning task @xcite has been proposed to exploit the model reasoning capability @xcite ) over text 1 . similar to the traditional mrc task, it also takes the context, question and options as inputs and requires the model to predict the final answer. due to the diverse logical characteristics implied in the text, logical reasoning * correspondence to qika lin and jun liu. 1 logical reasoning is a broad concept covering various tasks, but we mainly address the task in the form of mrc. task brings huge challenges to current lms. especially, faced with such tasks, large language models (llms), e.g., chatgpt foot_0 and palm 2 foot_1 , also struggle a lot which is proved by previous evaluation works @xcite . under such circumstances, this paper will focus more on addressing logical reasoning tasks with small lms, which are light-weighted and more flexible for future applications 4 . previous competitive lms expose two limitations in the performance. firstly, it lacks consistent model predictions on samples with equal logical semantics. for example in figure 1 (a), we make changes to the expression of the original context while maintaining the semantic unchanged, where not...unless is equally transformed into the expression of only if. however, the lms give inconsistent predictions between the original sample and the modified one. we blame the problem on the lack of training samples in logical reasoning. compared with some classic mrc datasets like squad @xcite , coqa @xcite with over 100,000 training samples, logical reasoning datasets like reclor @xcite and logiqa @xcite are much more sparse with only several thousand samples. thus, such sparsity limits the learning of logic semantics. previous work @xcite leverages general corpus to conduct continual pretraining, but it does not address the sparsity of logical text in essential. secondly, it remains a challenge to enhance the model perception for logical structures. for example in figure 1 (b), we randomly replace the explicit logical relation words or inverse the negations for the context, which destroys the original semantics. but the lms fail to change the prediction accordingly. it demonstrates that current lms are insensitive to the logical connectives, instead they focus more on facts within the text. considering that current lms are pre-trained with general objectives on the fact corpus (e.g., wikipedia), they are naturally weak in capturing the logical structures usually existing in logical-specific scenarios. some studies like dagn @xcite , logiformer @xcite , and adalogn @xcite have attempted to model the explicit logical relations from various perspectives, such as causal and co-occurrence. all of them build text graphs to conduct the reasoning, which limits the scalability to larger text and more complex scenarios. in view of the above challenges, we propose an architecture pathreasoner, which considers a new paradigm for logical reasoning tasks via reasoning path modeling. based on the predefined logical rule forms, we represent each natural sentence as an atom and transform each sample into reasoning paths with confidence scores. under such a paradigm, pathreasoner addresses the task from two views. from the view of expanding the data diversity, we first obtain equivalent atom combinations through external logical formulas, generating new reasoning paths and textualizing them as new samples. from the model view, we propose a reasoning path modeling network. it encodes both function symbols and variables in atoms and forms an atom embedding sequence as the input. in a path-attention module, we model high-order relations from both in-atom and cross-atom perspectives. through the fusion of token, atom, and path embedding, the prediction can be derived. our technical contributions are as follows, and additional key values are in appendix i: (1) we unify the text inputs into atoms and reasoning paths. based on it, an architecture pathreasoner is proposed to improve both the diversity of samples and logic perception capability. ## related work recent progress in mrc promotes the emergence of more complex tasks like logical reasoning. previously, several datasets on logical reasoning have been proposed, including reclor @xcite , logiqa @xcite and ar-lsat @xcite . they have attracted much attention since some lms fail to show superiority. previous works on the logical reasoning task can be categorized into two folds. sequence-based. these models are usually accompanied by data augmentation strategies. lreasoner @xcite proposes to extend text with logical formulas to enrich the context information. merit @xcite proposes a contrastive strategy based on the meta-path and leverages the extra data to pre-train the model. however, both of them lack the relation modeling of logical units in the sequence. graph-based. dagn @xcite is the first work to divide the text into discourse units and utilize the graph neural networks @xcite to update the representations. but its chaintype graph structure limits the expression of complex relations between logical units. focalreasoner @xcite focuses on the fact triplet extracted from the text and builds a supergraph for reasoning. but it ignores the effects of the logical connectives within the text. to better model the logic within text, adalogn @xcite designs an adaptive network to update the text graph progressively. logiformer @xcite proposes a two-branch graph transformer network to address the text from syntax and logic. however, it is costly to form and update the text graph during the reasoning process. in general, the graph-based methods naturally lack expansibility, especially when the text becomes larger. considering the above drawbacks, we propose a reasoning pattern based on the reasoning paths (instantiated logical rules) for the first time. it models the logical reasoning task from a special perspective and combines the advantages of both sequence and graph-based methods. ## preliminary this work considers unifying the inputs into the form of logical rules @xcite since it is a more natural way to uncover logical structures of the text while maintaining the important facts. the distinctive values of such definitions over firstorder logic @xcite and propositional logic are in appendix c. we introduce the following two definitions. definition 1: atom. we transform each natural sentence into one atom @xcite , which consists of one function symbol and several variables. for example, given the sentence paula will visit the dentist only if bill goes golfing, we define the expression onlyif(a, b) as the atom. onlyif is the function symbol that denotes the explicit connective phrase in the sentence. and a, b are called variables to represent abstract sentence constitutes, whose instantiation are paula will visit the dentist and bill goes golfing respectively. similarly, we can also derive other atoms from the text, such as unless(a, b), since(a, b), infact(a). according to the reasoning patterns, we define four categories of function symbols, shown in rule body functions as the modeling of the context part, which is represented as the conjunction of category representative connectives cause because, since, dueto, thereasonswhy... sa if, when, once, aslongas, ... na onlyif, unless, ... fact infact, actually, inall, toconclude ... implementation we use over 100 pre-defined function symbols, grouped into four categories, and apply hand-crafted rules to match them in the sentence. the function symbols along with the punctuation can be divided into one or two parts, as instantiated variables. this strategy is relatively complete, illustrated in appendix b. ## methods to tackle the challenges in the logical reasoning task, we propose the architecture pathreasoner, shown in figure 2 . it includes two main parts: (a) equivalent path extension (epe) and (b) reasoning path modeling (rpm). the former module is aimed at expanding the sample diversity to improve the consistency of model prediction. the latter one targets at improving the logic perception capability of the reasoning model. ## experiments this section provides comparison experiments with other strong baselines on two logical reasoning benchmarks. extensive ablation studies and generalization evaluations are also followed. ## conclusion to tackle the logical data scarcity and weak model perception of logical structures, we propose a new paradigm to model the logical reasoning task by representing each natural sentence as atom form and transforming logical samples into reasoning paths. based on such unique modeling, an architecture pathreasoner is proposed to address the challenges. it achieves sota performances on two logical reasoning datasets. also, extensive experiments demonstrate the effectiveness of each module and great generalization capability on other complex reasoning scenarios. in the future, we will propose a unified architecture based on pathreasoner to tackle the logical reasoning tasks over different modalities (e.g., images, text, graphs).