M eta L ogic: Logical Reasoning Explanations with Fine-Grained Structure (2022)

Abstract: In this paper, we propose a comprehensive benchmark to investigate models’ logical reasoning capabilities in complex real-life scenarios. Current explanation datasets often employ synthetic data with simple reasoning structures. Therefore, it cannot express more complex reasoning processes, such as the rebuttal to a reasoning step and the degree of certainty of the evidence. To this end, we propose a comprehensive logical reasoning explanation form. Based on the multi-hop chain of reasoning, the explanation form includes three main components: (1) The condition of rebuttal that the reasoning node can be challenged; (2) Logical formulae that uncover the internal texture of reasoning nodes; (3) Reasoning strength indicated by degrees of certainty. The fine-grained structure conforms to the real logical reasoning scenario, better fitting the human cognitive process but, simultaneously, is more challenging for the current models. We evaluate the current best models’ performance on this new explanation form. The experimental results show that generating reasoning graphs remains a challenging task for current models, even with the help of giant pre-trained language models.

## introduction being able to generate reasonable explanations is a crucial capability for a reliable reasoning system. most current works try to ask models to generate reasoning chains as profound explanations. from simple rationales @xcite to more complex multi-step explanations @xcite @xcite and deductive chains of reasoning @xcite @xcite , previous works attempt to encompass comprehensive information. however, the current explanation design still has limitations for logical reasoning texts sent1 certainty: contingent node formula: n/a sent4 certainty: necessary node formula: □(v 𝟐 → □v 𝟑 ) ## task definition overall generation task the desideratum is that a model reconstructs the fine-grained logic explanation for a given passage, which uncovers the model's understanding of the logic between the lines. the logic explanation is formatted as logic metagraphs with support or rebut inference steps, per-node logical formulae, and degrees of certainty, as demonstrated in figure 2 . the input for the models is a passage with multiple statements (s (0) , s (1) , ..., s (n ) ) and atomic sentences p n) , according to which they generate the logic metagraph. the logic metagraph has three main components: (1) the meta structure @xmath0, e), where @xmath1, and e s and e r are the two meta edge types, support and rebut, respectively, between the meta nodes u (n) ∈ v, @xmath2. (2) the set of node formulae f, where u (n) := f @xmath3. each formula is joined by logical triples. ## the logic metagraph in this section, we introduce the proposed logic metagraph in details. 3 ## metalogic in this section, we introduce the construction details of the metalogic dataset. since the logic metagraphs have fine-grained structures with multiple evaluation dimensions, which are all dispensable and supplement each other, we design a rigorous annotation process for the construction. ## experiment we evaluate the performance of the following explanation generative models on metalogic: (1) all-at-once (once) t5 @xcite , which performs sequence-to-sequence generation via generating the whole metagraph in a linearized sequence given the overall passages with the sentence and variable denotations; (2) multitask t5 (raffel et al., 2020), which complete the whole generation task with a combination of three sub-tasks: meta structure generation, formula generation, and certainty prediction; (3) metgen (hong et al., 2022), which is a module-based framework for structured explanation generation. it further introduces a reasoning controller and two modules for meta structure generation. to the best of our knowledge, metgen is the current state-of-the-art explanation generative model. further model details are in appendix e. following dalvi et al. (2021), we report the f1 and allcorrect scores for each dimension and the overall allcorrect score. for certainty, we report the accuracy of a five-label classification and an extra macro-f1 due to the unbalance of the degree labels. the overall allcorrect is the strictest metric since any difference in the predicted metagraph will make the prediction a wrong one. details can be found in appendix f. ## conclusion this paper extends the boundary of current research on logical graph generation for reliable reasoning systems. specifically, we carefully design a complete logic explanation form following previous research on cognitive science. accordingly, we built metalogic with a comprehensive annotation task design and quality examination. we also evaluate several recent models and show that the performance of current models is still unsatisfactory, even with giant pre-trained language models. we hope that this paper could motivate more future works on reliable reasoning systems that could generate the correct logical graphs to support their reasoning. the metalogic data and implementation code are available at https://github.com/ tencent-ailab/metalogic .