Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022)

Abstract: Pre-trained language models (LMs) struggle with consistent reasoning; recently, prompting LMs to generate explanations that self-guide the inference has emerged as a promising direction to amend this. However, these approaches are fundamentally bounded by the correctness of explanations, which themselves are often noisy and inconsistent. In this work, we develop Maieutic Prompting, which aims to infer a correct answer to a question even from the unreliable generations of LM. Maieutic Prompting induces a tree of explanations abductively (e.g. X is true, because ...) and recursively, then frames the inference as a satisfiability problem over these explanations and their logical relations. We test Maieutic Prompting for true/false QA on three challenging benchmarks that require complex commonsense reasoning. Maieutic Prompting achieves up to 20% better accuracy than state-of-the-art prompting methods, and as a fully unsupervised approach, performs competitively with supervised models. We also show that Maieutic Prompting improves robustness in inference while providing interpretable rationales.

## introduction following the remarkable success of few-shot prompting over large language models (e.g. @xcite , recent studies on prompting methods suggest that lms' reasoning capability can be further promoted by generating a sequence of explanation for a given problem, prior to inferring the answer @xcite @xcite . the so-called explanation-based prompting helps an lm better elicit its knowledge and reason by leveraging its own generated explanations -whether it be commonsense knowledge @xcite , a solution for a math word prob1 we share our code at https://github.com/ jaehunjung1/maieutic-prompting . therefore, the answer is false. ## problem setup and background our goal is to infer whether a given statement q makes sense, i.e. inferring the truth value a of q. conventionally, this can be done through prompting an lm with the following two methods: standard prompting let q be a statement we want to infer the truth value of (i.e., either true or false). in standard few-shot prompting, the modelinferred answer â is defined as: explanation-based prompting in explanationbased prompting, the inference process is factorized into two steps: here, e denotes the explanation generated prior to inferring the answer label, and @xmath0, e 1 , a 1 ), • • • , (q k , e k , a k )} includes k examples of questions, explanations and answers. since marginalizing over all e is intractable, prior works resort to a sampling based approximation: in this section, we introduce maieutic prompt-ing, which performs inference over a maieutic tree of generated explanations. first, we introduce logical integrity, a key concept that is used to determine the reliability of propositions. language models often generate logically inconsistent propositions; for instance, in figure 1 , the model infers true when prompted with either "one is a number that comes before zero." or "one is a number that comes after zero.". in this sense, p(true|q) does not provide a reliable value to determine whether q is true or not. we formalize this idea as logical integrity: a proposition q is logically integral when the lm consistently infers the truth value of q and ¬q (i.e. q as true and ¬q as false, or vice versa). formally, we define a boolean function integral(e) as follows: a statement is considered to be logically integral / true when condition 1 is met, and logically integral / false when condition 2 is met. intuitively, the truth values of logically integral propositions are more credible than non-integral ones, to which lms are inconsistent given a simple negation. for example, "one is a number that comes before zero." in figure 1 would not be logically integral, as the model assigns same truth value to both q and ¬q. for the rest of section, we first search for logically integral propositions by constructing the maieutic tree (section 3.1), then quantify the relations between the propositions (section 3.2), based on which we infer the final answer (section 3.3). ## experiments datasets we evaluate maieutic prompting on three commonsense reasoning and fact verification benchmarks in binary qa format: com2sense @xcite , csqa 2.0 @xcite , creak @xcite . despite the simple format, these datasets require a substantial amount of knowledge and robust reasoning, making them challenging even for the billion-scale fine-tuned lms ( baselines we compare our method with both the few-shot prompting methods and supervised models. along with the standard prompting, we include chain of thought @xcite , self-consistency @xcite and generated knowledge prompting (gkp) @xcite . for supervised models, we consider the strong baselines used for the respective dataset, such as t5 @xcite , unifiedqa @xcite and unicorn @xcite . configuration details for all prompting methods, we use the same set of 6 demonstration examples and the same version of gpt-3 (text-davinci-001) as the lm. we determine the hyperparameters of maieutic prompting and baselines based on the dev set performance on the benchmarks. in maieutic tree generation, we set the maximum depth to 2. for depth 1, we use nucleus sampling (@xmath1.0) @xcite ## related work prior works have leveraged natural language explanations (nles) to promote model reasoning, either by training a model to explain @xcite @xcite , or generating answers to templated queries and distantly supervised rationales @xcite . incorporated with in-context learning @xcite ; inter alia), these efforts have led to explanation-based prompting @xcite @xcite . other works aim to improve model interpretability with nles, training a model that explains its inference post-hoc or in parallel with the answer @xcite @xcite . unlike these works, the explanations in our work are designed to be intrinsic @xcite ; the explanations themselves explicitly participate in the inference. meanwhile, recent observations reveal that lm explanations are unreliable, as they often lack logical consistency and are not factually grounded @xcite . this is in part due to the broader limitations of generative lms, which assign high probability to unlikely sentences @xcite and are sensitive to semantic perturbations @xcite . maieutic prompting overcomes these limitations by avoiding the use of explanations "as-is", and modeling the relationships between explanations. another line of works apply symbolic methods on top of lms to improve their consistency, spanning from a lexical constraint on sequence decoding @xcite to a symbolic world model @xcite and discrete operations @xcite . other works explore how to train a model that simulates the symbolic reasoning process, such as logical transformation @xcite and consistent generation of beliefs @xcite . however, these models require a curated set of human annotations that limits their application to specific domains. maieutic prompting generalizes these neuro-symbolic approaches in an unsupervised setup, employing max-sat algorithm to symbolically determine the true subset from a noisy pool of neural generations. ## conclusion in this work, we propose maieutic prompting, a novel few-shot inference method inspired by the socratic way of conversation. we systematically generate a tree of explanations that bear logical relations between each other, then find the truth values that max-satisfy these relations. empirical results show that maieutic prompting is both competitive and robust compared to diverse baselines, while providing intrinsic interpretations over its inference.