Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning (2022)

Abstract: Our goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning. Such a capability would allow better understanding of why a model produced the answer it did. Our approach is to recursively combine a trained backward-chainingmodel, capable of generating a set of premises entailing an answer hypothesis, with a verifier that checks that the model itself believes those premises (and the entailment itself) through self-querying. To our knowledge, this is the first system to generate multistep chains that are both faithful (the answer follows from the reasoning) and truthful (the chain reflects the system’s own internal beliefs). In evaluation using two different datasets, users judge that a majority (70%+) of generated chains clearly show how an answer follows from a set of facts - substantially better than a high-performance baseline - while preserving answer accuracy. By materializing model beliefs that systematically support an answer, new opportunities arise for understanding the model’s system of belief, and diagnosing and correcting its misunderstandings when an answer is wrong.

## introduction although pretrained language models (ptlms) have shown remarkable question-answering (qa) performance, it is often unclear why their answers follow from what they know. while there has been substantial work on training models to also generate explanations for their answers @xcite , or produce them via few-shot prompting, e.g., "chains of thought" @xcite , those explanations may not be faithful (the answer does not necessarily follow from them) and may not be truthful, in the sense that the language model itself does not believe foot_0 the explanation state-figure 1 : given a question, entailer searches for an answer hypothesis that is supported by an entailment proof. first it over-generates candidate proofs, then it removes those that the model itself does not "believe" (i.e., confirms via self-querying that it considers all the generated proof elements to be true). finally it selects the best verified proof. multistep proofs are generated by iteratively backward chaining on the premises (section 3.2). our approach uses a combination of generation and verification, implemented in a system called entailer 2 . chains are constructed by backward chaining from candidate answers, recursively using a language model (lm) trained for a single backward-chaining step. for each step, entailer over-generates candidate entailments, then filters q: a magnet will stick to (a) a belt buckle (b) a wooden table (c) a plastic cup (d) a paper plate a: (a) a magnet will stick to a belt buckle because a belt buckle is sometimes magnetic. because metal is sometimes magnetic. a belt buckle is made of metal. a magnet will stick to magnetic metals. q: you can make a telescope with a (a) straw (b) glass (c) candle (d) mailing tube a: (d) you can make a telescope with a mailing tube. because a telescope is made of a tube for observing / seeing. a mailing tube is a kind of tube. q: quartz may produce rainbows when light is shined (a) around the crystal's area (b) through any of its sides (c) in the room its in (d) in to a mirror at it a: (b) quartz may produce rainbows when light is shined through any of its sides. because a rainbow is produced when light shines through a prism. because the rainbow is made of all different colors in visible light. a prism can split light into different colors. a quartz is a kind of prism. out those that do not conform to its own internal knowledge ("beliefs") by self-querying, asking itself whether (a) the generated premises (leaves of the proof step) are true, and (b) each entailment step is valid (figure 1 ). it then recursively backwardchains on premises until the overall proof confidence cannot be further improved (or a depth limit d is reached). finally, the candidate answer supported by the highest-scoring chain of a reasoning is returned. as a result, the system has materialized some of its latent knowledge from which the selected answer follows. most significantly, the resulting proof is thus both faithful (the answer follows from the proof) and truthful (the proof reflects the system's beliefs), providing a previously unavailable window into the model's beliefs about the world and their implications, e.g., figure 2 . to train the entailer model, we use a combination of the existing entailmentbank dataset @xcite , plus a new crowd-annotated dataset that we construct by bootstrapping (train an initial model, generate candidate entailment examples with it, then annotate those examples as extra training data). the model is then frozen, and entailer is then applied zero-shot to new datasets, i.e., entailer is a treated as a general-purpose, fixed model specialized for reasoning, rather than requiring finetuning for new tasks. we evaluate entailer on two existing datasets, obqa @xcite and quartz @xcite . we find that its reasoningbased qa accuracy is similar to its direct qa accuracy, with the advantage that a supporting chain of reasoning is also produced. we also perform a human evaluation, and find that 70% of time users judge the chains to clearly show how an answer followed from their premises, substantially higher than for explanations produced by a comparable high-performance qa system, macaw @xcite . our contributions are thus: recent work has shown that generating a freeform explanation ("chain of thought") before an answer can also improve performance on a variety of tasks @xcite @xcite . in these works, however, the explanations are unstructured, and there are no claims of faithfulness that the answer follows from the generation, nor that the explanations themselves represent model beliefs. ## approach like several previous systems (section 2), entailer treats reasoning as natural language inference (nli). in nli, the basic unit of knowledge is (represented as) a sentence rather than a structure, and a proof foot_3 is a tree of multi-step, multi-premise entailments, e.g., figures 2 and 3 . within this framework, given a question, entailer first generates candidate answers, then tries to prove each one, selecting the answer with the highest-scoring proof. we now describe these steps. ## model training the core of entailer is the model for one-step inference (section 3.2.1), with the three functionalities listed in ## evaluation our goal is to generate answers supported by faithful, truthful chains of reasoning, without significantly impacting performance. our two corresponding reserach questions are: for the second question, we collect human judgements of whether the hypothesis clearly follows from facts in the proof, and compare its proofs with explanations generated by a high-quality baseline model, macaw @xcite . ## towards teachable systems if a system can show how its answers systematically follows from its own beliefs, articulating those beliefs in the process, then this opens a window into the model's internal state for the user. as a result, new opportunities arise for identifying and correcting the model's misunderstandings, so the system gradually improves over time (sometimes referred to as explanatory interactive machine learning (xil) @xcite ). one vehicle for doing this is to use entailer's (currently unused) context field c at runtime: if the user asks a question, receives a wrong answer, and sees an incorrect belief in the proof, they would provide the corrected belief, then re-ask the question with the corrected belief in the context. this encourages the model to use the corrected belief in its new answer and proof, rather than just repeat the same bad belief. such overrides would then be stored and retrieved from a persistent memory to use for future questions also. a simple, hypothetical dialog illustrating this is shown in figure 8 . this is an exciting avenue made possible by this work, currently used by the teachme system @xcite . ## conclusion our goal is a system that can show how its answers systematically follow from its own internal beliefs, and materialize those beliefs in the process. entailer is, to our knowledge, the first sys-u: can a magnet attract a penny? entailer tries to prove h (a magnet attracts a penny) and the negation neg(h), and reports the best proof e: a magnet can attract a penny because: here entailer gives the wrong answer, arising from an incorrect belief #2. the user offers a correction: u: no. copper is not magnetic system tries again, trying to prove h and neg(h) with context @xmath0. c biases the model away from its prior mistake, effectively changing the model's belief. the new best proof is reported. e: a magnet cannot attract a penny because: the system now gets the right answer, having learned that copper is not magnetic from the user. c is added to memory for use in future questions, via information retrieva. u: yes figure 8 : a hypothetical dialog illustrating how a user (u) might identify and correct entailer's (e) incorrect beliefs through interaction. here entailer initially gets the wrong answer due to an incorrect model belief ("copper is magnetic"). the user offers a correction, which is then provided as context when re-asking the question, effectively overriding the prior bad belief (the user has "taught" the model). by storing such corrections in a memory, such belief updates persist over time.