Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework (2023)

Abstract: As large language models (LLMs) have become the norm in NLP, demonstrating good performance in generation and reasoning tasks, one of its most fatal disadvantages is the lack of factual correctness. Generating unfactual texts not only leads to lower performances but also degrades the trust and validity of their applications. Chain-of-Thought (CoT) prompting improves trust and model performance on complex reasoning tasks by generating interpretable reasoning chains, but still suffers from factuality concerns in knowledge-intensive tasks. In this paper, we propose the Verify-and-Edit framework for CoT prompting, which seeks to increase prediction factuality by post-editing reasoning chains according to external knowledge. Building on top of GPT-3, our framework lead to accuracy improvements in multiple open-domain question-answering tasks.

## introduction large language models (llms) have become the new norm in many downstream nlp tasks. in utilizing these llms, chain-of-thought (cot) prompting @xcite is found to improve performances for tasks that require complex reasoning, such as math word problems, commonsense reasoning, and symbolic manipulation. at the same time, it is able to generate interpretable reasoning chains. recent work further explored how to use these reasoning chains to select better predictions. however, the primary focus of these methods has been to improve end-task performance by utilizing generated cots as-is. for example, @xcite figure 1 : the verify-and-edit framework consists of five steps: (1) pass predictions with lower-than-average consistency to the next stages while leaving highly consistent predictions as-is; (2) produce verifying questions; in fact, improving the cot quality could be beneficial in enhancing both interpretability and endtask performance. @xcite point out that explanations judged as good by humans often indicate more accurate predictions. intuitively, a better set of cot prompts could provide better grounding and logically consistent thought processes, thus leading to more accurate predictions. to improve generation quality, one important aspect is factual correctness, which is currently one of the most fatal drawbacks of llms @xcite . in answering user queries, llms such as gpt-3 @xcite tend to make up facts and details, which is now flagged as a primary warning in their api usage. as a major use case of llms is the prospect of replacing traditional search engines and usage for more direct information access through questionanswering, factuality concerns could largely undermine their validity and degrade users' level of trust @xcite . fixing this issue is challenging and the concerns still persist even after the models are instruction-tuned with human feedback @xcite . this is because the source of truth can be unavailable during the finetuning process @xcite . thus, it is of urgent concern to better control the generation and increase the factual correctness of predictions. as llms could fail to recall accurate details when functioning as a knowledge base @xcite , if possible, knowledge from external sources could be introduced as assistance. assisted thought process is also common in human reasoning: when humans answer questions, they often search (or revisit) external knowledge sources for supporting facts in order to refresh their (internal) memory. inspired by this, in this work we propose a verify-and-edit (ve) framework to post-edit the reasoning chains for more factually aligned predictions. as shown in fig. 1 , we first select uncertain instances to edit, which have a less-than-majorityagree consistency. these instances, as implied by @xcite , often consist of plausiblesounding statements, such as the sentence "john nyskohus played for the norweigian football team odd greenland" in fig. 1 . when editing, we first generate a question to verify this detail, such as "what team did john nyskohus play for?" then, to answer this query, we introduce external knowledge through open-domain retrieval systems. for example, the fact "john nyskohus ... played for adelaide city.." is retrieved in this instance. then, the rationales are edited by providing the retrieved facts in the prompts as memory refreshments. thus, the edited rationales could be updated corresponding to the retrieved facts (fig. 1 ). given the edited rationales, the new prediction is generated, which considers more factually aligned reasoning traces. to our knowledge, our work is the first to postedit cot-style reasoning chains to enhance predic-tion performance. we perform experiments on two open-domain question answering (qa) tasks that require reasoning: adversarial hotpotqa @xcite and 2wikimultihop @xcite . we also test its performance on the fact verification task using fever @xcite . we find that the model is able to benefit from more factual reasoning chains, thus generating more accurate predictions. for example, for open-domain qa, our model demonstrates 3.8x accuracy improvement compared to similar retrieval-augmented models on advhotpot. on 2wikimultihop, verify-and-edit reaches 33.6% accuracy with open-domain search, while cot self-consistency stands at 27.7%. ## related work chain-of-thought or cot @xcite is a prompting method for improving the reasoning abilities of llms, which enables llms to decompose complex problems into multiple intermediate steps. cot provides interpretability and has been proven to be more capable of solving complex problems than standard prompting methods. however, hallucination is a long-standing problem in nlp, especially for llms, which has drawn significant attention from the research communities. the decoding process of llms is auto-regressive, which unavoidably makes it output nonfactual content without controlled generation @xcite . as such, the lack of supporting facts during the generation process of cot could largely undermine the validity of the final answer @xcite . @xcite demonstrate that the accuracy of the final answers largely correlates with the factuality and consistency of the reasoning explanations. the commonly proposed methods to improve the factuality of cot reasoning process can be grouped into two categories: prompt engineering and result calibration. prompt engineering methods are usually applied to guide llms to generate better intermediate reasoning explanations. react @xcite , which is the most comparable to our work, synergizes reasoning and acting in llms, where reasoning steps help the model induce and update actions, while action steps allow the model to consult additional information from wikipedia for a factuality check. compared to react, we generate more natural and conversational cots for better interpretability and easier learning. as such, our framework requires a much shorter prompt to learn. @xcite propose self-ask by instructing the llm to explicitly ask itself (and then answer) follow-up questions before answering the initial question. one natural way of solving a complex problem is to decompose the problem into subproblems and solve them sequentially. @xcite adopt the idea and propose least-to-most prompting. however, both self-ask and least-tomost prompting still rely on repetitively retrieving internal knowledge learned by the llm instead of connecting to external knowledge. thus, their ability to improve factuality is limited. result calibration functions on the output of the llms. @xcite train a calibrator to calibrate the weights of the final answers based on the factuality and consistency of the generated explanations, which efficiently improves the results. the decoding method in cot is naive greedy, which simply outputs the next token with the highest probability. @xcite propose a selfconsistency decoding method, which samples a diverse set of reasoning paths and then selects the most consistent answer by marginalizing out the sampled reasoning paths. selection-inference (si) @xcite framework is another stateof-the-art method that exploits llms as general processing modules. out of all the methods, it is also the first to systematically improve the factual correctness of cots in order to predict more accurately. it alternates between selection and inference to generate a series of interpretable, causal reasoning steps leading to the final answer, which is proven to be efficient. however, it is not designed for open-domain or commonsense question answering. moreover, another comparable line of work has been exploring retrieval-augmented language model pretraining (realm) @xcite , which first retrieves documents from an external knowledge source and then utilizes retrieved documents to process question-answering tasks. @xcite propose to include google search results of the question in the prompt to improve the factuality of the generated answer. however, such methods may fail in complex questions as it does not utilize the reasoning capability of llms. thus, we consider retrieval-augmented reasoning paths as a natural way to increase factual alignment. ## verify-and-edit framework our goal is to make llms generate more factual reasoning chains with cot prompting assisted with external knowledge, thereby also improving prediction accuracy of the final answer. we hypothesize that this can enhance llms' capability to solve complex knowledge-intensive tasks that require multiple reasoning steps to arrive at an answer. generally, we hope to follow the human reasoning process: when a person answers a question, if he/she is unsure, he/she would search for a supporting fact and consider it before giving the final answer. thus, we could separate the verify-and-edit (ve) framework into 3 different stages: finding uncertain predictions, editing their rationales by searching for supporting facts, and using the edited rationales to generate final answers (fig. 1 ). in designing the stages, we hope to maximally preserve the llms' biggest advantage: their opengeneration and reasoning ability. and we aim to design tasks and setups as natural and conversational as possible, thus making it easy to understand for humans and llms which are trained with natural texts. ## conclusions in this paper, we introduce a verify-and-edit framework for open-domain question-answering. it is a first attempt to post-edit cot-style reasoning chains for better end-task performance. by combining knowledge retrieval with reasoning, the framework edits cots in a natural and conversational way, which enhances prediction factuality. combined with google search, the framework also shows a promising direction that combines the open-generation ability of state-of-art llms with the updated facts provided by search engines.