Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information (2024)

Abstract: Chain-of-Thought (CoT) has become a vital technique for enhancing the performance of Large Language Models (LLMs), attracting increasing attention from researchers. One stream of approaches focuses on the iterative enhancement of LLMs by continuously verifying and refining their reasoning outputs for desired quality. Despite its impressive results, this paradigm faces two critical issues: (1) Simple verification methods: The current paradigm relies solely on a single verification method. (2) Wrong Information Ignorance: Traditional paradigms directly ignore wrong information during reasoning and refine the logic paths from scratch each time. To address these challenges, we propose Wrong-of-Thought (WoT), which includes two core modules: (1) Multi-Perspective Verification: A multi-perspective verification method for accurately refining the reasoning process and result, and (2) Wrong Information Utilization: Utilizing wrong information to alert LLMs and reduce the probability of LLMs making same mistakes. Experiments on 8 popular datasets and 5 LLMs demonstrate that WoT surpasses all previous baselines. In addition, WoT exhibits powerful capabilities in difficult computation tasks.

## introduction failure is the mother of success. ## preliminary this section introduces the framework that mainstream integrated multiple reasoning thoughts, iteratively enhancing llms by continuously reverifying and refining corresponding reasoning. xot @xcite , as shown in figure 2 , is an integrated reasoning framework that combines three reasoning modes: program-of-thought (pot) @xcite , equation-of-thought (eot) @xcite , and chain-of-thought (cot) @xcite . pot enables llms to generate python code and then uses the external python executor to run the results. eot involves llms generating mathematical equations, which are then solved using an external calculator. cot is a technique that guides llms to reason step-by-step. result verification verifier assertion verification verifier external executor eot eot solver planner wrong reasoning: @xmath0... pot solver reasoning process @xmath1.1) planning and solving ( §3.2) multi-perspective verification 2 answer external executor ( §3.3) wrong information utilization reasoning process plan fred has 5 cards. lisa bought 3 of fred's cards. how many cards does fred have now? question wrong information delivery @xmath2. please verify that the variables in this current process correspond one-toone with the information in the question. please answer this question from the beginning and verify the correctness of current result. the wrong information has been provided, avoid the same mistake! in the xot framework, the first step involves initiating the reasoning plan and selecting a reasoning method from either eot or pot to perform the reasoning. once the reasoning process is completed, the result is computed through an external executor. the answer is then verified using an assertion verification. if the reasoning result is determined to be correct, the answer is returned. if the initial reasoning is deemed incorrect and abandoned, the second step is to switch to an alternative reasoning mode and restart the process. after obtaining and verifying the new reasoning answer, if it is still incorrect, the third step is to directly use cot reasoning as the final answer. ## related work the rapid advancement of llms in recent years has introduced new opportunities in natural language processing @xcite @xcite . particularly, the introduction of chain-of-thought (cot) @xcite opens a novel direction in this domain, attracting many researchers @xcite @xcite @xcite . specifically, @xcite propose using manually constructed cot demonstrations to enhance llms performance. additionally, chen et al. (2022) introduce the programof-thoughts (pot), enabling llms to generate python programs to solve mathematical problems. liu et al. (2023b) propose equation-of-thoughts (eot), allowing llms to generate mathematical equations and then use external calculators to compute the results, offering a new perspective on problem-solving with llms. chen et al. (2024b) propose a framework that iteratively explores and self-evaluates trees of thoughts, allowing llms to learn from trial and error and improve the quality of final answers. xu et al. (2024b) propose transitioning llms from passive to active learning, thus enhancing their problem-solving capabilities. zhou et al. (2024) present a method for llms to improve self-criticism and self-discovery, thereby forming explicit structures to enhance reasoning performance. chen et al. (2023) propose using error code to implement code self-debug and improve the code generation capability of llms. in the realm of nonlinear problem solving, yao et al. (2023) introduce the tree-of-thoughts (tot) framework, enabling llms to generate multiple reasoning paths to tackle mathematical reasoning tasks. sel et al. (2023) propose the algorithm-of-thoughts (aot), which not only generates multiple paths but also selects the optimal nodes, allowing for the repeated utilization of reasoning pathways. besta et al. (2024) introduce graphof-thoughts (got), a framework that models the information generated by llms as arbitrary graphs, enabling the synergistic integration of all reasoning processes. ning et al. (2024) propose skeleton-of-thought (sot), which first generates the skeleton of the answer and then utilizes llms for batched resolution, enhancing inference efficiency. liu et al. (2023b) propose xot, which integrates multiple reasoning thoughts and utilizes single assertion verification to decide whether to switch reasoning methods, achieving impressive results. compared to previous research, wot employs multiple perspectives of verification while incorporating wrong information utilization. this greatly and effectively enhances overall reasoning performance. to our knowledge, this work is the first to incorporate multi-perspective verification and wrong information utilization within the continuously verifying and iterative framework. ## conclusion in this work, we propose wot, a framework that optimizes outputs by utilizing wrong information and multi-perspective verification. wot comprises two core modules: multi-perspective verification and wrong information utilization. wot achieves more accurate reasoning thought switching and utilizes wrong reasoning information. extensive evaluations on eight datasets and five models demonstrate that wot achieves superior performance. furthermore, wot exhibits powerful capabilities in difficult computation tasks.