Plan, Verify and Switch: Integrated Reasoning with Diverse X -of-Thoughts (2023)

Abstract: As large language models (LLMs) have shown effectiveness with different prompting methods, such as Chain of Thought, Program of Thought, we find that these methods have formed a great complementarity to each other on math reasoning tasks. In this work, we propose XoT, an integrated problem solving framework by prompting LLMs with diverse reasoning thoughts. For each question, XoT always begins with selecting the most suitable method then executes each method iteratively. Within each iteration, XoT actively checks the validity of the generated answer and incorporates the feedback from external executors, allowing it to dynamically switch among different prompting methods. Through extensive experiments on 10 popular math reasoning datasets, we demonstrate the effectiveness of our proposed approach and thoroughly analyze the strengths of each module. Moreover, empirical results suggest that our framework is orthogonal to recent work that makes improvements on single reasoning methods and can further generalise to logical reasoning domain. By allowing method switching, XoT provides a fresh perspective on the collaborative integration of diverse reasoning thoughts in a unified framework.

## introduction the ai community has long sought to achieve automated reasoning @xcite , which is an important component of artificial general intelligence @xcite . mathematical reasoning, as a cognitive skill essential for humans yet challenging for language models, attracts increasing interests and commitment from researchers @xcite @xcite . with the abilities endowed by in-context learning (icl), large language models (llms) xot integrates a verification module that makes a difference in method planning, enabling the attempts of diverse reasoning thoughts within an iterative framework. @xcite @xcite are able to solve mathematical problems through textual rationales with chain-of-thought prompting @xcite (cot) or through python functions with program-aided language model @xcite and program-of-thought prompting @xcite (pal or pot). these prompting methods exhibit unique strengths and limitations. cot generates a step-by-step reasoning flow in natural language and performs calculations on the fly. this approach enables a more flexible solution format, but may result in a loss of precision since language models often struggle with arithmetic calculations @xcite . on the other hand, pot or pal resolves problems through python statements, relying on python interpreters to ensure calculation accuracy. another noteworthy and intriguing prompting method is to form math problems as linear equation systems @xcite . similarly, inspired by linear algebra, we propose equation-of-thought (eot), which performs math reasoning in a more direct way. the diversity inherent in each method does not render them as competing or mutually exclusive alternatives. on the contrary, in practical problem solving scenarios, possessing multiple methods can always yield a range of complementary advantages. the distinct problem-solving approaches can contribute to synergistic benefits that surpass the outcomes of any single approach. we find that this intuition also applies to the realm of math reasoning. with the availability of cot, pot and eot, we hold the hypothesis that a model has the potential to solve a problem if it reaches the correct answer using any one of the prompting methods. as illustrated in figure 2 , our analysis shows that the model exhibits the potential to solve 92.72% of the problems, surpassing the best performing single method by over 10%. motivated by this observation, we propose xot, an integrated math problem solving framework, which improves the llm's reasoning ability by switching among diverse reasoning thoughts. since there is no guarantee that llms can always solve the problem in a single attempt, we follow the human intuition and allow the model to rethink and switch to a different method when encountering difficulties or obstacles. we apply two complementary verification methods to facilitate the model to decide whether it is time to switch to another method: passive and active verification. passive verification relies on the external executors to provide determinable results based on the generated programs @xcite . it offers shallow inspections, such as program syntax issues or the runtime errors. for active verification, we ask the model to verify the solution by checking whether the answer adheres to the conditions outlined in the original question. as shown in figure 1 , xot consists of three modules that work in an iterative framework: planning, reasoning and verification. given a problem as input, the planning module first proposes the most appropriate method. the reasoning module then generates one solution using the planned prompting method. with the outputs and the results from external executors, the model is asked to assess the answers in the context of the questions. if the answer fails the verification, we will go back to the planning module for another round of iteration and attempt alternative methods. the iterative process concludes when the verification confirms the correctness of the answer or after exhausting all available methods. to demonstrate the effectiveness of xot, we conduct extensive experiments on 10 popular mathematical reasoning datasets and achieve consistent improvement. empirical results suggest that xot can accommodate recent work that focuses on improving single reasoning methods. additional experiments also indicate that xot can generalise to other domains such as logical reasoning tasks. we summarize the main contributions as follows. first, we propose an integrated problem solving framework xot, utilising the complementarity of different reasoning thoughts. second, we introduce eot which solves math problems with a system of linear equations, serving as a complementary method to existing approaches. third, we incorporate passive and active verification to facilitate the framework to switch among diverse reasoning thoughts, empowering the framework to make informed decisions regarding the subsequent steps to be taken. more generally, xot sheds lights on a new direction of interacting with diverse reasoning methods and tools. as shown in figure 1 , instead of sticking to one determined method, llms can benefit from the verification and the flexible switching among available reasoning thoughts. foot_0 2 related work ## xot our goal is to develop a generalized problem solving framework that can automatically select the appropriate method for different problems and has the capability to switch among reasoning thoughts using both active and passive verification. we first describe the overall framework and introduce each module in detail. ## analysis in this section, we first analyze the effectiveness and necessity of each module within xot. then we provide comparison with majority voting and describe how model's self refinement can be integrated in our framework. ## conclusion we propose xot, an integrated problem solving framework that utilizes diverse reasoning thoughts to prompt llms. xot integrates planning, reasoning and verification into a unified framework, enabling the model to explore multiple methods based on the active and passive verification of the solutions. we conduct extensive experiments on 10 math reasoning datasets to thoroughly evaluate the advantages of each module and showcase the efficacy of our proposed approach. further results also show that the design ethos of xot can generalize to logic reasoning domain. we consider its generalisation to more diverse tasks as a compelling avenue for future exploration.