C hat C o T : Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models (2023)

Abstract: Although large language models (LLMs) have achieved excellent performance in a variety of evaluation benchmarks, they still struggle in complex reasoning tasks which require specific knowledge and multi-hop reasoning. To improve the reasoning abilities, we propose \textbf{ChatCoT} , a tool-augmented chain-of-thought reasoning framework for chat-based LLMs ( \textit{e.g.,} ChatGPT). In ChatCoT, we model the chain-of-thought (CoT) reasoning as multi-turn conversations, to utilize tools in a more natural way through chatting. At each turn, LLMs can either interact with tools or perform the reasoning. Our approach can effectively leverage the multi-turn conversation ability of chat-based LLMs, and integrate the thought chain following and tools manipulation in a unified way. Specially, we initialize the early turns of the conversation by the knowledge about tools, tasks, and reasoning format, and propose an iterative \textit{tool-augmented reasoning} step to perform step-by-step tool-augmented reasoning. The experiment results on two complex reasoning datasets (MATH and HotpotQA) have shown the effectiveness of ChatCoT on complex reasoning tasks, achieving a 7.9% relative improvement over the state-of-the-art baseline.

## introduction recently, large language models (llms) @xcite have shown great potential as generalpurpose task solvers in a variety of real-world applications. with excellent few-shot and zero-shot ability, llms, such as @xcite and llama @xcite , can even outperform full-data supervised-tuned models on many tasks with suitable prompting strategies. among these prompting strategies, chain-ofthought (cot) prompting @xcite has been a prominent approach to eliciting the reasoning abilities of llms. it incorporates the intermediate reasoning steps of exemplars into the input prompt, to instruct llms to solve a question step by step. despite the remarkable improvement by cot prompting, llms still have difficulties in solving complex reasoning tasks that involve specific functionalities, such as arithmetic calculation and information retrieval @xcite . to address this issue, external tools (e.g., calculator, search engine) have been employed to fulfill the basic functionalities @xcite , easing the burden of llms. with proper interfaces, llms can be guided by prompts to manipulate tools when necessary. however, as tools are not intrinsically integrated with llms, incorporating external tools would have to interrupt the cot reasoning process of llms. such an issue would become more intractable on complex reasoning tasks that frequently invoke the use of tools. to address it, existing work either relies on llms to prearrange the tool use plan for subsequent execution @xcite , or needs to design formal actions pertaining to specific tasks @xcite @xcite . despite the effectiveness, the two types of methods still suffer from potential issues: the former one cannot interact with tools after generating the plan, even seeing obvious mistakes; while the latter one has to frequently switch between reasoning with llms and taking actions, hurting the continuity of the cot reasoning process. to overcome these disadvantages, we seek a more unified way to integrate cot reasoning and tool manipulation. as the key idea, we consider tools manipulation by llms as the interaction between llms and tools, in which llms send the use requests and tools respond to support specific functions. further, inspired by the recent progress of chatgpt-like llms (called chat-based llms), we model the interaction process between llms and tools as a multi-turn conversation, and leverage the excellent chatting capacities for manipulating tools by llms. at each turn, the llm can freely interact with tools when in need, otherwise perform the reasoning by itself. the conversation continues until the final answer is derived by llms. in this process, as chat-based llms can well understand the multi-turn context, they can follow the thought chain in the whole conversation and naturally invoke the tools accordingly, thus keeping the continuity of the reasoning process. to this end, in this paper, we propose chatcot, a tool-augmented chain-of-thought reasoning strategy for chat-based llms. as the major merit, chat-cot can perform the cot reasoning across multiturn conversation, and freely interact with tools at immediate steps. concretely, we first store the useful knowledge at early turns of the conversation, including tools, tasks, and multi-turn reasoning format, to help llms utilize task-specific knowledge to perform reasoning or manipulate tools. then, we iterate a specially designed tool-augmented reasoning step in which llms interact with tools, to perform step-by-step tool-augmented reasoning, until obtaining the final answer. to evaluate the effectiveness, we implement chatcot on chatgpt, and conduct experiments on two complex reasoning benchmarks, i.e., math @xcite and hot-potqa @xcite . experimental results show that chatcot achieves very promising performance on math with 7.9% relative improvement in average over the sota baselines (i.e., php @xcite ). besides, our approach can also be integrated with other strategies, e.g., self-consistency, and chatcot can achieve better performance by incorporating these strategies. ## preliminary in this section, we present the task setting, then introduce the chain-of-though prompting strategy and the tool manipulation in reasoning tasks. task setting. in this work, we focus on improving the reasoning ability of large language models (llms) on complex tasks, e.g., solving mathematics competition problems. unlike tasks that can be solved by humans via straightforward skills or tools, complex tasks require advanced knowledge (e.g., mathematical theorem) and multi-step reasoning to reach the answer. typically, a complex problem includes three types of texts, namely problem statement, solution text, and answer key, denoted as 𝑄, 𝑆 and 𝐴, respectively. the problem statement 𝑄 introduces the background and description of a complex problem, and the solution text illustrates the detailed solving process to obtain the answer key. all of them are composed of a sequence of tokens, where each token is either a text word or a mathematical symbol. formally, given the problem statement 𝑄, we aim to utilize llms to perform multi-step reasoning, to finally generate its accurate answer 𝐴. ## approach in this section, we present our proposed chatcot, a new chain-of-thought (cot) prompting framework based on multi-turn conversations, for improving chat-based llms on complex reasoning tasks with tools. the overall illustration of our proposed chat-cot is shown in figure 1 . ## experiment in this section, we conduct experiments to evaluate the effectiveness of chatcot. the implementation details can be found in appendix b. ## conclusion in this paper, we have proposed chatcot, a new framework to manipulate the tools for the cot reasoning. it naturally integrates the reasoning process and manipulating tools through a form of multi-turn conversations. at each turn, llms can either interact with tools or perform the reasoning by itself. our approach can effectively leverage the multiturn conversation ability of chat-based llms. experimental results on two complex reasoning tasks including math and hotpotqa have verified the effectiveness of chatcot. currently, our experiments are mainly conducted on mathematical reasoning tasks, and we will test the effectiveness of the proposed approach to more types of reasoning tasks. besides, we will also consider extending the number of available tools for solving different tasks.