MATHSENSEI : A Tool-Augmented Large Language Model for Mathematical Reasoning (2024)

Abstract: Tool-augmented Large Language Models (TALMs) are known to enhance the skillset of large language models (LLMs), thereby, leading to their improved reasoning abilities across many tasks. While, TALMs have been successfully employed in different question-answering benchmarks, their efficacy on complex mathematical reasoning benchmarks, and the potential complementary benefits offered by tools for knowledge retrieval and mathematical equation solving are open research questions. In this work, we present MathSensei, a tool-augmented large language model for mathematical reasoning. We study the complementary benefits of the tools - knowledge retriever (Bing Web Search), program generator + executor (Python), and symbolic equation solver (Wolfram-Alpha API) through evaluations on mathematical reasoning datasets. We perform exhaustive ablations on MATH, a popular dataset for evaluating mathematical reasoning on diverse mathematical disciplines. We also conduct experiments involving well-known tool planners to study the impact of tool sequencing on the model performance. MathSensei achieves 13.5% better accuracy over gpt-3.5-turbo with Chain-of-Thought on the MATH dataset. We further observe that TALMs are not as effective for simpler math word problems (in GSM-8K), and the benefit increases as the complexity and required knowledge increases (progressively over AQuA, MMLU-Math, and higher level complex questions in MATH). The code and data are available at https://github.com/Debrup-61/MathSensei.

## introduction state-of-the-art large language models (llms), including gpt-3.5-turbo, gpt-4, and open-source counterparts, such as llama 2 have demonstrated impressive performance across a broad spectrum of nlp tasks @xcite @xcite . however, their consistent failure on established reasoning dimensions, such as mathematical, commonsense, abductive, and multi-hop reasoning @xcite @xcite have led the research community to explore various solutions for enhancing their reasoning abilities. this pursuit has given rise to techniques, such as -(1) intelligent prompting variations, such as chain of thought @xcite , program of thought @xcite , tree of thoughts @xcite , and self-refinement @xcite , (2) programguided solving that generates python code as intermediate steps and offloads execution to a symbolic interpreter @xcite , (3) multi-model interaction frameworks, such as multi-agent debate @xcite and round-table conference @xcite , 4) tool-augmented llms powered by external symbolic tools, apis, and libraries @xcite @xcite @xcite . in this work, we study the effectiveness of toolaugmented llms (talm) applied to problems involving mathematical reasoning. recent advancements in talm frameworks, such as chameleon @xcite , olagpt @xcite , art @xcite , and socraticai @xcite have explored the effectiveness of incorporating external tools for solving knowledge-intensive reasoning tasks and fundamental mathematical problems (such as, arithmetic and algebra). however, the effectiveness of talm framework is yet to be validated on mathematical reasoning tasks involving complex computations. in this context, it is imperative to assess the suitability of specific tool combinations across diverse mathematical domains (e.g., prealgebra, calculus, geometry, intermediate algebra, probability) at varying levels of difficulty. this motivated us to undertake a thorough evaluation of talm framework in the context of complex mathematical reasoning tasks. we propose and develop mathsensei, a talm-based framework, comprising a distinct set of tools (also referred to as modules), combined in a sequential fashion. these modules include llm-based components, such as -knowledge retriever (kr ). python code generator (pg ), code refiner (cr ), , and solution generator (sg ); and apis, such as -bing-web-search-api (bs ) and wolframalpha-api (wa ). as illustrated in fig. 1 , mathsensei adopts the modular architecture -from the wolfram alpha response, we find that the solutions are @xmath0 and @xmath1. question : find the positive difference between the solutions for @xmath2. figure 1 : an end-to-end workflow of mathsensei on the compositional setting from the math dataset. the final answer is higlighted in green font. our extensive ablations (varying the set and order of modules), show that complex mathematical problems, spanning different subdomains can be benefited by specific types, combinations, and order of the modules. this further highlights the need for planning strategies. we evaluate two advanced planning techniques within our pipeline, investigating methodologies such as plan-and-solve @xcite and react @xcite with mathsensei. we make the following contributions: 1. we comprehensively evaluate the effectiveness of talm frameworks across multiple mathematical datasets, such as gsm-8k, aqua-rat, math, mmlu-math, encompassing diverse mathematical problem types and tasks. compared to math, mmlu-math, our experiments on simpler mathematical datasets (e.g., gsm-8k, aqua-rat) reveal minimal benefit of using multiple modules on top of cot prompting. 2. through systematic ablations by varying the set and order of modules in our framework, we observe that complex mathematical problems spanning different domains (such as, algebra, calculus, number theory, and probability from the math dataset) can be benefited by certain types, combinations, and order of these modules. we observe that the bs module outperforms the kr module for retrieving relevant knowledge for mathematical problems. the setting of wa+bs+sg outperforms pg+sg, demonstrating that program-guided solving techniques @xcite may not be universally suitable for all mathematical problems. these findings motivate the necessity of exploiting better planning techniques. our best configuration of mathsensei, pg+wa+sg achieves an impressive performance accuracy of 47.6 % on the math dataset, surpassing gpt-3.5-turbo( ) with chain-of-thought (cot) prompting by 13.5% @xcite . the same setting shows a performance gain of +11.6% over gpt-4 (with cot prompting) on intermediate algebra problems. for precalculus, gpt-4 (with cot prompting) has an accuracy of 26.7%, which gets improved to 28.9% by our wa+pg+sg setting. improvements on aqua-rat and mmlu-math are lower, 2.4% and 3.3% respectively, showing the efficacy decreases as requirement of external knowledge decreases. 3. we quantify the performance of state-of-the-art planning techniques, such as plan-and-solve and re-act coupled with tool-augmented llms on the math dataset. however, we do not observe benefit of using the planners over our best configurations of pg+wa+sg, which may indicate a need for developing targeted planning strategies for mathematical talms. we include our planning related experiments in the appendix. in our tool-augmented framework (mathsensei), we incorporate several such techniques. we adopt cot prompting for the text generation modules, and use the methodology by @xcite to generate python code (using libraries like sympy) based on the current context and mathematical question; followed by execution of the code using python interpreter. while @xcite focuses on elementary level mwp (math word problems) and simple arithmetic datasets such as asdiv @xcite and singleeq @xcite , we explore complex mathematical datasets spanning diverse math problem types (math, aqua @xcite , mmlu-math). following self-refine, we employ a code refinement module to iteratively rectify syntactical errors in the original generated code, using error messages from the interpreter. tool-augmented llms. the emerging trend of toolaugmented llms has garnered increasing attention within the research community. large language models, trained on the objective of next-token prediction, excel at generating tokens based on probabilistic patterns in their training data, making them effective in dataintensive tasks. however, their proficiency falls short in capturing nuanced reasoning or token relationships, particularly in mathematical domains. consequently, there are instances or specific question types where it would be advantageous for an llm to leverage support from specialized tools or modules. for instance, consider a question requiring the solution to the roots of a 4th-degree polynomial. the llm, upon generating a special token followed by a query, can pause its generation and invoke a mathematics computing platform wol-framalpha. wolframalpha, in turn, can utilize its api to process the query and return the answer to the llm, which can then continue its generation. toolformer @xcite leverages data annotated with such tool calls (using special tokens for tools) and responses to train language models to employ tools as needed in a self-supervised manner. similarly, the tool-augmented llm framework chameleon @xcite ) adopts a plug-and-play approach to utilize tools sequentially. in their setup, the sequence of execution of the tools is predetermined based on a target task; the output of each tool is added to the context for subsequent downstream tools in the pipeline. they perform evaluation on multi-modal knowledge-intensive datasets, such as scienceqa and tabmwp. similarly, frameworks such as art @xcite engage in multi-step reasoning, where each step is linked to a tool call. utilizing search and code tools, art tackles various tasks across datasets such as mmlu @xcite and bigbench @xcite . our work adopts the generic backbone of popular tool-augmented llm frameworks such as toolformer and chameleon. in comparison to the previous work, we distinguish ourselves by conducting a comprehensive analysis and comparison specific to tools useful for addressing diverse mathematical problems. notably, chameleon lacks evaluation on mathematical datasets, and art focuses exclusively on algebra, leading to gaps in the assessment of tool-augmented llms. furthermore, our study incorporates a comparison of planning techniques within tool-augmented llm frameworks for mathematical reasoning, an aspect not adequately addressed in the current literature. to the best of our knowledge, planning techniques like react @xcite have primarily been tested on knowledgeintensive reasoning datasets such as fever @xcite and hotpotqa @xcite . ## methodology we first discuss some notations to formalize the problem. let m denote the set of modules foot_0 (each performing a specific task), p i be the input prompt for module m i , and q be the set of mathematical queries. ## experimental setup we first introduce the mathematical datasets used in our study ( §4.1), followed by the experiments that we perform with various combinations of modules ( §4.2). we use gpt-3.5-turbo as the default llm in llmbased modules unless mentioned otherwise. this is mainly because it is more accessible and cheaper compared to gpt-4. for querying a search-engine, we use bing-web-search-api. please refer to 23 in appendix for details about online resources that we use. ## conclusion we introduce a tool-augmented large language model (talm) framework, aka mathsensei, targeted for mathematical reasoning. we utilize tools for webbased knowledge retrieval, program generation and execution and symbolic equation solving. we perform extensive ablations over the individual tools, along with varying the order and combination on complex mathematical reasoning datasets (such as math). our best configuration achieves a 13.5% improvement over gpt-3.5-turbo (with cot prompting) on math. our experiments with tool-sequencing methods does not improve over our best configuration. we also observe that benefit of mathematical talms are minimal for simpler math word problems (in gsm-8k) and its benefit increases as the required complexity and knowledge for the problem increases through aqua, mmlu-math.