M ath P rompter: Mathematical Reasoning using Large Language Models (2023)

Abstract: Large Language Models (LLMs) have limited performance when solving arithmetic reasoning tasks and often provide incorrect answers. Unlike natural language understanding, math problems typically have a single correct answer, making the task of generating accurate solutions more challenging for LLMs. To the best of our knowledge, we are not aware of any LLMs that indicate their level of confidence in their responses which fuels a trust deficit in these models impeding their adoption. To address this deficiency, we propose ‘MathPrompter’, a technique that improves performance of LLMs on arithmetic problems along with increased reliance in the predictions. MathPrompter uses the Zero-shot chain-of-thought prompting technique to generate multiple algebraic expressions or python functions to solve the same math problem in different ways and thereby raise the confidence level in the output results. This is in contrast to other prompt based CoT methods, where there is no check on the validity of the intermediate steps followed. Our technique improves over state-of-the-art on the ‘MultiArith’ dataset (78.7% - 92.5%) evaluated using 175B parameter GPT-based LLM.

## introduction recent advancements in natural language processing (nlp) can be attributed to massive scaling of large language models (llms) @xcite @xcite @xcite @xcite . a very interesting recent discovery that the llms are naturally good (in-context) zero-shot or few-shot learners turned out to be very useful @xcite @xcite . this led to the development of 'prompting' technique, where the user provides a small context for solving the task at-hand to the llm. this conditioning of the models on a few examples is termed as few-shot prompting, while providing instructions to solve a task is known as zero-shot prompting. extensive research efforts are being poured into designing these prompts, either manually @xcite or automatically @xcite . although quite successful for single-step system-i tasks @xcite , the prompting techniques were inadequate in their performance on system-ii tasks where multi-step reasoning is required @xcite . as humans, we tend to break down a problem and attempt to solve them step-by-step. extending this intuition to llms led to the development of 'chain-of-thought' (cot) prompting technique @xcite . the use of cot has led to improved performance on a range of nlp tasks @xcite @xcite @xcite @xcite in this work, we investigate zero-shot-cot methods for solving mathematical reasoning tasks. to the best of our knowledge, we found the recent work by @xcite that proposed a zeroshot-cot technique to be the state-of-the-art where they demonstrated a remarkable accuracy improvement on the 'multiarith' @xcite data (17.7% → 78.7%). now, we identify two key aspects that lacks in the previous cot prompting based sota, namely (1) although, the chainof-thought followed by the model improved the results, but there is no check on the validity of the steps followed by the chain-of-thought prompting and (2) the confidence in the predictions of llms are often not provided. in order to address these gap to some extent, we derive inspiration from how we humans solve a math question by breaking it down to a simpler multi-step procedure and make use of multiple ways to validate our approach at each step. specifically, given a question q, (i) generating algebraic template: we first gen- python prompt write a python function that returns the answer. algebraic prompt write a mathematical equation and generate the answer format starting with `@xmath0: mathprompter flow. we outline the mathprompter process with an example alongside. erate its corresponding algebraic expression qt that replaces the numerical entries by variables. (ii) math-prompts: then, we provide multiple prompts p to the llm that can solve qt analytically in different ways. for eg. p can be 'derive an algebraic expression' or 'write a python function' etc. following this procedure, we end up with p expressions that analytically solves qt in terms of its variables. (iii) compute verification: we then evaluate the p analytical solutions by allotting multiple random values to the qt variables. (iv) statistical significance: if the solutions of the p analytical functions are in 'consensus' over n ∼ 5 different variable choices, then we substitute the original values from q to obtain the final solution. in the case where there is no definite consensus, we repeat the steps (ii), (iii) & (iv). our method, mathprompter, uses 175b parameter llm called gpt3 davinci completion engine (brown et al., 2020). we were able to improve the accuracy on the multiarith data from 78.7% → 92.5%. ## method since the llms are generative models, it becomes very tricky to ensure that the generated answers are accurate, especially for mathematical reasoning tasks. we take clues from the process followed by students to solve arithmetic problems. we narrowed down a few steps that students take in order to verify their solutions, namely ## conclusions & discussions we introduced mathprompter, a novel approach that improves llm performance on mathematical reasoning problems. it also addresses an important concern of building the user trust to some extent in the llm predictions. we translated our intuition on how students solve arithmetic problems to a llm model by utilizing the zero-shot chain-of-thought prompting technique. mathprompter incorporates ideas like cross-checking the intermediate steps and solving the same math problem using multiple approaches in its design. we empirically show that our model is comparable to sota few-shot-cot models as well as the larger zero-shot-cot models that have 540b parameters. in future, we plan to further evaluate performance on additional datasets and explore incorporating additional prompts into @xmath1:14, b:5, c:6} # algabraic answer @xmath2, b, c): return a*c -b @xmath3.