Reasoning in Large Language Models Through Symbolic Math Word Problems (2023)

Abstract: Large language models (LLMs) have revolutionized NLP by solving downstream tasks with little to no labeled data. Despite their versatile abilities, the larger question of their ability to reason remains ill-understood. This paper addresses reasoning in math word problems (MWPs) by studying symbolic versions of the numeric problems, since a symbolic expression is a “concise explanation” of the numeric answer. We create and use a symbolic version of the SVAMP dataset and find that GPT-3’s davinci-002 model also has good zero-shot accuracy on symbolic MWPs. To evaluate the faithfulness of the model’s reasoning, we go beyond accuracy and additionally evaluate the alignment between the final answer and the outputted reasoning, which correspond to numeric and symbolic answers respectively for MWPs. We explore a self-prompting approach to encourage the symbolic reasoning to align with the numeric answer, thus equipping the LLM with the ability to provide a concise and verifiable reasoning and making it more interpretable. Surprisingly, self-prompting also improves the symbolic accuracy to be higher than both the numeric and symbolic accuracies, thus providing an ensembling effect. The SVAMP-Sym dataset will be released for future research on symbolic math problems.

## introduction large language models (llms), with hundreds of billions of parameters, can solve a wide range of nlp tasks such as machine translation, question-answering, etc., taking us closer to general-purpose intelligent agents. the initial success of gpt-3 @xcite has led to many other llms @xcite @xcite which have, perhaps surprisingly, taken big strides in solving difficult tasks like common sense reasoning, math and science problems @xcite , and writing code @xcite . despite the incredible successes, we have little understanding of why llms are effective at problems that require reasoning. in fact we have limited techniques to quantifiably study the question of reasoning beyond just evaluating accuracy. recent ideas like chain-of-thought prompting (cot) @xcite encourage the model to "think step by step" and output a verbose reasoning in text. however, verifying such reasoning at scale will incur the infeasible cost of manually going over the text outputs. furthermore, we would like the model's reasoning to be consistent with its outputted answer, in order to trust the presented reasoning. for these considerations, we would like our models to output a concise reasoning or explanation for its answer that can be automatically verified. in particular, we desire reasoning in the form of explanations that are • verifiable: for ease of evaluating correctness of the outputted reasoning, and in this work we use concise and verifiable explanations to study reasoning abilities of llms in math word problems @xcite . llms have shown to achieve good zero-shot accuracy on many numeric mwp benchmarks @xcite . ## experimental results we pick 150/1000 examples from the svamp dataset (due to budget constraints) and run each examples 5 times. we use gpt-3's davinci-002 model with temperature 0.0 for (mostly) deterministic outputs, with a max token length of 256. ## conclusions and future work this paper studies reasoning in llms for mwps and results suggest that lms are good at zero-shot solving of symbolic mwps, and that this ability can lead to concise explanations. self-prompting emerges as a promising idea to generate better explanations and the ensembling effect demonstrated by it can potentially have other applications (left for future work). alignment with self-prompting, while significantly better than with raw symbolic outputs, still has a lot of scope for improvement. aspects that are not considered are few-shot learning of explanations and the role of temperature, which could improve accuracy and alignment. finally the notion of "concise explanation" to study reasoning can have implications beyond mwps.