Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024)

Abstract: Mathematical reasoning serves as a cornerstone for assessing the fundamental cognitive capabilities of human intelligence. In recent times, there has been a notable surge in the development of Large Language Models (LLMs) geared towards the automated resolution of mathematical problems. However, the landscape of mathematical problem types is vast and varied, with LLM-oriented techniques undergoing evaluation across diverse datasets and settings. This diversity makes it challenging to discern the true advancements and obstacles within this burgeoning field. This survey endeavors to address four pivotal dimensions: i) a comprehensive exploration of the various mathematical problems and their corresponding datasets that have been investigated; ii) an examination of the spectrum of LLM-oriented techniques that have been proposed for mathematical problem-solving; iii) an overview of factors and concerns affecting LLMs in solving math; and iv) an elucidation of the persisting challenges within this domain. To the best of our knowledge, this survey stands as one of the first extensive examinations of the landscape of LLMs in the realm of mathematics, providing a holistic perspective on the current state, accomplishments, and future challenges in this rapidly evolving field.

## introduction mathematical reasoning is crucial to human intelligence, driving ongoing efforts in the ai community to autonomously tackle math challenges. this pursuit inherently calls for an augmentation of ai capabilities, delving into the intricate realms of textual comprehension, image interpretation, tabular analysis, symbolic manipulation, operational logic, and a nuanced grasp of world knowledge. as the ai landscape evolves, the endeavor to empower machines with a comprehensive understanding of diverse mathematical facets becomes not only a testament to technological prowess but also a pivotal stride towards achieving a more generalized and adept ai. in recent times, the landscape of ai has been reshaped by the ascendancy of large language models (llms) as formidable tools for automating intricate tasks. notably, llms have proven to be potent assets in unraveling the nuances of mathematical problem-solving @xcite . their language capabilities fuel focused exploration in utilizing them for mathematical reasoning, uncovering fresh insights into the synergy between language and logic. however, amid this progress, the current state of llm-oriented research in mathematics presents a complex panorama. diverse mathematical problem types pose a formidable challenge, exacerbated by the varied evaluation metrics, datasets, and settings employed in the assessment of llm-oriented techniques @xcite . the lack of a unified framework hampers our ability to gauge the true extent of progress achieved and impedes a coherent understanding of the challenges that persist in this evolving field. this survey endeavors to cast a spotlight on the multifaceted landscape of llms in the realm of mathematics. we plan to traverse four crucial dimensions: a meticulous exploration of math problem types and the datasets associated with them; an in-depth analysis of the evolving techniques employed by llms in mathematical problem-solving; an examination of factors that affect the llms solving math problems; and a critical discussion on the persisting challenges that loom over this burgeoning field. to our knowledge, this survey marks one of the first comprehensive examinations of llms specifically tailored for mathematics. by weaving together insights from various dimensions, we aim to provide a holistic understanding of the current state of affairs in llm-driven mathematical reasoning, shedding light on achievements, challenges, and the uncharted territories that await exploration in this captivating intersection of language and logic. ## related work to the best of our knowledge, the existing literature on summarizing mathematical research, particularly within the context of llms, remains limited. notably, @xcite conducted a comprehensive evaluation of llms, incorporating an examination of their performance in mathematical problem-solving, albeit with a relatively brief exploration of the mathematical field. conversely, both @xcite and @xcite delved into the application of deep learning in the domain of mathematical reasoning. our work distinguishes itself on three fronts: firstly, we concentrate on llms, providing a more in-depth analysis of their various advancements; secondly, beyond merely reporting progress, we engage in a thorough discussion of the challenges inherent in this trajectory; and thirdly, we extend our scrutiny to encompass the perspective of mathematics pedagogy. in doing so, we contribute a nuanced perspective that seeks to broaden the understanding of llms in the context of mathematical research. the only work contemporaneous with ours is @xcite . in comparison, our contribution lies in: i) not only introducing various methods but also paying more attention to various factors affecting model performance; ii) taking a broader perspective on the progress of llm in the field of mathematics, elucidating not only from the ai perspective but also from the perspective of education. it emphasizes that the pursuit of model performance alone, while neglecting human factors, is something that needs attention. ## math problems & datasets this section concisely overviews prominent mathematical problem types and associated datasets, spanning arithmetic, math word prob-lems, geometry, automated theorem proving, and math in vision context. ## methodologies we summarize these methods into three progressive levels: i) prompting frozen llms, ii) strategies enhancing frozen llms, and iii) fine-tuning llms. ## conclusion this survey on llms for mathematics delves into various aspects of llms in mathematical reasoning, including their capabilities and limitations. the paper discusses different types of math problems, datasets, and the persisting challenges in the domain. it highlights the advancements in llms, their application in educational settings, and the need for a human-centric approach in math education. we hope this paper will guide and inspire future research in the llm community, fostering further advancements and practical applications in diverse mathematical contexts.