A Survey of Deep Learning for Mathematical Reasoning (2023)

Abstract: Mathematical reasoning is a fundamental aspect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems in language has garnered significant interest in the fields of machine learning and natural language processing. For example, mathematics serves as a testbed for aspects of reasoning that are challenging for powerful deep learning models, driving new algorithmic and modeling advances. On the other hand, recent advances in large-scale neural language models have opened up new benchmarks and opportunities to use deep learning for mathematical reasoning. In this survey paper, we review the key tasks, datasets, and methods at the intersection of mathematical reasoning and deep learning over the past decade. We also evaluate existing benchmarks and methods, and discuss future research directions in this domain.

## neural networks for mathematical reasoning neural networks have become a popular tool in the field of mathematical reasoning, mirroring their success in nlp. in recent years, a number of different neural network architectures have been proposed for mathematical reasoning tasks, including seq2seq-based networks, graph-based networks, and attention-based networks. these methods are outlined in more detail in ## pre-trained language models for mathematical reasoning pre-trained language models @xcite @xcite have demonstrated remarkable performance gains on a wide range of nlp tasks. by pre-training on a large corpus of text, the models learn valuable world knowledge @xcite , which could be applied to downstream tasks. similar ideas can be applied to math-related problems, and previous work has shown the promising performance of pretrained language models in answering math word problems @xcite , assisting with theorem proving @xcite , as well as solving other mathematical tasks @xcite . however, though large language models excel in modeling natural language, there are several challenges to using them for mathematical reasoning. first, pre-trained language models are not specifically trained on mathematical data. this likely contributes to them being less proficient in math-related tasks compared to natural language tasks. there is also less mathematical or scientific data available for large-scale pre-training compared to text data. second, the size of pre-trained models continues to grow, making it expensive to train the entire model from scratch for specific downstream tasks. additionally, downstream tasks may deal with different input formats or modalities, such as structured tables @xcite or diagrams @xcite . to address these challenges, researchers have to adjust pre-trained models by finetuning them on downstream tasks or adapting the neural architectures. ## in-context learning for mathematical reasoning large language models (llms), such as gpt-3 @xcite , have recently revolutionized the field of natural language processing (nlp), especially on account of their powerful few-shot incontext learning capabilities @xcite . in-context learning (icl) enables llms to perform target tasks by providing some task examples as conditions at inference time, without updating model parameters @xcite . icl allows users to quickly build models for new use cases without worrying about fine-tuning and storing a large amount of new parameters for each task, so it is widely used in fewshot settings nowadays @xcite ). an in-context example typically contains an input-output pair with some prompt words, e.g., please select the largest number from the list. input: [2, 4, 1, 5, 8]. output: 8, and few-shot works by giving multiple examples, and then a final input example, where the model is expected to predict the output. however, such standard few-shot promptings, in which the llm is given in-context examples of input-output pairs in front of test-time examples, have not yet proved sufficient to achieve high performance on challenging tasks such as mathematical reasoning @xcite . chain-of-thought prompting (cot) @xcite leverages intermediate natural language rationales as prompts to enable llms to first generate reasoning chains and then predict an answer for an input question. for example, a cot prompt for solving the math word problem could be question: roger has 5 tennis balls. he buys 2 more cans of tennis balls. each can has 3 tennis balls. then, how many tennis balls does roger have now? answer: roger started with 5 balls. 2 cans of 3 tennis balls each are 6 tennis balls. 5 + 6 = 11. the answer is 11. apart from @xcite showing that llms are decent zero-shot reasoners when given the "let's think step by step!" prompt, most of the recent work has focused on how to improve chainof-thought reasoning under the few-shot setting. this work is mainly divided into two parts, (i) selecting better in-context examples and (ii) creating better reasoning chains. ## conclusion in this paper, we present a comprehensive survey of deep learning for mathematical reasoning. we review the various tasks, datasets, and deep learning approaches. we also identify several gaps in the existing datasets and methods. finally, we outline directions for future research and highlight the potential for further exploration in this field. our goal with this paper is to provide a comprehensive and useful resource for readers interested in the development of deep learning for mathematical reasoning. to aid in this effort, we have created a reading list that will be continually updated in a github repository at https://github.com/lupantech/dl4math .