FinanceMATH: Knowledge-Intensive Math Reasoning in Finance Domains (2024)

Abstract: We introduce FinanceMath, a novel benchmark designed to evaluate LLMs' capabilities in solving knowledge-intensive math reasoning problems. Compared to prior works, this study features three core advancements. First, FinanceMath includes 1,200 problems with a hybrid of textual and tabular content. These problems require college-level knowledge in the finance domain for effective resolution. Second, we provide expert-annotated, detailed solution references in Python program format, ensuring a high-quality benchmark for LLM assessment. We also construct a finance-domain knowledge bank and investigate various knowledge integration strategies. Finally, we evaluate a wide spectrum of 44 LLMs with both Chain-of-Thought and Program-of-Thought prompting methods. Our experimental results reveal that the current best-performing system (i.e., GPT-4o) achieves only 60.9% accuracy using CoT prompting, leaving substantial room for improvement. Moreover, while augmenting LLMs with external knowledge can improve model performance (e.g., from 47.5% to 54.5% for Gemini-1.5-Pro), their accuracy remains significantly lower than the estimated human expert performance of 92%. We believe that FinanceMath can advance future research in the area of domain-specific knowledge retrieval and integration, particularly within the context of solving reasoning-intensive tasks.

## introduction large language models (llms) have been increasingly recognized for their potential for complex problem-solving in real-world scenarios @xcite @xcite . solving math reasoning problems has emerged as a key method for assessing llms' capabilities (roy first, we know from the table that the total liabilities for company a in 2018 is 1200. (...abbreviate…) therefore, the final answer is 1,200 model output with chain-of-thought prompting: def solution(): @xmath0: figure 1 : an example of financemath. to answer the given question, llms are required to comprehend specialized financial terms, such as "passive equity ownership interest" and "proportionate consolidation method". additionally, they must interpret tabular data within the question and accurately identify question-relevant data points in the table. @xcite @xcite , as it demands both understanding contextual information and reasoning over complex logics. recent advancements in llms have led to remarkable progress in solving fundamental math problems @xcite @xcite @xcite . however, as illustrated in dataset domain level source # examples table knowledge-solution format reasoning? intensive? mawps @xcite math elem. school generated 3,320 ✗ ✗ text asdiv @xcite math elem. school internet 2,305 ✗ ✗ math equation svamp @xcite math elem. school asdiv 1,000 ✗ ✗ math equation math23k @xcite math elem. school internet 23,162 ✗ ✗ math equation gsm8k @xcite math middle school crowdsource 8,500 ✗ ✗ text math @xcite math high school competition 12,500 ✗ ✗ text aqua @xcite math college gmat, gre 100,000 ✗ ✗ text mathqa @xcite math college aqua 100,000 ✗ ✗ math equation @xcite math college aqua 23,914 ✗ ✗ python program mathvista @xcite math elem. to college internet+expert 6,141 few few text tabmwp @xcite ) math middle school textbooks 38,431 ✓ ✗ text finqa @xcite finance college expert 8,281 ✓ ✗ math program tat-qa @xcite finance college expert 16, 552 ✓ ✗ text multihiertt @xcite finance college expert 10,440 ✓ ✗ math equation docmath-eval @xcite finance college expert 5,974 ✓ few python program theoremqa @xcite al., 2023c) stem college internet+expert 800 ✗ ✓ text financemath (ours) finance college internet+expert 1,200 ✓ ✓ python program measuring progress in specialized areas such as finance and healthcare typically involves addressing domain-specific and knowledge-intensive problems, which goes beyond the scope of general mathematical reasoning. recognizing this gap in the existing benchmarks, we focus on the finance domain. we chose this domain because, as illustrated in figure 1 , it often involves scenarios requiring not only basic mathematical skills but also a deep understanding of financial concepts @xcite @xcite . additionally, the finance domain frequently employs tables to represent data @xcite @xcite @xcite , which adds another layer of complexity to the knowledge-intensive problem-solving. we introduce financemath, the first benchmark tailored for evaluating llms in the context of knowledge-intensive math reasoning in the finance domain. the dataset contains 1,200 problems that cover a broad range of finance subareas, with 40.2% of the problems necessitating data interpretation over tabular data. each problem is accompanied by expert-annotated, python-formatted solutions, pro-viding a comprehensive reference for evaluating the llms' performance. additionally, we collect and release a comprehensive knowledge bank, which includes detailed definitions and explanations for 864 financial terms and concepts, facilitating future research on improving knowledge-intensive problem-solving through knowledge retrieval. we evaluate a wide spectrum of open-source and proprietary llms, specifically, 51 model models from 16 organizations. notably, this includes math-specific @xcite @xcite , code-based @xcite @xcite llms, as well as mixture of experts (moe) llms @xcite . two prompting methods, chain-of-thought @xcite and program-of-thought @xcite , are adopted for experiments. our experimental results demonstrate a significant gap between existing llms and human experts. specifically, the current best-performing system (i.e., gpt-4o) achieves only 60.9% accuracy with cot prompting, which still lags far behind human expert performance in the open-book setting, which stands at 92%. these results highlight the challenges of financemath, underscoring the need for further advancements in llms for knowledge-intensive problem-solving capabilities. next, we investigate how to integrate domainspecific knowledge to enhance the problem-solving capabilities of llms. we investigate various popular knowledge integration strategies and reveal that including question-relevant knowledge into the prompt can consistently improve llms' performance. this provides insights for future work to develop more advanced knowledge-augmented strategies to realize higher performance gains. our contributions are summarized below: ## financemath benchmark in this section, we describe the dataset construction process for financemath. we begin by constructing a knowledge bank that includes wellformulated definitions of 864 financial terms. we then instruct expert annotators to use knowledge terms within the constructed knowledge bank to create knowledge-intensive questions with a hybrid of textual and tabular content. ## evaluated systems this section discusses the investigated llms and prompting methods in our work. ## knowledge augmentation analysis in this section, we provide a comprehensive analysis to understand the performance of llms and the quality of knowledge incorporated into the input context, aiming to provide insights for future work on solving knowledge-intensive tasks. ## related work the development of general-purpose intelligent systems is significantly dependent on the foundational aspect of mathematical reasoning, a topic that has garnered considerable attention in the academic community. as illustrated in ## conclusion this paper introduces financemath, a benchmark aimed at assessing llms in knowledge-intensive math reasoning. our comprehensive evaluations of 51 llms, using both cot and pot prompting methods, identify significant areas where llms need to enhance their specialized knowledge for complex problem-solving in expert domains. additionally, our knowledge augmentation analysis indicates that integrating domain-specific knowledge can improve llms' problem-solving abilities. we believe this research provides valuable insights into advancing llms within expert domains.