XF in B ench: Benchmarking LLM s in Complex Financial Problem Solving and Reasoning (2025)

Abstract: Solving financial problems demands complex reasoning, multimodal data processing, and a broad technical understanding, presenting unique challenges for current large language models (LLMs). We introduce **XFinBench**, a novel benchmark with 4,235 examples designed to evaluate LLM’s ability in solving comple**X**, knowledge-intensive **Fin**ancial problems across diverse graduate-level finance topics with multi-modal context. We identify five core capabilities of LLMs using XFinBench, i.e., _terminology understanding_, _temporal reasoning_, _future forecasting_, _scenario planning_, and _numerical modelling_. Upon XFinBench, we conduct extensive experiments on 18 leading models. The result shows that o1 is the best-performing text-only model with an overall accuracy of 67.3%, but still lags significantly behind human experts with 12.5%, especially in temporal reasoning and scenario planning capabilities. We further construct a knowledge bank with 3,032 finance terms for knowledge augmentation analysis, and find that relevant knowledge to the question only brings consistent accuracy improvements to small open-source model. Additionally, our error analysis reveals that rounding errors during calculation and blindness to position and intersection of curves in the image are two primary issues leading to model’s poor performance in calculating and visual-context questions, respectively.

## introduction finance constitutes a critical domain, characterized by the necessity for sophisticated problemsolving skills. beyond domain-specific knowledge, it necessitates advanced capabilities such as temporal reasoning @xcite , future forecasting @xcite , scenario planning (valmeekam et al., 1 code and dataset are accessible via github: https://github.com/zhihan72/xfinbench . 2022; @xcite , and numerical modeling @xcite . besides, complex finance problems in real world usually involves rich multimodal information, covering time series @xcite , long tabular @xcite and various charts @xcite . these complexities present significant challenges for large language models (llms), thereby rendering finance an appropriate testbed for the evaluation of llms. numerous datasets have been curated to assess the reasoning abilities of ai systems in the finance domain (see to bridge this gap, we introduce xfinbench, a novel benchmark specifically designed to evaluate llm's ability in solving complex, knowledgeintensive financial problems across diverse financial topics with multi-modal context. xfinbench consists of 4,235 examples derived from graduatelevel finance textbooks that ensures the complexity of financial problems in our dataset, and brings convenience to annotation of ground-truth knowledge to each problem. different from existing datasets that only evaluate the model's grasp of specialized financial vocabulary, i.e, terminology understanding, xfinbench identifies four more advanced capabilities essential for complex finance problemsolving ( §a and figure 2 ): (1) temporal reasoning, involving the comprehension of time-based data and temporal relationships ( §a.2.2); (2) future forecasting, testing logical reasoning in predicting financial trends based on theoretical finance models ( §a.2.3 ); (3) scenario planning, analyzing different potential future scenarios to assess their impact on financial decisions and strategies ( §a.2.4 ); and (4) numerical modelling, which involves constructing structured representations of companies and products' financial performance ( §a.2.5). moreover, xfinbench includes three tasks: statement judging, which evaluates the model's understanding of finance concepts; multi-choice question answering, which assesses strategic decisionmaking and predictive capabilities with visual data; and financial calculation, which tests mathematical reasoning in finance. to further investigate how domain-specific knowledge could boost llm's performance on our complex financial problems, we also develop a knowledge bank with 3,032 finance terms, which is integrated with financial problems through human annotation. we conduct extensive experiments on xfin-bench to evaluate the complex finance problemsolving ability of 18 leading llms , along with knowledge augmentation analysis and error analysis. we implement chain-of-thought (cot) method for all three tasks, and additionally apply program-of-thought (pot) for financial calculation. moreover, we establish a human performance baseline of human experts with finance degree. our results indicate that o1 is the best-performing textonly model with an overall accuracy of 67.3%, while claude-3.5-sonnet achieves the highest ac-curacy of 64.0% when visual-context questions included ( §4.2). despite that llms achieve comparable performance with human in terminology understanding, they significantly lag behind human experts in more advanced capabilities for complex finance problem-solving, including temporal reasoning and scenario planning-especially when visual context is involved (figure 1 ). these findings highlight that xfinbench represents a rigorous and challenging benchmark, offering a critical tool for advancing the development of llms in complex financial problem-solving and reasoning. our contributions are summarized as follows: • we propose xfinbench, a novel benchmark designed to evaluate llm's ability in solving complex, knowledge-intensive financial problems with multi-modal context ( §3). ## related work a wide range of datasets has been developed to evaluate the reasoning abilities of ai systems in the finance domain, as shown in tat-qa @xcite 16,552 quantity extraction tabular ✗ ✗ financial report w. crowdsource pacific @xcite 2,757 quantity extraction tabular ✗ ✗ existing dataset w. automatic pipeline finqa @xcite 8,281 quantity extraction tabular ✗ ✗ financial report w. crowdsource convfinqa @xcite 3,892 quantity extraction tabular ✗ ✗ existing dataset w. crowdsource fineval @xcite existing multi-modal datasets covering the finance domain primarily assess models' visual recognition abilities, overlooking domain-specific reasoning that derives meaningful insights from financial charts ( §a.3). benchmarks like mmmu @xcite , mmlu-pro @xcite , and mathvista @xcite include chart-based questions, but they typically focus on descriptive tasks such as identifying values or trends and recognizing technical terms. additionally, chart-oriented benchmarks such as chartqa @xcite , mmc @xcite , and charxiv @xcite emphasize general visual recognition and reasoning, while overlooking the contextual financial interpretation of visual data. in contrast, xfin-bench introduces a domain-specific perspective that requires models to integrate visual understanding with financial reasoning, enabling a more comprehensive assessment of ai capabilities in realistic financial scenarios. ## dataset construction our benchmark, xfinbench, is meticulously designed to facilitate complex reasoning in knowledge-intensive financial tasks. the dataset construction begins with the collection of questions and answers from three graduate-level finance textbooks and their solution manuals, accompanied by the creation of a knowledge bank of financial terms. to enrich the dataset, human experts annotate each question-answer pair with relevant financial terms and associated capabilities. given the evaluation challenges posed by open-ended questions from textbooks, we leverage gpt-4o within a generatethen-verify framework to expand the dataset and enhance its suitability for assessing llms. lastly, a rigorous quality validation process, conducted by human experts, ensures the dataset meets the highest standards of accuracy and relevance. ## experiments we conduct qualitative and quantitative studies for comprehensive evaluation of leading llms in knowledge-intensive finance tasks. ## conclusion in this work, we introduced xfinbench, a benchmark comprising 4,235 examples designed to evaluate the ability of llms to solve complex, knowledge-intensive financial problems across diverse topics and multi-modal contexts. evaluation results indicate that while o1 is the bestperforming text-only model with an overall accuracy of 67.3%, it falls significantly behind human experts by 12.5%, particularly in temporal reasoning and scenario planning capabilities. further analysis revealed that integrating ground-truth knowledge yields only limited performance improvements in tackling complex financial problems, and that limitations in models' calculation ability and visual information recognition present significant barriers to progress in finance domain. these findings underscore the critical role of xfin-bench in driving the development of ai agents capable of effectively solving complex, multi-modal financial problems.