F in E val: A C hinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2025)

Abstract: Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored, and their performance on complex tasks like financial agent remains unknown. This paper presents FinEval, a benchmark designed to evaluate LLMs’ financial domain knowledge and practical abilities. The dataset contains 8,351 questions categorized into four different key areas: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge, and Financial Agent. Financial Academic Knowledge comprises 4,661 multiple-choice questions spanning 34 subjects such as finance and economics. Financial Industry Knowledge contains 1,434 questions covering practical scenarios like investment research. Financial Security Knowledge assesses models through 1,640 questions on topics like application security and cryptography. Financial Agent evaluates tool usage and complex reasoning with 616 questions. FinEval has multiple evaluation settings, including zero-shot, five-shot with chain-of-thought, and assesses model performance using objective and subjective criteria. Our results show that Claude 3.5-Sonnet achieves the highest weighted average score of 72.9 across all financial domain categories under zero-shot setting. Our work provides a comprehensive benchmark closely aligned with Chinese financial domain. The data and the code are available at https://github.com/SUFE-AIFLMLab/FinEval.

## introduction with the development of the financial industry, its integration with large language models has become increasingly close. the financial sector needs large language models to process massive amounts of * corresponding authors. † these authors contributed equally to this work. data, predict market trends, and optimize decisionmaking processes, thereby helping financial institutions enhance efficiency and reduce risks. this integration requires large models to possess critical capabilities in areas such as financial academic knowledge, industry knowledge, financial security, and financial agents. financial academic knowledge necessitates that models have a foundational understanding of finance, serving as the baseline for applying large language models in the financial domain. financial industry knowledge considers language interactions in practical application scenarios, requiring large models to have generalization capabilities across different contexts. financial security involves various aspects of privacy for individuals and enterprises, which is a priority for the financial industry. meanwhile, financial agents represent complex tasks within financial scenarios, involving numerous terms and methods that make them difficult for ordinary people to navigate. there are several well-established benchmarks for evaluating english and chinese foundation models, such as mmlu @xcite , @xcite and @xcite . nevertheless, there are some significant drawbacks to these benchmarks for financial tasks: they only cover a small portion of the financial scenarios, and are not widely applicable in real-world financial circumstances. in addition, there are other benchmarks specifically designed to focus on advanced llms abilities that become apparent as the scale increases, such as hard math problem-solving @xcite , and coding @xcite . additionally, there are financial-specific benchmarks such as finqa @xcite , fi-nanceiq (duxiaoman-di, 2023) and cflue @xcite . although these benchmarks contribute differently to financial tasks, they are all hindered by their limited applicability in real-world situations, narrow scopes, and inability to adequately capture the complexities of financial reasoning. as mentioned in @xcite , with the rapid development of llms, they are gradually acquiring the ability to handle complex tasks, but there are still privacy and security issues. @xcite indicates that llms still face challenges in managing complex trading tasks in the financial domain. therefore, for tasks like financial security and financial agent, which are more closely integrated with real-world financial scenarios and require higher standards, appropriate datasets are needed to assess llm capabilities. financial security emphasizes that large models must ensure the protection of personal information and cybersecurity in real-world financial applications. meanwhile, financial agent highlights the need for large models to possess strong information processing and reasoning capabilities in the complex and dynamic financial market, as well as the ability to flexibly use various financial tools to decompose and solve complex financial tasks. therefore, evaluating llms' capabilities in financial security and financial agent tasks is crucial for the financial domain. we introduce fineval, an extensive benchmark designed to evaluate the practical capabilities of large language models in the financial domain, with a particular focus on financial security and financial agent tasks within the chinese context. the dataset contains 8,351 questions distributed divided into major domains: financial academic knowledge, financial industry knowledge, financial security knowledge and financial agent, as illustrated in figure 1 . financial academic knowledge comprises 4,661 multiple-choice questions spanning 34 subjects like finance and accounting, testing the theoretical foundation of models. financial industry knowledge, with 1,434 questions, targets real-world financial practices, covering areas such as investment research. financial security knowledge is assessed through 1,640 questions, covering eleven financial security tasks, including security analysis, vulnerability protection, etc. these questions evaluate the comprehensive capabilities of large language models in terms of security from multiple dimensions. finally, financial agent consists of 616 questions, assessing the performance of large language models under complex information in real financial markets across three major dimensions and seven tasks. our experiments evaluated various models, such as claude 3.5-sonnet, gpt-4o, qwen2.5-72b-instruct, and xuanyuan3-70b-chat, to demonstrate their capabilities in various fi-nancial tasks. these models were assessed through zero-and few-shot standard prompting, as well as chain-of-thought prompting @xcite ). in a series of experiments conducted under a zero-shot setting to evaluate llm performance on financial knowledge, claude 3.5-sonnet performed the best among over 19 models, with the highest weighted average score (72.9). gpt-4o also demonstrated strong capabilities, particularly in financial security, achieving a notable score of 81.8. in addition, the open-source model qwen2.5-72b-instruct outperformed claude 3.5-sonnet in financial security and matched gpt-4o with a score of 81.8, making it a highly competitive model in this domain. however, this level of accuracy and similarity implies that there is still significant room for improvement in the field of finance for all llms. in summary, our main contributions include: ## related work general benchmark current general benchmarks primarily focus on conventional tasks such as natural language understanding, text generation, logical reasoning, programming skills, professional knowledge, and multi-turn dialogues. these benchmarks rarely address tasks in the financial domain, particularly the security issues that are highly valued in finance and complex tasks like those involving agents. there are several well-established benchmarks for evaluating english and chinese foundation models, including mmlu @xcite , helm @xcite , agieval @xcite , @xcite . other benchmarks focus on large language models' advanced abilities, like hard math problem-solving @xcite and coding @xcite , which become more apparent as model scale grows. truthfulqa (lin et al., 2021) measures the authenticity of language models when answering questions. big-bench (srivastava et al., 2022) evaluates language models across various domains. cblue (zhang et al., 2021) is a collection of language understanding tasks in the biomedical field, including named entity recognition and information extraction. gaokao-bench (zhang et al., 2023) gathers questions from the chinese gaokao examination to evaluate the language comprehension and logical reasoning abilities of llms. similarly, agieval (zhong et al., 2023) assess the performance of foundation models on human-centric standardized exams, such as college entrance exams. financial benchmark however, the availability of benchmarks specifically catering to the financial domain remains limited, and current research mainly focuses on the financial academic and financial industry sectors. flue @xcite , convfinqa @xcite , bbt-cfleb @xcite , and finqa @xcite in the english domain all focus solely on knowledgebased question answering. in the chinese domain, financeiq @xcite also emphasizes knowledge-based questions. cflue @xcite provides questions and nlp tasks related to chinese financial knowledge, but its actual evaluation tasks are still limited to the knowledge level and do not include important topics such as financial security, nor do they delve into more complex agent-related tasks designed for financial business scenarios. additionally, there is no comparison between large models and ordinary individuals or experts in the financial domain, making it difficult to accurately assess the true capabilities of large models in financial scenarios. ## experiments our following experiments show the evaluation results of diverse llms on fineval to analyze their performance and provide baselines for future usage of fineval. ## error analysis to further identify the shortcomings of large language models in financial knowledge and tasks, we analyze the errors made by the model during the testing process. financial agent includes a series of relatively open-ended, subjective tasks that place high demands on the model, and the model displayed a diverse and rich variety of error types when completing more complex tasks. three types of errors were identified for answers that did not receive full scores from the evaluation model: logical reasoning error, contextual misunderstanding, and ambiguity handling weakness. logical reasoning error occurs when the model fails to draw correct logical conclusions. contextual misunderstanding happens when the model misses or misinterprets relevant context. ambiguity handling weakness refers to the model's inability to handle unclear or ambiguous questions properly. detail examples can be found in appendix b. we selected ten representative models and randomly sampled 27 erroneous responses for each model on each task. the results are shown in figure 3 . each bar in the chart represents the proportion of a specific type of error among all errors made by a particular model. across all the erroneous responses, the average proportion for open-source models in logical reasoning error, contextual misunderstanding, and ambiguity handling weakness are 29.0%, 41.6%, and 29.4%, respectively. in comparison, the corresponding average proportion for closed-source models are 21.3%, 37.6%, and 41.1%. this suggests closed-source models are better at reasoning and understanding long texts, but more prone to semantic issues and hallucinations. this could be due to their larger parameter sizes, which, while improving reasoning and comprehension, also lead to over-interpretation of ambiguous content. specifically, the results show that open-source models are more likely to encounter logical reasoning errors and contextual misunderstanding, while closedsource models are more prone to ambiguity handling weakness. for logical reasoning errors, models struggle with multi-step reasoning tasks, often misusing formulas or making calculation mistakes. in contextual misunderstanding, models fail to connect distant parts of the text. for ambiguity handling weakness, models have trouble interpreting vague information, sometimes leading to hallucinations. they often include unnecessary tools or data in their responses, indicating a need for better semantic understanding and relevance filtering. ## conclusion this study introduces fineval, a benchmark for evaluating large language models' capabilities in the financial domain. unlike previous financial evaluation benchmarks, fineval delves deeper into financial security and financial agent, covering pressing security issues and complex agent tasks in the financial field, assessing models' security and their ability to handle complex tasks. our results indicate that claude 3.5-sonnet performs the best among the 19 models evaluated, but it still faces challenges with more complex tasks. while it surpasses ordinary individuals, it has not reached the level of human experts. this study illustrates that although large language models have made certain breakthroughs in the financial domain, they still require a more in-depth and detailed understanding to enhance their task generalization capabilities in a wider and more complex diverse financial market environment. as an important benchmark for future research on large language models in the financial field, fineval provides a structured frame-work for measuring and improving the capability of large language models, contributing to the development of evaluation benchmarks in the chinese financial domain.