CF in B ench: A Comprehensive C hinese Financial Benchmark for Large Language Models (2025)

Abstract: Large language models (LLMs) have achieved remarkable performance on various NLP tasks, yet their potential in more challenging task like finance, has not been fully explored. In this paper, we present CFinBench: a meticulously crafted, the most comprehensive evaluation benchmark to date, for assessing the financial knowledge of LLMs under Chinese context. In practice, to better align with the career trajectory of Chinese financial practitioners, we build a systematic evaluation from 4 first-level categories: (1) Financial Subject: whether LLMs can memorize the necessary basic knowledge of financial subjects, such as economics, statistics and auditing. (2) Financial Qualification: whether LLMs can obtain the needed financial qualified certifications, such as certified public accountant, securities qualification and banking qualification. (3) Financial Practice: whether LLMs can fulfill the practical financial jobs, such as tax consultant, junior accountant and securities analyst. (4) Financial Law: whether LLMs can meet the requirement of financial laws and regulations, such as tax law, insurance law and economic law. CFinBench comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment. We conduct extensive experiments on a wide spectrum of representative LLMs with various model size on CFinBench. The results show that GPT4 and some Chinese-oriented models lead the benchmark, with the highest average accuracy being 66.02%, highlighting the challenge presented by CFinBench. All the data and evaluation code are open sourced at https://cfinbench.github.io/

## introduction recently, there has been a significant advancement in llms, exemplified by the representative mod-* equal contribution † corresponding author els like chatgpt (openai, 2021), gpt4 (ope-nai, 2023), llama @xcite , baichuan @xcite , in-ternlm @xcite and chatglm @xcite , etc. at the same time, the corresponding evaluation works for llms are flourishing and a series of evaluation benchmarks have been proposed like mmlu @xcite , c-eval @xcite , xiezhi @xcite and agieval @xcite , etc. these benchmarks have been instrumental in catalyzing the progress of llms, as they enable the quantitative assessment of advanced knowledge and complex reasoning abilities. finance is the backbone of modern society, playing a vital role in facilitating economic growth and prosperity @xcite . however, mastering the intricacies of financial knowledge is challenging for individuals, due to its intricate nature and dynamic environment. therefore, endowing llms with financial knowledge is essential as it can provide significant convenience and insight to humanity. for example, bloomberggpt @xcite , which possesses 50 billion parameters, exhibits superior performance across multiple financial tasks. similarly, finma @xcite is crafted by fine-tuning llama @xcite with the financial instruction data. of course, a comprehensive financial evaluation benchmark is also essential for financial llms. several benchmarks have been introduced for better evaluating financial llms. flue @xcite first introduces the financial benchmark across 5 nlp tasks in english domain, and its successor, flare @xcite , further extends it with financial time-series reasoning task like stock price movements forecasting. in addition to english domain, benchmarks in chinese are another one of significant importance. bbt-cfleb @xcite presents the first chinese financial evaluation benchmark, which includes 6 cfinbench political economy (政治经济学) western economics (西方经济学) microeconomics (微观经济学) macroeconomics (宏观经济学) industrial economics (产业经济学) public finance (财政学) international trade (国际贸易学) statistics (统计学) auditing (审计学) economic history (经济史) finance (金融学) tax practitioner qualification (税务从业资格) futures practitioner qualification (期货从业资格) fund practitioner qualification (基金从业资格) real estate practitioner qualification (地产从业资格) insurance practitioner qualification (保险从业资格) securities practitioner qualification (证券从业资格) banking practitioner qualification (银行从业资格) certified public accountant (cpa) (注册会计师) figure 1 : cfinbench comprises 4 first-level categories and 43 second-level categories, which are more align with the career trajectory of chinese financial practitioners. in this paper, we present cfinbench: a meticulously crafted, the most comprehensive evaluation benchmark to date, for assessing the capabilities of llms on chinese financial tasks. the design philosophy of our benchmark aligns with the career progression trajectory of financial practitioners, which can be likened to a 'leveling up' in a game. specifically, it begins with mastering the required foundational knowledge in financial subjects, followed by obtaining the necessary qualified certifications, and subsequently honing skills through practical experience in industry applications. last but not least, compliance with financial laws and regulations is also a crucial aspect. as described in figure 1 , we include 4 first-level categories: (1) financial subject: examining whether llms can memorize the foundational knowledge in financial subjects, such as political economy, statistics and macroeconomics, etc. (2) financial qualification: examining whether llms can obtain the necessary qualified certifications for financial practitioners, such as tax practitioner qualification, futures practitioner qualification and fund practitioner qualification, etc. (3) financial practice: examining whether llms can fulfill the specific tasks in financial jobs, such as junior banking professional, asset appraiser and junior statistician, etc. (4) financial law: examining whether llms can comply with the financial laws and regulations, such as securities law, insurance law and economic law, etc. cfinbench are primarily sourced from the mock exams and financial reports freely available on the internet. to enhance the quality and diversity of the benchmark, and mitigate the problem of data contamination, we perform a series of rigorous data processing pipelines, encompassing data cleaning, internal and external de-duplication, llm-assisted rephrasing, option shuffling, and multi-round human-in-the-loop cross-validation. cfinbench comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment. we conduct extensive experiments on a wide spectrum of representative llms with various model size on cfinbench. the results show that gpt4 and some chinese-oriented models like qwen @xcite , yi @xcite , and xuanyuan @xcite , etc. lead the benchmark, with the highest average accuracy being 66.02%, highlighting the challenge presented by cfinbench. it also indicates that there is still significant room for improvement in current large language models (llms) within the chinese financial domain. bloomberggpt @xcite , the first proprietary llm comprising 50 billion parameters, has been meticulously tailored for the financial domain. the successors like fingpt @xcite and finma @xcite introduce the financial llms based on fine-tuning llama, by means of low-rank adaptation or full-parameters. also, the chinese-oriented financial llms like disc-finllm @xcite , cfgpt @xcite , xuanyuan @xcite and yunshan @xcite , etc. have also demonstrated excellent performance across multiple chinese financial tasks. at the same time, to objectively and quantitatively measure the capabilities of llms, a comprehensive and thorough evaluation benchmark is crucial. financial benchmarks flue @xcite first introduces a financial evaluation benchmark across 5 nlp tasks in english context, and its successor, flare @xcite , further includes financial time-series reasoning task like stock movement prediction. the finben @xcite reorganizes 35 public english datasets across 23 financial tasks into three spectrums of difficulty. for the chinese domain, bbt-cfleb @xcite presents the first chinese financial evaluation benchmark, which includes 6 datasets covering both understanding and generation tasks. by integrating flare @xcite and bbt-cfleb @xcite , ice-flare @xcite enables the evaluation of bilingual financial tasks. cfbenchmark @xcite assesses the text processing capabilities across recognition, classification, and generation tasks. cgce @xcite incorporates 200 general and 150 finance-specific question-answering questions. the recent study of cflue @xcite builds the datasets tailored for both knowledge assessment and application assessment. by the way, the most similar works to ours are financeiq @xcite and fineval @xcite . fi-nanceiq @xcite encompasses 10 primary categories, including economists, securities professionals and actuaries, etc. with a total of 36 subcategories and 7,173 single-choice questions. fineval @xcite includes four primary categories, i.e., finance, economy, accounting, and certificate, with a total of 34 subcategories and 4,661 single-choice questions. however, the two benchmarks are limited in size. in contrast, the proposed cfinbench is more comprehensive, comprising a more reasonable dimension of financial abilities, with a total of 99,100 questions across 3 question types. the detailed comparison of the cfinbench with other chinese-oriented financial benchmarks is summarized in ## conclusion in this paper, we present cfinbench, the most comprehensive evaluation benchmark to date, for assessing the financial domain knowledge of llms under chinese context. we improve the quality and diversity of the data and mitigate the issue of data contamination through a series of processes, including data cleaning, internal and external deduplication, llm-assisted question rephrasing, option shuffling, and multiple rounds of human crossvalidation. four first-level categories are included in cfinbench: financial subject, financial qualification, financial practice, and financial law, which are more align with the career trajectory of financial practitioners in china. the cfinbench comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment. we conduct extensive evaluations on a wide spectrum of llms with various model size. the results show that there is still significant room for improvement for current llms in the chinese financial domain.