Benchmarking Large Language Models on CFLUE - A C hinese Financial Language Understanding Evaluation Dataset (2024)

Abstract: In light of recent breakthroughs in large language models (LLMs) that have revolutionized natural language processing (NLP), there is an urgent need for new benchmarks to keep pace with the fast development of LLMs. In this paper, we propose CFLUE, the Chinese Financial Language Understanding Evaluation benchmark, designed to assess the capability of LLMs across various dimensions. Specifically, CFLUE provides datasets tailored for both knowledge assessment and application assessment. In knowledge assessment, it consists of 38K+ multiple-choice questions with associated solution explanations. These questions serve dual purposes: answer prediction and question reasoning. In application assessment, CFLUE features 16K+ test instances across distinct groups of NLP tasks such as text classification, machine translation, relation extraction, reading comprehension, and text generation. Upon CFLUE, we conduct a thorough evaluation of representative LLMs. The results reveal that only Qwen-72B, GPT-4, and GPT-4-turbo achieve an accuracy exceeding 60% in answer prediction for knowledge assessment, suggesting that there is still substantial room for improvement in current LLMs. In application assessment, while GPT-4 and GPT-4-turbo rank as the top two performers on average, their significant advantage over open-source LLMs is noticeably diminished, given that Qwen-72B achieves the best performance in 2 out of 5 tasks. The datasets and scripts associated with CFLUE are openly accessible at https://github.com/aliyun/cflue .

## introduction recently, the remarkable capabilities exhibited by large language models (llms) have brought significant advancements and revolutionized natural language processing (nlp). in response to the emergence of llms, new benchmarks have been intro-* corresponding author duced to assess a varied range of abilities exhibited by these models. notable examples include mmlu @xcite , helm @xcite , big-bench @xcite , and glue @xcite , which are widely employed to evaluate llms' capabilities in english. additionally, benchmarks such as cleu @xcite , cmmlu @xcite , and ceval @xcite are widely used to evaluate their capabilities in chinese. however, assessing llms in the financial domain, especially within the chinese context, presents considerable challenges due to limitations in existing chinese financial evaluation datasets. notably, datasets like financeiq (duxiaoman-di, 2022) and fineval @xcite have restrictions in terms of size and diversity, hindering a comprehensive evaluation of llm performance. while financeiq and fineval focus solely on multiple-choice tasks, there is a lack of datasets allowing a thorough examination of llm capabilities in text generation. additionally, existing shared tasks, such as those in ccks @xcite @xcite , predominantly concentrate on event extraction tasks, limiting the objective and quantitative measurement of llm performance. inspired by the flue benchmark @xcite , which encompasses a comprehensive set of datasets across five financial domain tasks in english, this paper introduces a novel dataset named cflue (chinese financial language understanding evaluation). cflue addresses the aforementioned challenges by providing a benchmark for evaluating llm performance through various nlp tasks, categorized into knowledge assessment and application assessment. in the knowledge assessment section, cflue contains 38k+ questions from 15 types of financial qualification mock exams of varying difficulty levels and subjects. these questions are in the multiple-choice question format to enable standardized and objective evalua-tions. additionally, each question is accompanied by a solution explanation, facilitating open-ended or chain-of-thought assessments of llms' reasoning capabilities. in the application assessment section, cflue contains 16k+ test instances covering five groups of classical nlp tasks: text classification, machine translation, relation extraction, reading comprehension, and text generation. these instances are sourced from existing shared tasks or are annotated by professionals using real data sources. based on cflue, we assess the effectiveness of several representative llms across general and financial domains. this includes an examination of three openai llms, nine open-source llms in the general domain, and three open-source llms in the financial domain. from the experimental results, we have the following key findings: in summary, this paper provides valuable insights into the performance of llms in chinese financial contexts from multiple perspectives. our findings suggest that there still exists much room for improvement even for the current best performers. we hope that cflue could guide the developers to understand the abilities of their models from multiple dimensions and facilitate the growth of foundation models in chinese financial domain. financial evaluation datasets. a summary of recent benchmarks in the financial domain is presented in in chinese, the ccks series since 2019 ccks has released various datasets for event extraction tasks @xcite @xcite . additionally, both financeiq and fineval offer thousands of multiple-choice question answering pairs that could be used as evaluation suites for llms @xcite . however, compared to english, chinese datasets tend to focus on either event extraction @xcite or multiple-choice question-answering @xcite @xcite , demonstrating limited task diversity. in contrast, our paper introduces language source question type (task) english finqa @xcite question answering tat-qa @xcite question answering bizbench @xcite quantitative reasoning financebench @xcite question answering flue @xcite sentiment classification @xcite al., 2014) sentiment analysis (fiqa (fiqa, 2018)) news headline classification (sinha and khandait, 2020) named entity recognition (alvarado et al., 2015) structure boundary detection (finsbd3, 2021) question answering (fiqa (fiqa, 2018)) chinese ccks @xcite @xcite event extraction, event entity (and causality) extraction financeiq @xcite multiple-choice question answering fineval @xcite multiple-choice question answering fingpt-fineval @xcite multiple-choice question answering cflue (chinese financial language understanding evaluation), featuring a set of heterogeneous benchmark tasks for a more comprehensive evaluation. additionally, beyond evaluation suites, there are financial datasets like smoothnlp 2 , iree @xcite , suitable for training or fine-tuning models in the finance domain. other benchmark datasets. the development of lms @xcite has witnessed heterogeneous benchmarks to probe their diverse abilities. in english, traditional benchmarks focus on single tasks such as natural language understanding @xcite , reading comprehension @xcite , and reasoning @xcite . recently, researchers have introduced broader benchmarks like truth-fulqa @xcite , evaluating truthfulness across 38 categories spanning various domains. mmlu @xcite offers a multitask evaluation covering 57 tasks across different fields. additionally, big-bench @xcite and helm @xcite benchmarks cover as much to 204 and 42 tasks, respectively. in chinese, cleu @xcite is the first large-scale chinese benchmark that covers tasks such as classification and reading comprehension. aigeval @xcite evaluates llm capabilities in human-centric exams, including college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. ## cflue: chinese financial language understanding evaluation while various llms may exhibit similar performance on tasks within the general domain, it is often the domain-specific tasks that set them apart. the primary objective of cflue is to assess the financial domain knowledge capabilities of models from multiple dimensions. as illustrated in figure 1 , cflue evaluates the proficiency of llms from both knowledge and application perspective. the statistics of cflue are presented in ## conclusion in summary, this paper have presented cflue, comprehensive chinese datasets in the financial domain designed for evaluating the performance of llms across diverse nlp tasks. with over 38k multiple-choice questions and 16k instances across generation tasks, cflue offers a robust benchmark for assessing llm capabilities. the detailed results presented for several llms on cflue provide valuable insights into their performance. we hope that cflue will facilitate future advancements in llms, particularly in enhancing their proficiency of chinese nlp tasks within the financial domain.