Z hu J iu: A Multi-dimensional, Multi-faceted C hinese Benchmark for Large Language Models (2023)

Abstract: The unprecedented performance of LLMs requires comprehensive and accurate evaluation. We argue that for LLMs evaluation, benchmarks need to be comprehensive and systematic. To this end, we propose the Zhujiu benchmark, which has the following strengths: (1) Multi-dimensional ability coverage: We comprehensively evaluate LLMs across 7 ability dimensions covering 51 tasks. Especially, we also propose a new benchmark that focus on knowledge ability of LLMs. (2) Multi-faceted evaluation methods collaboration: We use 3 different yet complementary evaluation methods to comprehensively evaluate LLMs, which can ensure the authority and accuracy of the evaluation results. (3) Comprehensive Chinese benchmark: ZhuJiu is the pioneering benchmark that fully assesses LLMs in Chinese, while also providing equally robust evaluation abilities in English. (4) Avoiding potential data leakage: To avoid data leakage, we construct evaluation data specifically for 37 tasks. We evaluate 10 current mainstream LLMs, and conduct an in-depth discussion and analysis of their results. The ZhuJiu benchmark and open-participation leaderboard are publicly released at http://www.zhujiu-benchmark.com and we also provide a demo video at https://youtu.be/qypkJ89L1Ic.

## introduction with the continuous development of large language models (llms), the emergence of gpt4 @xcite is enough to trigger a new wave of technology. various types of llms have recently been rapidly developing, such as llama2 @xcite and chatglm2 @xcite , demonstrating impressive generalization abilities and broad applicability. therefore, it is crucial to 1 *co-first authors, they contributed equally to this work. specifically, on the one hand, for applicators, they need to understand the overall performance of llms or the advantages of llms in a specific aspect. constructing comprehensive and authoritative benchmarks can help applicators significantly improve the efficiency of using llms. on the other hand, for developers, the improvement direction of llms requires accurate evaluation results as guidance. an objective and fair benchmark can help them carry out relevant research work on llms more targetedly. to this end, scholars conduct extensive research on evaluations for llms and construct some superior benchmarks. normally, the evaluation for llms includes two aspects: ability evaluation and evaluation method. although traditional benchmarks such as glue @xcite , su-perglue @xcite and cuge @xcite still have a role to play in evaluating llms, their limitations are becoming increasingly apparent due to the growing diversity of evaluation dimensions and methods for llms. for the ability evaluation of llms, recent work proposes excellent benchmarks for llms in one or several aspects, such as knowledge, reasoning, language, safety and hallucination @xcite @xcite @xcite @xcite . however, a comprehensive evaluation of llms remains insufficient. for the evaluation method of llms, there are currently 3 main categories: (1) metrics evaluation: evaluating llms using existing datasets and corresponding metrics @xcite ; (2) chatgpt evaluation: using gpt-like llms to generate evaluation data and compare the response results of different llms @xcite ; (3) model arena: constructing one-on-one model arenas where humans compare the evaluation results of models based on their own judgment @xcite . despite these successful efforts for llms' evaluations, existing studies still suffer from several limitations: (1) current benchmarks tend to focus on evaluating llms on a single dimension of their abilities, which can not provide a comprehensive evaluation of llms. (2) most benchmarks only use a single evaluation method, which may not provide an accurate evaluation of all the abilities of llms. for example, while helm @xcite uses metrics to evaluate llms, it may not measure all abilities such as long-text generation or machine translation, etc. (3) the cross-lingual abilities of llms, especially for chinese, have garnered growing attention. however, the lack of a comprehensive chinese benchmark for llms remains a critical issue. (4) many current benchmarks only use public datasets for evaluation, risking potential data leakage. the results of evaluations based on this data lack credibility. in this paper, we propose the zhujiu benchmark to solve above mentioned problems, which can fill the gap in the development of a comprehensive benchmark for evaluating llms in chinese. the advantages of the zhujiu are as follows: (1) multi-dimensional ability coverage: we evaluate llms from 7 ability dimensions, including knowledge, chinese-specific, language, reasoning, refusal, safety and robustness abilities, covering 51 datasets to provide a comprehensive performance assessment. in addition, we also proposed a new paradigm for evaluating the knowledge ability. (2) multi-faceted evaluation methods coordination: we use metrics evaluation, scoring evaluation, and comparative evaluation for comprehensively evaluating llms to ensure authoritative and accurate evaluation results. (3) comprehensive chinese benchmark: zhujiu is the pioneering chinese benchmark that can comprehensively evaluate llms, while allowing equivalent assessment in english. (4) avoiding potential data leakage: in addition to collecting 14 commonly used datasets, we construct 37 datasets for the evaluation of llms, ensuring maximum avoidance of data leakage and evaluation fairness. the overall evaluation process is shown in figure 1 . we also release an online evaluation platform that supports multiple functions including visualizations of evaluation results, participating in model arena and submission of evaluation model, etc. moreover, we evaluate 9 publicly available llms, including chatglm @xcite , belle (yunjie ji and li, 2023), chatgpt (openai, 2022), and so on. based on the experimental results, we observe some interesting phenomena and summarize them in 4.2. in summary, the contributions of this paper are as follows: ## zhujiu benchmark as stated above, the zhujiu benchmark uses 3 evaluation methods to assess the abilities across seven dimensions of llms. this section provides a detailed introduction to the zhujiu benchmark covering the evaluation methods, datasets, and ability dimensions. we also detail the specific scoring rules in appendix a. the evaluation framework is shown in figure 2 . ## platform we develop an online platform to provide a range of services for the community as follows: visualizations of evaluation results we publish the rankings of all model evaluations on the platform, including specific scores for each ability and evaluation method, and the rankings will be updated continuously as the evaluations progress. ## experiment 4.1 evaluated models to facilitate the utilization and advancement of llms, the primary emphasis of zhujiu's inaugural evaluation phase is directed towards opensource llms with a parameter magnitude of approximately 10 billion, including: chatglm-6b @xcite , chatglm2-6b @xcite , belle-7b (yunjie ji and li, 2023), chatflow @xcite , phoenix-inst-chat-7b @xcite , chatyuan-large-v2 (xuanwei zhang and zhao, 2022), moss-moon-003-sft @xcite 2021). concurrently, we employ chatgpt (ope-nai, 2022) as a comparative benchmark and conduct an assessment of the gpt-3.5-turbo api service. ## conclusion and future work in this work, we present zhujiu, the pioneering multi-dimensional ability coverage, multi-faceted evaluation methods collaboration chinese benchmark. zhujiu is capable of using 3 evaluation methods to comprehensively evaluate llms across 7 ability dimensions, using 51 datasets. additionally, we independently construct 37 evaluation datasets to maximize the avoidance of data leakage issues in llm evaluation. we also focus on expanding the evaluation of knowledge ability, providing a new framework for assessing llms' knowledge ability. finally, we provide a comprehensive and continuously updated evaluation platform with multiple functions and in the first season of zhujiu, we evaluate 9 open-source llms. in the future, we plan to (1) continuously construct high-quality evaluation datasets to enrich zhujiu, (2) further perfect the assessment of knowledge ability and develop new evaluation methods for chinese characteristic ability, (3) further perfect the platform's functionality and update the platform's information.