CMMLU : Measuring massive multitask language understanding in C hinese (2024)

Abstract: As the capabilities of large language models (LLMs) continue to advance, evaluating their performance is becoming more important and more challenging. This paper aims to address this issue for Mandarin Chinese in the form of CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural sciences, social sciences, engineering, and the humanities. We conduct a thorough evaluation of more than 20 contemporary multilingual and Chinese LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. This highlights that there is substantial room for improvement in the capabilities of LLMs. Additionally, we conduct extensive experiments to identify factors impacting the models’ performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models for Chinese.

## introduction large language models (llms) have driven remarkable advancements in natural language processing and artificial intelligence, revolutionizing the field @xcite @xcite @xcite @xcite . however, assessing the knowledge and reasoning abilities of these models has become increasingly challenging, especially with the proliferation of llms that generate fluent and plausible responses. to this end, researchers have created various benchmarks intended to evaluate different model capabilities @xcite @xcite @xcite . @xcite proposed mmlu, a benchmark that encompasses various tasks ranging from elementary mathematics and computer science to management and law, which can be used to comprehensively measure llm capabilities in terms of the knowledge embedded in them. due to its multiple-choice question format, which facilitates easy evaluation, and the breadth of subject areas it encompasses, it has become widely used as a fundamental assessment tool of the knowledge encoded by llms. however, this benchmark is in english, which limits its ability to assess llms in other languages. although some researchers @xcite have attempted to automatically translate it to evaluate llms in other languages, the inherent bias towards western (and specifically us) culture in the dataset renders it unsuitable and even inappropriate for assessing llms across diverse cultures and languages. in this paper, we propose cmmlu (figure 1 ), a comprehensive chinese assessment suite specifically designed to evaluate the advanced knowledge and reasoning abilities of llms in a chinese linguistic and cultural context. cmmlu covers a wide range of subjects, comprising 67 topics from elementary to advanced professional levels. it includes subjects that require computational expertise, such as physics and mathematics, as well as disciplines within the humanities and social sciences. many of these tasks are not easily translatable from other languages due to their specific contextual nuances and wording. furthermore, numerous tasks within cmmlu have answers specific to china, which may not be universally applicable or considered correct in other regions or languages. we assess gpt4, chatgpt, and more than 20 advanced open-source multilingual and chinese llms on cmmlu. the results reveal that the majority of these models struggle to achieve an accuracy score of 60%, relative to random accuracy of 25%. notably, gpt4 achieves an average accuracy of 71%. these findings highlight the considerable room for improvement in llms in terms of chinese knowledge and language understanding. to gain a deeper understanding of the proficiency of the models in handling chinese knowledge, we conduct a comprehensive analysis. we first focus on examining model performance across various subjects and find that all models exhibit uneven performance across different subjects, with comparatively higher scores in humanities and social sciences, but lower scores in china-specific and stem subjects. furthermore, through extensive experiments, we find that: (1) most existing models do not benefit from chain-of-thought prompts in cmmlu; (2) few-shot examples help base models in the comprehension of tasks and enhance their reasoning abilities but do not help models that have undergone supervised fine-tuning (sft) or reinforcement learning from human feedback (rlhf); (3) llms perform worse on questions with negation words compared to those without negation words, but recently-released models mitigate this disparity either through better pre-training data or fine-tuning; and (4) questions with sub-options (section 4.2) are difficult for all existing llms, with even gpt4 dropping 20% in accuracy over such questions. ## related work benchmarking plays a crucial role in measuring ai development, particularly in the domain of llms. while benchmarks such as glue @xcite and @xcite have played an important role in tracking progress in natural language understanding (nlu) tasks, they primarily focus on specific language skills. with an increasing move to generative models which are highly adept at generating fluent outputs, the value of these benchmarks has diminished, and new datasets have been proposed to evaluate llm abilities over more general tasks, such as reading comprehension @xcite @xcite , summarization @xcite , commonsense reasoning @xcite @xcite , mathematical reasoning @xcite , and code generation @xcite . in order to comprehensively assess the capabilities of llms, some benchmarks have incorporated massive multi-task evaluations @xcite @xcite . an example is mmlu @xcite , which includes multiple domains and tasks based on real-world exams. it has become very popular for english llm evaluation due to its standardized and simplified format, comprehensive nature, and real-world relevance. there have also been variants of mmlu in other languages @xcite . given that mandarin chinese has the largest number of speakers worldwide, several benchmarks have been proposed for chinese llm evaluation. following in the footsteps of @xcite introduced clue, a benchmark for chinese nlu that is widely used today. they also recently proposed superclue @xcite , which specifically focuses on llms. recently, several chinese benchmarks have emerged in the style of mmlu, which are concurrent work with this paper. in detail, @xcite proposed aclue, focusing on ancient chinese language understanding. zeng (2023) presented mmcu, which covers four major domains (medicine, law, psychology, and education), with a particular focus on medicine and education. agieval @xcite includes problems from chinese and english standardized exams. c-eval @xcite and @xcite cover more than 50 tasks from standard exams in china, while c-eval covers various professions, and m3ke focuses on education exams. compared to these benchmarks, cmmlu has several distinct features. firstly, it includes more than 10 subjects that are not typically found in standard exams but are relevant to daily life, such as chinese food culture, and chinese driving rules. secondly, it covers not only china-specific knowledge but also general world knowledge, such as world religion, world history, and global facts. lastly, we have made our data completely public, enabling the community to evaluate their models freely and conveniently. a detailed comparison between cmmlu and other concurrent benchmarks is provided in section 3. ## cmmlu task overview we created an extensive multitask test for mandarin chinese, which covers diverse areas of knowledge, including the humanities, social sciences, stem (science, technology, engineering, and mathematics), and other areas that are important in daily life. it includes common test questions in subjects like mathematics, physics, and chemistry with answers that are not language or region specific, but also several tasks that are very region-specific, such as chinese driving rules, chinese food culture, and chinese teacher qualifications. the questions in these tasks involve lots of china-related knowledge and can test a model's understanding and adaptability to chinese. in addition, cmmlu also contains tasks that can only expressed in chinese, such as ancient chinese language and chinese literature. the terms and concepts involved in these tasks rely heavily on specific chinese expressions that cannot reasonably be attained through translation. the full list of subjects, the concepts tested in each subject, the number of questions, and the statistics of question and answer lengths are provided in appendix a. ## experiments to provide an overview of existing llms on language understanding for chinese, we evaluate two commercial llms and more than 20 opensource llms of different sizes, language combinations, and training strategies (i.e. either base or sft/rlhf models). we analyse their performance and investigate several factors that could affect llm performance. this indicates that our dataset is more inclined toward examining knowledge related to social, cultural, and regional factors. we name this the next token prediction strategy. a third strategy is to select the answer with the lowest perplexity when concatenated with the question. we compared the different strategies in appendix g, and found that next token prediction to be the most effective method. therefore, for the remainder of the paper, we report results for next token prediction, other than in section 4.2 where we use free generation for part of our analysis, using a regex customized to chatgpt and chatglm responses (see appendix h for details). prompt we introduce each question with the phrase 以下是关于[主题]的单项选择题，请直 接给出正确答案的选项 "here are some multiplechoice questions about [subject], please provide the correct answer directly", and evaluate models in both zero-shot and few-shot settings. for zeroshot evaluation, we present a question with choices directly after the prompt. for few-shot evaluation, we provide up to 5 demonstration examples with answers before the question. the prompt concludes with the phrase 答案是："answer:", as shown in the example in figure 2 . if the context exceeds the model's maximum length with few-shot examples, we dynamically remove the longest examples based on sub-token count. ## conclusion we introduce cmmlu, a groundbreaking benchmark designed to assess multi-task language understanding capabilities in chinese. our experimental findings reveal substantial opportunities for improvement within existing large language models. through extensive analysis, we identify several factors that impact model performance and propose actionable directions for enhancing llms. we are confident that our benchmark dataset and analytical insights will empower researchers to effectively evaluate and design chinese llms.