Paper: CMMLU: Measuring massive multitask language understanding in Chinese (ICLR-2024)

Reviewer comment (Evaluation):
All questions are formatted as multiple-choice with 4 choices. This may make it difficult to comprehensively test the performance of LLMs.

Author response:
We acknowledge that the multiple-choice format of testing has its limitations. However, this format offers a standardized and accessible evaluation method, as evidenced by its prevalence in popular LLM evaluation frameworks and leaderboards, such as the one hosted by Hugging Face (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), of which around half is  multiple-choice tasks. Additionally, multiple-choice questions can be converted to a perplexity comparison of the different choices, which is the dominant approach for pre-trained LLMs evaluation, given that this is how instruction fine-tuning is carried out. The decision to use a multiple-choice format was made after careful consideration.

Currently, there is no universally-recognized benchmark capable of comprehensively evaluating the performance of LLMs. Most existing leaderboards use a combination of multiple datasets to provide a holistic assessment of LLMs. In this context, CMMLU is not intended to provide a holistic evaluation of LLMs, but rather be one of several benchmarks in a comprehensive evaluation strategy.

Reviewer comment (Experiments):
The experimental methodology of most experiments is language-agnostic. It only simply compares general LLMs vs Chinese LLMs and non-Chinese-specific tasks vs Chinese-specific tasks. More experiments should be deeply combined with the Chinese cultural and Chinese linguistic characteristics.

Author response:
Your suggestion to incorporate a more in-depth exploration of Chinese cultural and linguistic characteristics is well-received. To avoid misunderstanding, we want to first clarify that all data in CMMLU is in Mandarin Chinese. Our initial focus was to establish a broad comparison between multilingual and Chinese-specific LLMs using CMMLU. This approach was intended to set a baseline for future, more detailed studies. We agree that delving deeper into the nuances of the Chinese language and culture will further enrich our understanding of LLM capabilities, but this is beyond the scope of this work.

Reviewer comment (Evaluation):
This work needs to analyze the correlation between the performance reported by CMMLU and the real performance measured in the representative downstream NLU tasks. Otherwise, it is difficult to determine whether CMMLU can reflect the NLU performance of a LLM.

Author response:
We have included an analysis of the correlation between CMMLU performance and five prominent downstream English benchmarks in Appendix Section I. Our findings indicate that performance on CMMLU has a strong correlation with results on four of these benchmarks (with Pearson’s Correlation r > 0.94), which span areas such as mathematics, commonsense reasoning, and coding. The slight exception is the PIQA task (with Pearson’s Correlation still high at r = 0.88), where the task relevance is somewhat reduced due to most models achieving high scores (>80%) on this task.

Reviewer comment (Reproducibility):
What form will this dataset be released in the future?

Author response:
The CMMLU dataset will be released under a 'Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License' to facilitate open-source research and ensure accessibility for non-commercial use.

Reviewer comment (Experiments):
Besides the Chinese-specific tasks and data source, is there any other Chinese-specific feature that has been considered in CMMLU?

Author response:
No. Our focus is on the dataset tasks and sources, which we believe adequately address the knowledge-based evaluation needs of Chinese LLMs, consistent with MMLU for English and IndoMMLU for languages of Indonesia. We are open to exploring additional features in future iterations of CMMLU based on community feedback and evolving research needs.

Reviewer comment (Evaluation):
However, an LLM passing professional exams doesn't necessarily indicate its true capabilities, raising concerns about construct validity.

Author response:
To comprehensively evaluate the capabilities of an LLM, evaluation needs to encompass aspects including knowledge, commonsense reasoning, fairness, and generative capabilities. This paper focuses on evaluating knowledge-related aspects in the Chinese language, as a component of a broader evaluation, consistent with the use of MMLU in English.

School exam questions have been extensively employed as a proxy for evaluating knowledge in LLMs. Examples include the use of the English MMLU dataset in the official release documentation/technical reports for GPT-4, LLaMA-2, Falcon, and various other LLMs, and the HuggingFace Open LLM leaderboard (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), the most popular LLM leaderboard today.

We believe that developing such a benchmark for Mandarin Chinese is essential for more inclusive evaluation of LLMs beyond the English language.

Reviewer comment (Reproducibility):
The crisis of research replication based on language models was severe, and the evaluation methods had limitations.

Author response:
We absolutely share your concerns regarding the importance of reproducibility in research, but do not agree that this is a limitation of the dataset. To ensure transparency, we have meticulously documented the experimental settings in Section 4, including the number of shots for few-shot learning, and the exact prompt used. As such, all experiment results in the paper can be reproduced. Additionally, a number of subsequent studies have successfully replicated our results based on the arXiv preprint version of this paper, and we've also made our code completely public. In adherence with the policy of anonymity, we refrain from specifically referencing these papers here as they cite the arXiv preprint, compromising the anonymity of this paper, but we would be happy to provide the links after the review period.

Additionally, the issue of LLM research replication is an issue with models and model training rather than evaluation datasets, which is the focus of this paper.

Reviewer comment (Evaluation):
Assessing the political biases inherent in the language models presented in the benchmark was challenging and required naturalistic observation.

Author response:
Our paper focuses on knowledge evaluation, not assessing the political biases of LLMs, which is an orthogonal dimension of LLM evaluation, as stated above.

Reviewer comment (Evaluation):
The outputs of LLMs were uncertain. Even a minor change in a prompt could lead to variations in the output. In light of this benchmarking paper, how did the authors perceive this issue? How should the benchmark address the inherent unpredictability of LLMs?

Author response:
This is a more general issue with any evaluation of LLMs, and not specific to our benchmark. We have strived to mitigate this by employing simple and commonly-used prompts, to minimize variance in model outputs. This approach leads to fairer comparison of different models. Also of note is that we tested several (i.e., 3) widely-used prompts for multiple choice questions on more than 5 models. We find that the standard deviation in the average score is <1%.

Reviewer comment (Evaluation):
Typically, the Chain-of-Thought method had proven successful on LLMs. However, this paper concluded that the Chain-of-Thought was not effective in enhancing model performance, which contradicted the feedback received from practical use of LLMs with the Chain-of-Thought. A more detailed analysis and explanation were requested.

Author response:
The effectiveness of Chain-of-Thought (COT) in LLMs primarily comes about in tasks requiring reasoning. For memory-intensive tasks (e.g., recalling historical facts), COT can not only be less effective but potentially impede performance.

In the CMMLU benchmark, the majority of tasks are knowledge-intensive rather than reasoning-based. This explains why COT does not enhance performance in our case. Furthermore, as discussed in Section 4.2 of our paper, issues like regex matching failures can further diminish effectiveness.

Reviewer comment (Experiments):
LLMs demonstrated strong In-Context Learning capabilities. It would be worth exploring whether adding appropriate knowledge to the prompt could answer benchmark questions to validate the benchmark's effectiveness.

Author response:
In-context learning is indeed a vital aspect of LLM evaluation, often utilized to convey task requirements to the model. In our benchmark, which focuses on knowledge-intensive tasks, in-context learning primarily serves to clarify task objectives rather than contribute additional instruction on how to answer the question. This is also the standard approach to in-context learning in evaluation work, i.e. to provide a few example questions and answers before asking the question. In this paper, we performed substantial analysis of this kind of in-context learning (Section 4).

Your suggestion to explore the addition of knowledge to prompts to validate the benchmark's effectiveness is valuable. Our focus is on assessing the inherent knowledge of LLMs, as acquired during pre-training and fine-tuning. Incorporating external knowledge would not align with this objective, as it would not accurately reflect the model's intrinsic capabilities. However, we acknowledge that exploring in-context learning (with external knowledge) could be a direction for future research. Reading comprehension (RC) tasks are well represented in existing RC datasets and could be used for this.

Reviewer comment (Evaluation):
It was known that LLMs would respond cautiously to safety questions when posed in English. However, when asked in less common languages, they might provide bolder answers, potentially bypassing restrictions. Did the CMMLU safety benchmark consider addressing this phenomenon?

Author response:
Once again, CMMLU is focused on knowledge evaluation, consistent with MMLU and IndoMMLU. Safety considerations are a vital consideration in the deployment of any LLM, but not part of knowledge evaluation.

Reviewer comment (Experiments):
How did the authors ensure that the proposed test benchmark was free from data contamination?

Author response:
As mentioned in Section 3 “Data Collection”, more than 80% of our data was crawled from PDFs (after OCR), reducing the possibility of it occurring in LLM training data. Indeed, our specific efforts to mitigate the effects of data contamination are acknowledged by Reviewer ZnNq (See the strengths part). Moreover, the low performance for most subjects suggests that there is little data contamination in LLM pre-training.

Reviewer comment (Experiments):
A human baseline is lacking for the benchmark. It'd be great to see what level of accuracy human can get on the benchmark.

Author response:
Thank you for your suggestion on incorporating a human baseline. Given the wide range of knowledge areas covered in this benchmark, including STEM and social science, establishing a comprehensive human baseline is challenging. The broad knowledge spectrum makes it difficult for any single individual to provide a meaningful baseline. Moreover, recruiting subject matter experts for each domain, given the extensive scope of the benchmark, is not feasible. 

However, we appreciate the need to provide some form of human comparison. To address this, we propose using statistical data from exams as a proxy for human performance. In China, most school exams require a minimum passing score of 60%, drive licence exams require a score of 90%, etc., suggesting that a competent human test-taker would likely achieve at least that accuracy rate in each subject area. We can supplement this with additional statistics on average scores for various levels of tests to provide a more nuanced view of human performance, although this will be at an aggregate level from educational authorities

We believe this approach, while not perfect, offers a practical and informative way to include a human baseline in our paper. It would provide readers with a point of reference to better understand the performance of LLMs in relation to human capabilities.

We are open to further suggestions and would be grateful for your opinion on this proposed method.

Reviewer comment (Experiments):
In page 1, "numerous tasks within CMMLU have answers speciﬁc to China, which may not be universally applicable or considered correct in other regions or languages.". Do you think it would be a good idea to have questions **with answers that are not generally agreed upon worldwide** in the datasets? How many samples of this kind are there in the benchmark?

Author response:
Our stance is that a language-specific benchmark should reflect the knowledge and perspectives relevant to its target audience. For contentious historical or political topics, aligning with regional consensus is in line with the benchmark's purpose.

We first analyzed the different subject areas, and concluded that such questions will only appear in education, history, politics related subjects.. 

Through careful selection, we identified 7 subjects which may contain such questions:: Chinese history, world history, high school politics, world religions, journalism, legal and moral basis, and Marxist theory. The total number of questions across these subjects is 1362. We then sampled 10%=136 questions to do human annotation, sourcing an expert with a Master's degree in International Relations, currently researching Chinese history, world history, and politics. Their role was to identify questions whose answers might not be universally accepted or agreed upon globally.

This annotation process revealed that 19 questions within our sample had answers potentially subject to regional or cultural differences in interpretation. Based on this finding, we estimate that approximately 190 questions in our dataset (roughly 2% in total across the 7 subjects) might have answers that are not universally agreed upon. Given this small proportion, we believe that the presence of such questions will not significantly impact the overall evaluation results of our study

We can add this analysis to the paper if you consider it will enhance the content.

Reviewer comment (Evaluation):
When you evaluate open-source models, have you seen "emergent ability" in terms of model's size? More precisely, are there some tasks that can only be solved by a large model? If so, then what are the difficulty distributions of those tasks?

Author response:
Recent studies such as "Are Emergent Abilities in Large Language Models just In-Context Learning?" (https://arxiv.org/abs/2309.01809) and “Are Emergent Abilities of Large Language Models a Mirage?” (https://arxiv.org/abs/2304.15004) have raised critical concerns about “emergent abilities”, suggesting that what had previously been identified as emergent abilities in large language models may merely have been the consequence of discretization effects in evaluation metrics, in-context learning, or the effects of memorability, data leakage, and quality. In our analysis, we have observed findings that align with this perspective.

We assessed the concept of 'emergent ability' using the LLaMA-2 model family, and added a section to the Appendix (Section D) to discuss this. Our analysis indicates that while larger models generally show improved performance, it's both overall smooth, but also not consistent across all subjects. In fact, it’s only in a small minority of subjects like college education and human sexuality, that performance jumps between smaller (7B, 13B) and larger (70B) models. While this study is limited to LLaMA-2 and there is further analysis that could be done, these results are not supportive of the hypothesis of emergent abilities.


We are very appreciate of your insightful reviews, and hope these new analyses answer your questions.