Paper: KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
Benchmark on emerging corpora is a great idea and it is quite encouraging to see the authors promised to refresh the benchmark regularly. However, it is not clear how to maintain such benchmark in the long term.

Author response:
As stated in Section 2 of our paper, we are committed to updating our evaluation for at least six seasons. After this, we will further extend tasks to introduce new versions of KoLA. The data for every new season will be up-to-date to ensure the test data is not leaked, maintaining the relative fairness of the evaluation. For the ended seasons, we do not plan to add them to the known set immediately. Instead, they will be publicly released after the completion of six seasons, and a retrospective analysis of all past seasons will be conducted. The primary workload for running a new season includes collecting and annotating new data, constructing a new list of models, and publishing evaluation results. To host these tasks, we have established a maintenance team and carefully controlled costs. This includes adding a large amount of automated pre-annotation (as described in Appendix B) and controlling the size of the evaluation set. As mentioned in our discussion with reviewer 1m8m, we expect the cost of every season will be under USD 2,000. Therefore, we are confident to fulfill the committed maintenance and we also welcome contributions from the community.

Reviewer comment (Novelty):
It is not clear why we need another new LLM benchmark. Given all different benchmarks available publicly, I am not convinced KoLA is a must-have addition.

Author response:
We believe KoLA is unique in two key aspects. (1) As described in Section 1, rather than assessing the breadth of model capabilities like most existing benchmarks (such as MMLU), KoLA focuses more on analyzing the deep interconnections of model knowledge capabilities through carefully-designed multi-level testing. (2) Besides combining some existing tasks, KoLA maintains a unique evolving mechanism, aimed at minimizing data leakage. Recent studies [4,5,6] have widely revealed a serious issue of 'test set contamination' in the evaluation of many LLMs with existing benchmarks, where the results may not genuinely reflect the actual capabilities of the models. Therefore, we believe KoLA holds a distinctive value in the current landscape.

Ref:

[4] Wei T, Zhao L, Zhang L, et al. Skywork: A More Open Bilingual Foundation Model[J]. arXiv preprint arXiv:2310.19341, 2023.

[5] Yang S, Chiang W L, Zheng L, et al. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples[J]. arXiv preprint arXiv:2311.04850, 2023.

[6] Zhou K, Zhu Y, Chen Z, et al. Don't Make Your LLM an Evaluation Benchmark Cheater[J]. arXiv preprint arXiv:2311.01964, 2023.

Reviewer comment (Evaluation):
It is not clear why the standardized overall scoring can give better idea than simple ranking.

Author response:
Our considerations about employing standardized scoring are as follows: Standardized scores have become the mainstream choice in fields like educational assessment and intelligence testing [7], with systems like the SAT, GRE, and Wechsler Intelligence Scales (IQ Test) adopting them. Compared to absolute values, using standard scores in the multi-task evaluation of LLMs offers the following benefits:

1. Standardized scores enable fair comparisons across different seasons and datasets. The varying difficulty of tasks and the different sensitivities of metrics mean that the absolute numerical performance of models is incomparable. For example, if there is a dataset where models typically perform at 80, and another where the usual performance is 40, directly comparing a model's performance scores on these two datasets cannot reflect whether the model is better at one task over the other. However, comparing standardized scores derived from relative performance can yield conclusive insights.
2. For overall rankings, standardized scores can negate the bias caused by specific tasks. For instance, without standardized scoring, ChatGLM (6B) would rank much higher in overall rankings solely due to its high performance in the 2-2 task, surpassing its improved version ChatGLM2-32k.

We acknowledge that the inclusion of new models indeed results in variations in the overall standardized scores. We believe that considering the benefits mentioned above, such variations are acceptable. One of the objectives of KoLA is to provide a reference for users in selecting models. We trust that if new models demonstrate superior capabilities, the relative downscaling of other models' evaluations is beneficial for users.

Ref:

[7] Haladyna T M, Nolen S B, Haas N S. Raising standardized achievement test scores and the origins of test score pollution[J]. Educational Researcher, 1991, 20(5): 2-7.

Reviewer comment (Writing):
The paper identified four knowledge capabilities. It is clear on knowledge memorization and knowledge creation. However, it is vague to distinguish knowledge understanding and knowledge application. Take knowledge understanding as an example, would reading comprehension a task of knowledge understanding or knowledge attention? Why knowledge understanding only has extraction tasks?

Author response:
This is a valuable question, and we are pleased to discuss it with you. 

In Bloom's Cognitive Taxonomy [1], *Understanding* can be described as ”Determining the meaning of instructional messages“, typically involving actions like “Explain, Summarize.” *Applying* is described as “Carrying out or using a procedure in a given situation”, generally including actions such as 'Implement, Execute’. Indeed, since the revision of Bloom's theory [2], there has been considerable debate about the demarcation between these two cognitive levels. 

Despite ongoing discussions, one consensus principle to differentiate them is whether it involves information and abilities beyond the given content [3]. Consequently, we categorize tasks involving information extraction (which ensures that the output information is faithfully extracted from the input text) as Understanding. It is worth noting that the Knowledge Understanding (KU) level is not limited to information extraction; it also includes the ability of conceptual abstraction, as seen in our investigated task COPEN. Under this criterion, if a reading comprehension question requires only knowledge from the text, we would classify it as a KU task. Once it involves more complex background knowledge and reasoning, it might lean more towards a Knowledge Application (KA) task.

Ref:

[1] Krathwohl D R. A revision of Bloom's taxonomy: An overview[J]. Theory into practice, 2002, 41(4): 212-218.

[2] Forehand M. Bloom's taxonomy: Original and revised[J]. Emerging perspectives on learning, teaching, and technology, 2005, 8: 41-44.

[3] Wilson L O. Anderson and Krathwohl Bloom’s taxonomy revised understanding the new version of Bloom’s taxonomy[J]. The Second Principle, 2016, 1(1): 1-8.

Reviewer comment (Novelty):
How much does the new KoLA benchmark differ from existing LLM benchmark? There are so many available benchmark published, each focusing on one/multiple capabilities of the model. Why do we need KoLA? What if we combined existing ones?

Author response:
We believe KoLA is unique in two key aspects. (1) As described in Section 1, rather than assessing the breadth of model capabilities like most existing benchmarks (such as MMLU), KoLA focuses more on analyzing the deep interconnections of model knowledge capabilities through carefully-designed multi-level testing. (2) Besides combining some existing tasks, KoLA maintains a unique evolving mechanism, aimed at minimizing data leakage. Recent studies [4,5,6] have widely revealed a serious issue of 'test set contamination' in the evaluation of many LLMs with existing benchmarks, where the results may not genuinely reflect the actual capabilities of the models. Therefore, we believe KoLA holds a distinctive value in the current landscape.

Ref:

[4] Wei T, Zhao L, Zhang L, et al. Skywork: A More Open Bilingual Foundation Model[J]. arXiv preprint arXiv:2310.19341, 2023.

[5] Yang S, Chiang W L, Zheng L, et al. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples[J]. arXiv preprint arXiv:2311.04850, 2023.

[6] Zhou K, Zhu Y, Chen Z, et al. Don't Make Your LLM an Evaluation Benchmark Cheater[J]. arXiv preprint arXiv:2311.01964, 2023.

Reviewer comment (Evaluation):
It is not clear the contribution/motivation with standardized overall scoring. As the score would depend on the evaluated models, it will change a lot as more evaluated models would be added in. Also why this is better than simple ranking?

Author response:
Our considerations about employing standardized scoring are as follows: Standardized scores have become the mainstream choice in fields like educational assessment and intelligence testing [7], with systems like the SAT, GRE, and Wechsler Intelligence Scales (IQ Test) adopting them. Compared to absolute values, using standard scores in the multi-task evaluation of LLMs offers the following benefits:

1. Standardized scores enable fair comparisons across different seasons and datasets. The varying difficulty of tasks and the different sensitivities of metrics mean that the absolute numerical performance of models is incomparable. For example, if there is a dataset where models typically perform at 80, and another where the usual performance is 40, directly comparing a model's performance scores on these two datasets cannot reflect whether the model is better at one task over the other. However, comparing standardized scores derived from relative performance can yield conclusive insights.
2. For overall rankings, standardized scores can negate the bias caused by specific tasks. For instance, without standardized scoring, ChatGLM (6B) would rank much higher in overall rankings solely due to its high performance in the 2-2 task, surpassing its improved version ChatGLM2-32k.

We acknowledge that the inclusion of new models indeed results in variations in the overall standardized scores. We believe that considering the benefits mentioned above, such variations are acceptable. One of the objectives of KoLA is to provide a reference for users in selecting models. We trust that if new models demonstrate superior capabilities, the relative downscaling of other models' evaluations is beneficial for users.

Ref:

[7] Haladyna T M, Nolen S B, Haas N S. Raising standardized achievement test scores and the origins of test score pollution[J]. Educational Researcher, 1991, 20(5): 2-7.

Reviewer comment (Evaluation):
The knowledge-wise strength of Rouge-L used in Eq. 3 doesn't look to be strong enough in capturing the knowledge association especially when T is a free-generation result, maybe replacing the measurement with another model (e.g. a entailment model) would be better? (w/ additional computational cost)

Author response:
Employing models for automatic evaluation of knowledge aspects is indeed a valuable suggestion and represents a promising direction. In the design phase of KoLA, we considered such an approach. However, we ultimately chose model-free evaluation metrics to potentially prevent the evaluation model from favoring certain types of models. In the early experiments, we tried various metrics, including some model-based metrics like BertScore, BLEU, etc., and test whether their results align with human evaluations. Bases on these, we finally choose Rouge-L. We plan to continue exploring the integration of your suggested model-based evaluation methods in subsequent seasons. Thank you once again for your insightful recommendation!

Reviewer comment (Presentation):
The caption of the figures should include necessary details to understand them if the space allows (e.g. Figure 4).

Author response:
We appreciate the reviewer's suggestion to elaborate on this detail. Although we mentioned this part in the first paragraph of Appendix B.4, we overlooked emphasizing this information in the caption of Figure 4. Following your reminder, we have *enriched the captions of all the tables and figures in the paper*, further enhancing the readability of the article.