Z hu J iu-Knowledge: A Fairer Platform for Evaluating Multiple Knowledge Types in Large Language Models (2024)

Abstract: The swift advancement in large language models (LLMs) has heightened the importance of model evaluations. LLMs have acquired a substantial amount of knowledge, and evaluating the knowledge of these LLMs is crucial. To address this, we introduce the ZhuJiu-Knowledge benchmark which carefully considers the following factors: (1) For knowledge scope, we concentrate on three domains: commonsense knowledge, world knowledge, language knowledge, which comes from ATOMIC, Conceptnet, Wikidata, and Wordnet. (2) For data construction, to prevent data contamination, we utilize knowledge derived from corpora and knowledge graphs to formulate novel questions which are ensured not to appear in the training corpus. A multitude of prompts is purposefully devised to mitigate the impact of prompt design on evaluation and to further analyze the LLMs’ sensitivity to various prompts. (3) For evaluation criteria, we propose a novel voting methodology for assessing generative text, aligning the model’s evaluation with human preferences to reduce biases inherent in individual model assessments. We evaluate 14 current mainstream LLMs and conduct a comprehensive discussion and analysis of their results. The ZhuJiu-Knowledge benchmark and open-participation leaderboard are publicly released at http://zhujiu-knowledge.top and we also provide a demo video at https://youtu.be/QJp4qlEHVH8.

## introduction the unprecedented performance of llms, such as gpt4 @xcite and llama2 @xcite , has garnered significant attention and made their evaluation a focal point as the field progresses @xcite ). ## †corresponding author llms have acquired a substantial amount of knowledge, and evaluating the knowledge of these llms is crucial. existing efforts have focused on evaluating the knowledge stored within the model (e.g., @xcite . however, these works still face several challenges. constructing a reasonable benchmark for evaluating knowledge involves careful consideration of several key factors: (1) knowledge scope. most evaluations are limited to world knowledge related to entities and relations, lacking assessments of commonsense knowledge and language knowledge @xcite . this limitation arises because these two types of knowledge are often expressed in the form of events or sentences. currently, making unbiased evaluations of the generated sentences is still a difficult problem for llms. (2) data constructinon. most evaluation platforms have limitations on assessing data. firstly, the evaluated data usually are leaked and embedded in the target llms in the process of pre-training or sft, after such data are released publicly. as a result, making an evaluation on such data would be biased @xcite . secondly, existing knowledge probing strategies usually relied on the given prompts heavily. existing methods only used just one prompt for each piece of knowledge. if the target llm does not understand the given prompt well, it will not obtain better results @xcite . (3) evaluation criteria. evaluating knowledge using multiplechoice questions and true or false questions may introduce certain biases obviously. assessing generated text with qa questions requires a reasonable evaluation metric for the generated content. traditional evaluation criteria such as glue @xcite , rouge @xcite , and recall have inherent limitations, often leading to a gap between evaluation results and users' subjective experiences. while manual evaluation is highly reliable, it is time-consuming and labor-intensive @xcite . therefore, designing a rational evaluation metric is also a critically important issue. this paper constructs zhujiu-knowledge, a fairer platform for evaluating multiple knowledge types in llms, which is designed to assess llms' capabilities in commonsense knowledge, world knowledge, and language knowledge. during the construction process, we fully consider the aforementioned three factors. for knowledge scope, we choose to evaluate commonsense knowledge, world knowledge, and language knowledge for the following reasons: (i) commonsense knowledge is often implicitly embedded in the texts, which presents a challenge for llms to acquire substantially from textual corpora. a proficient understanding of commonsense is fundamental for llms to generate reasonable responses @xcite . (ii) world knowledge plays a critical role in llms' performance. based on memory and manipulation for world knowledge, llms can provide more accurate and relevant responses to user questions @xcite . (iii) a deep understanding of grammar, semantics, and pragmatics enables llms to grasp language nuances and process text effectively @xcite . for data construction, we simultaneously address data contamination @xcite and prompt sensitivity issues. (i) to address data contamination, some studies adopt machine-unreadable @xcite ) data or constantly evolving data. considering the pressing demand for high-quality data in the training process of llms, it can be foreseen that these data will also be trained by llms in the near future. to overcome this limitation, the paper proposes to leverage knowledge from existing knowledge graphs and corpora to construct evaluation questions. the advantage of this method is to allow for the automatic generation of questions tailored to the targeted knowledge domains. (ii) to better assess llms' prompt sensitivity, we propose designing multiple prompts for each piece of knowledge. for each knowledge, we design multiple prompts to investigate whether an llm can consistently generate correct answers when given varying prompts for evaluation criteria, conventional methods typically relied on traditional evaluation metrics or more advanced models. however, these methods often fail to fully capture llms' comprehensive abilities and users' subjective experiences @xcite , while purely manual evaluations are labor-intensive. to bridge the gap between objective metrics and subjective preferences, we propose an innovative approach integrating human-computer collaboration, which employs a vote-like evaluation strategy. multiple llms are employed to evaluate the given answers, where some pivot llms are selected as the evaluation models because they have higher correlations with human evaluations. the final result is obtained through voting. we also release an online evaluation platform that supports multiple functions including visual-izations of evaluation results and submission of evaluation models, etc. moreover, we evaluate 14 publicly available llms. based on the experimental results, we obtain some intriguing observations and summarize them in section 4. in summary, the contributions of this paper are as follows: ## zhujiu-knowledge benchmark as stated above, the zhujiu-knowledge benchmark conducts precise evaluations of models across three knowledge types. this section provides a detailed introduction to the zhujiu-knowledge benchmark covering the knowledge selection, task design, data construction, and evaluation method. we also detailed the process of problem construction in appendix a. the evaluation framework is shown in figure 1 . ## platform we have developed an online platform that offers a diverse array of services to the community, as outlined below: evaluation process and questions we provide a detailed introduction for our evaluation process in figure 5 and present a subset of evaluation questions in figure 7 . ## conclusion and future work in this work, we presented zhujiu-knowledge, a fairer benchmark for evaluating multiple knowledge types of llms. zhujiu-knowledge extends the current knowledge evaluation scope to commonsense knowledge, language knowledge, and world knowledge. we introduce a novel data construction methodology that mitigates the risks of data contamination and prompt sensitivity and proposes a novel voting methodology to evaluate generative text. our comprehensive evaluation of 14 mainstream llms provides significant insights into their performance, revealing the strengths and weaknesses of each model of various knowledge abilities. finally, we provide a comprehensive knowledge evaluation platform for llms in the zhujiu-knowledge. in the future, our objectives include: (1) broadening the scope of the zhujiu-knowledge benchmark to cover a wider array of knowledge assessment dimensions; and (2) enhancing our evaluation platform by integrating more features and improving the user interface, which will facilitate more efficient and user-friendly assessments.