CLEVA : C hinese Language Models EVA luation Platform (2023)

Abstract: With the continuous emergence of Chinese Large Language Models (LLMs), how to evaluate a model’s capabilities has become an increasingly significant issue. The absence of a comprehensive Chinese benchmark that thoroughly assesses a model’s performance, the unstandardized and incomparable prompting procedure, and the prevalent risk of contamination pose major challenges in the current evaluation of Chinese LLMs. We present CLEVA, a user-friendly platform crafted to holistically evaluate Chinese LLMs. Our platform employs a standardized workflow to assess LLMs’ performance across various dimensions, regularly updating a competitive leaderboard. To alleviate contamination, CLEVA curates a significant proportion of new data and develops a sampling strategy that guarantees a unique subset for each leaderboard round. Empowered by an easy-to-use interface that requires just a few mouse clicks and a model API, users can conduct a thorough evaluation with minimal coding. Large-scale experiments featuring 23 Chinese LLMs have validated CLEVA’s efficacy.

## introduction large language models (llms) have fundamentally revolutionized natural language processing. transformer models with more than 100b parameters have exhibited remarkable generalization ability across diverse tasks without the need for finetuning. the success of @xcite and chatgpt sparked a trend of training chinese llms, with new models launching almost every week @xcite @xcite @xcite . these rapid developments aggravate the need for chinese llm evaluation. ## related work llm evaluation is a rapidly developing field in recent years to delineate the boundary of llm's capability. in english, various systematic evaluation benchmarks have been proposed. for example, big-bench (bench authors, 2023) is the largest collection that covers more than 200 tasks. helm @xcite organizes tasks into core scenarios, which focus on use cases, and targeted evaluation, which aims to better understand models. helm also presents a multi-metric measurement that enables analysis of tradeoffs for each scenario. recently, agieval @xcite is proposed to evaluate llms using challenging human exams. promptbench @xcite , on the other hand, measures the robustness of llms to prompts via adversarial attacks. mt-bench @xcite collects multi-turn questions and presents the chatbot arena platform that treats @xcite as the judge. while cleva shares the same fundamental motivation with helm @xcite , to holistically evaluate language learning models in their original languages, cleva is far from a mere chinese replica of helm. building on the foundation of helm's taxonomy, cleva introduces a range of tasks, with particular emphasis on those unique to chinese, to better assess the capabilities of chinese llms. it offers a new perspective on prompts, providing abundant prompt templates to standardize evaluation and encourage in-depth exploration of models' sensitivity. in terms of metrics, cleva expands into new areas of diversity and privacy for a more comprehensive evaluation. finally, cleva proactively mitigates train-test contamination by collecting a significant amount of new data, creating unique test sets by sampling, and regularly updating the leaderboard. all of these evaluation designs are neatly packaged in a user-friendly platform to facilitate community usage. there is also a lot of progress in evaluating chinese llms @xcite @xcite @xcite . open-compass (contributors, 2023b) and flageval (contributors, 2023a) are two important attempts to evaluate chinese llms. opencompass pools 53 public datasets and uses standard accuracy-like metrics as the only measurement for each dataset. flageval, with a smaller number of datasets and metrics, still needs further expansion to achieve sufficient coverage. compared to previous efforts, cleva offers chinese data from 84 datasets, including 33.98% original queries, while employing the broadest range of metrics to promote holistic evaluation. cleva standardizes prompts ( § 4) and mitigates data contamination issues, pioneering new paths for llm evaluation in general. ## preliminaries to measure the model performance on a task, a relevant test set is constituted from a collection of instances. a test instance will contain multiple input fields (string typically) and a list of references. we then adopt a prompt template, which essentially describes how to assemble the model input, a.k.a, prompt, from input fields @xcite . for example, a chinese paraphrase identification prompt template (and its translation) is: chinese example: "{sentence1}"和"{sentence2}"这两个问题是在 问同一件事情吗？ english translation: are the questions "{sentence1}" and "{sen-tence2}" asking the same thing? finally, all model predictions and the corresponding test instances will be passed into a metric to obtain a numerical value that indicates how well the model performs. following helm @xcite , a metric in this paper is an umbrella for a dimension of measures that share similar purposes. for example, the accuracy metric corresponds to bleu for translation and pass@k for code synthesis. we employ nine metrics, foregrounding metrics beyond accuracy and ensuring a holistic evaluation. ## system design cleva aims to deliver the following two key assets to users who try to evaluate their own llms: ## usage example upon authentication, users are immediately presented with an interactive summary of our evaluation results of 23 llms. users can select from these models, freely exploring the evaluation results from all 9 metrics and 31 tasks. cleva simplifies the evaluation process of new models with minimal coding required. if a user has a model to evaluate, the user only needs a few minutes to finish these three steps: entering the model's api, selecting relevant tasks from 31 choices, and picking desired metrics from 9 options. cleva will autonomously call the user's model, extract the corresponding responses, and compute the final metrics. detailed descriptions and screenshots of cleva are listed in appendix a. figure 2 : the mean win rate of 23 models in 31 tasks. the mean win rate is the probability of a model outperforming a random different model on a random task. we exclude toxicity, privacy, and efficiency metrics as all models excel in the former two, and the latter is often paired with other metrics to deliver meaningful comparisons. since robustness and fairness involve expensive data augmentation, we only evaluate chatgpt and claude-instant. ## evaluation setup. we sample 6.43% of our data to test 23 models that support chinese (see appendix e). as for the cost, for example, it takes roughly 1600 gpu hours (nvidia a100 80g) to evaluate bloomz-176b-mt @xcite . ## conclusion we present cleva, a chinese llm evaluation platform. with the largest scale of chinese instances and broadest metrics, cleva provides a comprehensive benchmark to holistically evaluate chinese llms. cleva standardizes key components, such as prompt templates, to make evaluation comparable. it also proactively mitigates the contamination issue by collecting large-scale new data, sampling for unique test sets, and regularly updating the leaderboard.