Papers
arxiv:2305.08322

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

Published on May 15, 2023
Authors:
,
,
,
,
,
,
,
,
Yao Fu ,
,

Abstract

New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

Community

Hi

Sign up or log in to comment

Models citing this paper 39

Browse 39 models citing this paper

Datasets citing this paper 3

Spaces citing this paper 205

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.