E - EVAL : A Comprehensive C hinese K-12 Education Evaluation Benchmark for Large Language Models (2024)

Abstract: The rapid development of Large Language Models (LLMs) has led to their increasing utilization in Chinese K-12 education. Despite the growing integration of LLMs and education, the absence of a dedicated benchmark for evaluating LLMs within this domain presents a pressing concern. Consequently, there is an urgent need for a comprehensive natural language processing benchmark to precisely assess the capabilities of various LLMs in Chinese K-12 education. In response, we introduce E-EVAL, the first comprehensive evaluation benchmark specifically tailored for Chinese K-12 education. E-EVAL comprises 4,351 multiple-choice questions spanning primary, middle, and high school levels, covering a diverse array of subjects. Through meticulous evaluation, we find that Chinese-dominant models often outperform English-dominant ones, with many exceeding GPT 4.0. However, most struggle with complex subjects like mathematics. Additionally, our analysis indicates that most Chinese-dominant LLMs do not achieve higher scores at the primary school level compared to the middle school level, highlighting the nuanced relationship between proficiency in higher-order and lower-order knowledge domains. Furthermore, experimental results highlight the effectiveness of the Chain of Thought (CoT) technique in scientific subjects and Few-shot prompting in liberal arts. Through E-EVAL, we aim to conduct a rigorous analysis delineating the strengths and limitations of LLMs in educational applications, thereby contributing significantly to the advancement of Chinese K-12 education and LLMs.

## introduction large language models have made significant advancements in the field of natural language processing and artificial intelligence. the evaluation * equal contribution. † corresponding author. in this paper, we introduce e-eval, the first comprehensive evaluation suite focusing on chinese k-12, aimed at evaluating basic models' knowledge and reasoning ability within the context of k-12. e-eval comprises 4,351 multiplechoice questions across primary, middle, and high school stages, as depicted in figure 1 covering 23 subjects including primary school chinese, primary school mathematics, primary school english, primary school science, primary school ethics, middle school chinese, middle school mathemat-our evaluation of multiple open-source and commercial models on e-eval reveals that chinesedominant models outperform english-dominant ones in chinese k-12 performance, with several models even surpassing gpt 4.0. however, performance in complex subjects like mathematics remains subpar. additionally, we observed that almost all advanced chinese models struggle to achieve higher performance at lower educational levels compared to higher ones. this underscores the need for improvement in llms within the chinese k-12 educational domain and highlights the potential value of e-eval as an important evaluation suite. ## experiment here we detail our experiments. we tested 15 advanced models on this benchmark, encompassing various sizes, language orientations, and stages (pretrained or fine-tuned). this in-depth analysis of their performance offers a reliable reference point for future research in this field. ## related work with the continuous advancement of natural language processing technology, benchmarks have expanded to encompass more complex tasks, such as machine translation @xcite and summarization @xcite . the advent of comprehensive benchmarks like glue @xcite and super-glue @xcite heralded a new era. these benchmarks amalgamate various natural language understanding tasks, including textual entailment, sentiment analysis, and question answering, thereby providing a standard for evaluating the holistic performance of models. popular for their ability to assess models on both understanding and generating natural language, these benchmarks have gained prominence. yet, the emergence of llms like bert and gpt shifted the focus towards assessing performance on higher-level, more intricate tasks. such models have even surpassed human-level performance on certain tasks, notably in text summarization @xcite and reading comprehension @xcite . however, some recent work @xcite have demonstrate that llm can perform even better than human or human annotators on some tasks such as summarization, leading to a re-evaluation of the appropriateness of using these benchmarks. to offer a more encompassing evaluation, new benchmarks like mmlu @xcite include a multitude of domains and tasks, ranging from real-world exams to book knowledge, assessing ability in language understanding, common sense reasoning @xcite @xcite , mathematical reasoning @xcite , and code generation @xcite . the big-bench @xcite includes 204 diverse tasks, some of which are deemed beyond the current ability of llms. the helm @xcite benchmark comprises 42 distinct tasks, evaluating llms across seven metrics. with the burgeoning development of chinese large models, an increasing number of chinese benchmarks have emerged. clue @xcite , an influential chinese nlu benchmark, has been widely adopted in the field. additionally, the team has recently introduced superclue @xcite , a benchmark tailored specifically for llms. concurrently, chinese benchmarks akin to mmlu @xcite have surfaced, such as mmcul (zeng, 2023), which emphasizes medicine and education within its four major domains. agieval @xcite focuses on standardized chinese exams like the college entrance exam, while c-eval @xcite encompasses questions across four levels of difficulty from middle school to professional tests. m3ke @xcite gathers 71 tasks from the chinese education examination system, akin to the coverage of c-eval. cmmlu @xcite , designed for the chinese language and cultural context, is a fully localized chinese benchmark. compared to these benchmarks, e-eval distinguishes itself by (1) focusing on the field of k-12 education in china, covering all subjects from primary to high school. (2) including the often-overlooked domain of elementary education. (3) sourcing data from homework and smaller-scale exams, ensuring a high degree of data privacy. ## discussion and conclusion although llms have potential in k-12 education, their accurate assessment remains critical for practical application, and the introduction of the e-eval benchmark, customized for k-12 education in china, provides a more accurate and comprehen-sive evaluation benchmark. although e-eval is not a competitive ranking, it serves as a key tool for tracking the progress of llms in chinese k-12 education. this may pave the way for a wider and more effective utilization of large-scale language models in the field of k-12 education in china. in this work, we obtained the following potentially insightful observations.