AC - EVAL : Evaluating A ncient C hinese Language Understanding in Large Language Models (2024)

Abstract: Given the importance of ancient Chinese in capturing the essence of rich historical and cultural heritage, the rapid advancements in Large Language Models (LLMs) necessitate benchmarks that can effectively evaluate their understanding of ancient contexts. To meet this need, we present AC-EVAL, an innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese. AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension: general historical knowledge, short text understanding, and long text comprehension. The benchmark comprises 13 tasks, spanning historical facts, geography, social customs, art, philosophy, classical poetry and prose, providing a comprehensive assessment framework. Our extensive evaluation of top-performing LLMs, tailored for both English and Chinese, reveals a substantial potential for enhancing ancient text comprehension. By highlighting the strengths and weaknesses of LLMs, AC-EVAL aims to promote their development and application forward in the realms of ancient Chinese language education and scholarly research.

## introduction the advent of large language models (llms) has significantly impacted natural language processing (nlp), highlighting their importance in understanding and generating human languages @xcite @xcite . with the rise of chinese as a major global language, there has been a surge in chinese-specific llms @xcite @xcite . ancient chinese, a crucial part of the chinese language, records a rich historical and cultural heritage, and has garnered considerable attention from computational linguists @xcite . llms present significant opportunities for enhancing the pedagogy of chinese literary education through convenient text analysis and comprehension. therefore, assessing the ancient chinese comprehension capabilities of llms holds significant importance. initially, benchmarks for llms primarily targeted the assessment of english language understanding, exemplified by mmlu @xcite , @xcite and helm @xcite . subsequently, several benchmarks focusing on chinese, such as c-eval @xcite , cmmlu @xcite , and super-clue @xcite , were introduced. these benchmarks aim to evaluate the reasoning performance of llms across a broad spectrum of fields including stem, social sciences, and humanities. however, these benchmarks tend to lean towards modern chinese comprehension. while some include tasks related to chinese language, literature and history, they are often relegated to minor categories, insufficient for a comprehensive coverage of ancient chinese knowledge and language assessment. existing benchmarks for ancient chinese understanding, such as cclue foot_0 and wyweb @xcite , cover various aspects but primarily focus on linguistic feature analysis, frequently overlooking the assessment of historical knowledge hidden in literature. furthermore, the diversity in format across these datasets, tailored for specific tasks rather than providing a unified assessment framework, complicates the evaluation of llms, presenting challenges in conducting uniform assessments. to bridge this gap, we propose ac-eval (as illustrated in figure 1 ), a benchmark meticulously designed for a comprehensive evaluation of llms' proficiency in ancient chinese language understanding and historical knowledge. ac-eval comprises 3,245 multiple-choice questions, spanning three distinct dimensions and thirteen subjects, covering historical periods from the pre-qin to the qing dynasty. these tasks, which progressively increase in difficulty, are categorized into general historical knowledge, short text understanding, and long text understanding. the general historical knowledge tasks address a diverse range of contents, including but not limited to, ancient historical facts, geography, social customs, art, religion and philosophy. short text understanding covers lexical semantics and pragmatics, allusions and idioms, sentence translations, and event extraction. long text understanding tasks focus on long text pauses, classical prose summarization and analysis, and the appreciation of themes, emotions and styles in poetry. in our evaluation of llms on the ac-eval benchmark across answer-only (ao) and chainof-thought (cot) settings in zero-and few-shot scenarios, only ernie-bot 4.0 and glm-4 with accuracies over 70%. results reveal significant improvement potential, especially in long text comprehension. our analysis shows that chinese llms outperform english ones in ancient chinese. this distinction underscores the unique challenge that ancient chinese as a low-resource area for models like gpt-4, despite their commendable performance on other chinese benchmarks. moreover, the broad range of knowledge required in our tasks reveals that llms encounter difficulties in grasping underlying rules, affecting few-shot learning outcomes. interestingly, zero-shot cot shows an advantage in larger models, underscoring the value of reasoning steps for complex tasks. through the ac-eval benchmark, our goal is to provide a multidimensional evaluation tool, highlighting potential improvement areas to advance the development of llms in the understanding and education of ancient chinese. ## results in this section, we explore the comparative performance of various models under four distinct settings: zero-shot ao as discussed in section 5.1, few-shot ao in section 5.2, zero-and few-shot cot in section 5.3. ## conclusion we introduce ac-eval, a benchmark designed to evaluate llms' proficiency in ancient chinese, addressing a gap by covering historical knowledge and language understanding extensively. our experiments reveal significant improvement areas for existing llms. we identify critical factors influencing llm performance and suggest practical directions for enhancing these models. ac-eval aims to advance llm application in ancient chinese education, offering a valuable tool for assessing and developing chinese llms.