Can Large Language Model Comprehend A ncient C hinese? A Preliminary Test on ACLUE (2023)

Abstract: Large language models (LLMs) have demonstrated exceptional language understanding and generation capabilities. However, their ability to comprehend ancient languages, specifically ancient Chinese, remains largely unexplored. To bridge this gap, we introduce ACLUE, an evaluation benchmark designed to assess the language abilities of models in relation to ancient Chinese. ACLUE consists of 15 tasks that cover a range of skills, including phonetic, lexical, syntactic, semantic, inference and knowledge. By evaluating 8 state-of-the-art multilingual and Chinese LLMs, we have observed a significant divergence in their performance between modern Chinese and ancient Chinese. Among the evaluated models, ChatGLM2 demonstrates the highest level of performance, achieving an average accuracy of 37.45%. We have established a leaderboard for communities to assess their models.

## introduction the study of ancient languages provides valuable insights into the past civilizations' thoughts, languages, societies, and histories @xcite @xcite ). ancient china, as one of the oldest civilizations, has left a significant impact on contemporary societies including japan, korea, and vietnam. however, existing research in ancient chinese language processing have primarily focused on specific time periods or genres @xcite @xcite @xcite . typically, the previously proposed models require customized fine-tuning for particular tasks. recently, the significant advancements made in large language models (llms) underscore their remarkable proficiency across a range of tasks, showcasing their potential in performing various tasks without the need for fine-tuning (brown et al., 1 https://github.com/isen-zhang/aclue 2020; @xcite @xcite . these models encapsulate extensive knowledge and sophisticated reasoning capabilities. notably, the emergence of @xcite and chineseoriented llms such as chatglm @xcite , has accentuated their remarkable ability in comprehending and generating modern language. however, due to the lack of ancient language benchmarks, the abilities of llms in handling ancient language remains largely unexplored. we present the ancient chinese language understanding evaluation (aclue), an evaluation benchmark consisting of 15 tasks. these tasks are derived from a combination of manually curated questions from publicly available resources, and automatically generated questions from classical chinese language corpora. the range of questions span from the xia dynasty (2070 bce) to the ming dynasty (1368 ce), covering a broad temporal range. similar to the well-established llm benchmarks such as arc @xcite and mmlu @xcite , aclue adopts multiplechoice question format for all tasks. this ensures simplicity and uniformity in evaluating models, accommodating variations in different training or finetuning procedures and prompting methodologies. in our preliminary experiments, we assessed the performance of 8 advanced llms, where the chinese llm chatglm2 demonstrates the best performance with an average accuracy of 37.4%, slightly surpassing chatgpt. however, considering the baseline accuracy of 25% from random guessing and the average accuracy of around 50% achieved by the same models on contemporary modern chinese benchmarks such as agieval @xcite and cmmlu @xcite , we believe there is still ample room for improvement in the proficiency of existing llms in understanding ancient chinese. ## aclue benchmark aclue consists of 15 tasks that encompassing lexical, syntactic, semantic, inference, and general knowledge of ancient chinese. the details of the tasks are provided in appendix a, where basic statistics can be found in aclue serves as an evaluation suite for llms ability in understanding ancient chinese without task-specific fine-tuning. to ensure fair comparison among different models trained with varying approaches, all tasks are formatted into multiplechoice questions with four choices, of which only one is correct. the task details and dataset construction process are elaborated in this section. ## experiment to provide an overview of the language ability of existing open-sourced llms on ancient chinese, we assess 8 models including 4 multilingual models: chatgpt (openai, 2023), llama @xcite , falcon @xcite , bloomz @xcite , and 4 chinese models: chatglm @xcite , baichuan, foot_3 chatglm2 @xcite , and moss (openlmlab, 2023). details about these models are introduced in appendix c. for models optimized to function as chatbots, such as chatgpt and chatglm, we generate output and use regular expressions to extract the answer key. for other models, we directly obtain the probability of the next tokens after the prompt and selected the one with the highest probability among the answer keys (i.e., 'a','b','c','d'). we employ both zero-shot (do not provide examples) and in-context five-shot (provide few examples) evaluation. an example of evaluation instance is shown in figure 1 . ## related work a lot of research has been conducted on various aspects of ancient chinese language processing, encompassing topics such as ancient chinese to modern chinese translation @xcite , chinese couplets generation @xcite @xcite , classic chinese poem generation @xcite @xcite @xcite , and ancient chinese sentence segmentation @xcite , as well as general language model pre-training @xcite . however, many of these studies focus on specific types or literary formats that were popular during specific time periods. recently, large language models have demonstrated remarkable language understanding and generation capabilities @xcite @xcite . researchers have began to evaluate these llms based on their performance across a wide range of tasks @xcite @xcite . however, the absence of a comprehensive evaluation benchmark poses a challenge in assessing the performance of llms in ancient language understanding. existing ancient chinese evaluation datasets either have a narrow focus on specific tasks, limiting the scope of evaluation, or require model fine-tuning prior to evaluation. in contrast, aclue provides a natural support for evaluation under zero-shot and in-context learning settings, making it more compatible with llms. ## conclusion we propose aclue, the first evaluation benchmark for ancient chinese language understanding. our preliminary evaluation of 8 large language models reveals that, despite their exceptional performance in modern language understanding, they struggle with even basic tasks in ancient chinese. through analysis, we illustrate that the autogenerated questions possess similar difficulty levels to those found in actual school tests.