WYWEB : A NLP Evaluation Benchmark For Classical C hinese (2023)

Abstract: To fully evaluate the overall performance of different NLP models in a given domain, many evaluation benchmarks are proposed, such as GLUE, SuperGLUE and CLUE. The field of natural language understanding has traditionally focused on benchmarks for various tasks in languages such as Chinese, English, and multilingual, however, there has been a lack of attention given to the area of classical Chinese, also known as "wen yan wen (文言文)", which has a rich history spanning thousands of years and holds significant cultural and academic value. For the prosperity of the NLP community, in this paper, we introduce the WYWEB evaluation benchmark, which consists of nine NLP tasks in classical Chinese, implementing sentence classification, sequence labeling, reading comprehension, and machine translation. We evaluate the existing pre-trained language models, which are all struggling with this benchmark. We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on classical Chinese NLU. The github repository is https://github.com/baudzhou/WYWEB .

## introduction classical chinese, as a written form of the chinese language, had been widely used in the confucian cultural circle, including china, japan, korea, @xcite @xcite @xcite . as far as we know, there are about 400 million words, 3 million ancient articles have been passed down, covering literature, art, history, philosophy, etc, half of which are of great value @xcite . however, in recent centuries, the use of classical chinese has been gradually phased out and replaced by modern languages, resulting in increasing difficulty in comprehending it. therefore, it is necessary to introduce efficient nlp technology to process, understand, and research such literature. while pre-trained language models such as bert @xcite and bert-like models @xcite @xcite @xcite @xcite have shown remarkable performance on english nlp benchmarks, including glue @xcite and super-glue @xcite , there are also many efforts @xcite @xcite in chinese nlp community, achieving significant improvement on modern chinese nlp benchmark @xcite @xcite . however, since classical chinese differs from modern chinese in writing and grammar, these benchmarks can not be applied well to the studies in the classical chinese domain. in order to better adapt to the understanding of classical chinese, many tasks and datasets are required to be redesigned, such as sequence labeling and sentence pair similarity. meanwhile, due to the performance of the model being closely related to the pre-training corpus @xcite , such as scale, language, domain, etc., the existing language models pre-trained on modern chinese corpus can not adapt well to classical chinese tasks. considering that previous studies @xcite @xcite for classical chinese have typically evaluated models on few or different nlu tasks, it is difficult to compare the performances of these models. to facilitate such research in classical chinese, it is necessary to design a standard classical chinese nlp evaluation benchmark. in this paper, we introduce wyweb (wen yan wen evaluation benchmark), which will be open, and continually developed as much as we can. to evaluate the performance of the models of classical chinese language representation, we create and refine nine tasks for different aspects of language understanding. specifically, considering the importance of breaks and pauses in sentences for the comprehension of classical chinese, for sequence labeling, we design two novel tasks, including punctuation punc and named entity recognition glner, to evaluate word separation capability of pre-trained language models. for sentence classification, we design three novel tasks, including text category classification gjc, written time classification tlc, and emotion classification of poems task fspc. for reading comprehension, we create a multiple choice task named wywrc. and on the other hand, the assessment of the natural language comprehension capability of a model through previous reading comprehension tasks is challenged by the extensive use of rare vocabulary and idiomatic expressions in classical chinese. therefore, we design a novel reading comprehension task, irc, from the exam papers and idiom dictionary. meanwhile, since machine translation of classical chinese is also a problem of great concern, we design a novel wywmt task to study this topic. in addition, considering that some tokens in the classic chinese language have the functions of prepositions, conjunctions and auxiliaries, and the same token has different meanings in different sentences, we design a new task, xuci, for token comparison. more details of these tasks are described in section 4 and appendix a. and we describe the principles we used to design tasks and the process of data collection in section 3. furthermore, to better understand the challenges provided by wyweb, we build a baseline for each task and evaluate several pre-trained models released by the community. the experimental results demonstrate that current state-of-the-art methods are struggling with these tasks, which suggests that those tasks in wyweb can constitute a useful testbed for developing and comparing nlp systems for classical chinese. the contributions of our work are summarized as follows: ## wyweb overview in this section, we introduce the principles and methods we applied during the construction process of wyweb, and describe a brief overview of tasks in ## tasks in this section, we describe tasks and datasets designed for specific aspects of classical chinese nlp. these datasets, except glner and fspc, are firstly created by ourselves. more details are shown in appendix a. ## conclusions and future work in this paper, we introduce a nlp benchmark for classical chinese, which contains nine nlp tasks and datasets respectively to help researchers to evaluate and analyze nlp models. also, we build a toolkit for reproduction and a leaderboard online for the community. the study of ancient chinese is a highly specialized subject, so the professionalism of this benchmark may need to be further improved. on the other hand, there is a big gap between the performance of the classical chinese models on this benchmark with other leader-boards. better models are needed to handle more linguistic features of classical chinese. furthermore, to resolve traditional and simplified character issue, traditional style tasks are meaningful to researchers. we consider it as a future work of the community. classical chinese is a treasure of the entire human cultural history. we contribute this work with the hope of helping the entire community to be more prosperous. our work will be an open, community-driven project which improves with the advancement of technology.