HUE : Pretrained Model and Dataset for Understanding Hanja Documents of A ncient K orea (2022)

Abstract: Historical records in Korea before the 20th century were primarily written in Hanja, an extinct language based on Chinese characters and not understood by modern Korean or Chinese speakers. Historians with expertise in this time period have been analyzing the documents, but that process is very difficult and time-consuming, and language models would significantly speed up the process. Toward building and evaluating language models for Hanja, we release the Hanja Understanding Evaluation dataset consisting of chronological attribution, topic classification, named entity recognition, and summary retrieval tasks. We also present BERT-based models continued training on the two major corpora from the 14th to the 19th centuries: the Annals of the Joseon Dynasty and Diaries of the Royal Secretariats. We compare the models with several baselines on all tasks and show there are significant improvements gained by training on the two corpora. Additionally, we run zero-shot experiments on the Daily Records of the Royal Court and Important Officials (DRRI). The DRRI dataset has not been studied much by the historians, and not at all by the NLP community.

## introduction large-scale historical records in korea were mostly produced during the joseon dynasty (1392-1897), and the institute for the translation of korean classics (itkc) keeps a comprehensive database of korean classics at a scale of approximately 9 billion characters. this digital archive is a great resource for korean historians, but the documents remain in the original hanja language 2 . hanja is an extinct 1 all codes, models, and dataset are available at https: //github.com/haneul-yoo/hue.git 2 hanja is a set of characters (script) used in ancient korean, while hanmun is a writing style (language) in the same era. however, we will refer to hanja as a language following the conventions of the previous works. ## hue dataset the hue (hanja understanding evaluation) dataset is built to assist history scholars to understand korean historical records written in hanja. hue consists of chronological attribution, topic classification, named entity recognition, and summary retrieval, which are tasks that can provide helpful information for studying the documents. we expect that the language models based on hue will ultimately help historians to interpret unseen historical documents and public to grasp basic concept of those documents. we describe each task in detail below. ## hanja pretrained model as far as we know, there have been no pretrained language models for the hanja language. one can use related lms, the pretrained models for ancient chinese as well as multilingual bert @xcite which includes traditional chinese in its training corpus. anchibert @xcite is pretrained in ancient chinese with the chinese anthologies written around 1000bc to 200bc. there is some vocabulary overlap between the hanja documents and traditional chinese corpora, we can adopt multilingual bert and anchibert to learn the representations of the hanja texts. we propose the pretrained language models suitable for hanja documents by continuing pretraining those two models on both ajd and drs. ## conclusion we present hue (hanja understanding evaluation) dataset and bert-based pretrained language models for classical hanja documents. hue dataset includes diverse tasks that can support analyzing historical documents written in hanja which is an extinct language in korea: chronological attribution (ca), topic classification (tc), named entity recognition (ner), and summary retrieval (sr). our models pretrained on hanja corpora outperform other language models and we observe their performance on zero-shot settings with drri which is the dataset never been introduced in nlp community. the experimental results in king prediction imply that our models capture the historical events or facts disclosed in the texts. we also explore several methods to support hanja language models such as masking named entities and giving document age as input sources, based on the analyses on textual features in ajd. help of adequate resources in hanja documents might could fill some caveats in our work which lacks additional experiments and analyses on the records of different genre such as poetry, novel, and humanities resulting from the low resources that we can exploit. however, we expect that our dataset and accompanying language models might facilitate future works on historical documents written in hanja by providing fundamental resources to leverage unknown hanja corpora. we crawl ajd foot_3 , drs foot_4 , and drri foot_5 from the comprehensive database for korean classics which are publicly available published by iktc. all source corpora are fully tagged with the written ages and named entities, while their entity maps differ to each other. ajd also provides topics which is tagged by the experts in the translation process.