Restoring and Mining the Records of the J oseon Dynasty via Neural Language Modeling and Machine Translation (2021)

Abstract: Understanding voluminous historical records provides clues on the past in various aspects, such as social and political issues and even natural science facts. However, it is generally difficult to fully utilize the historical records, since most of the documents are not written in a modern language and part of the contents are damaged over time. As a result, restoring the damaged or unrecognizable parts as well as translating the records into modern languages are crucial tasks. In response, we present a multi-task learning approach to restore and translate historical documents based on a self-attention mechanism, specifically utilizing two Korean historical records, ones of the most voluminous historical records in the world. Experimental results show that our approach significantly improves the accuracy of the translation task than baselines without multi-task learning. In addition, we present an in-depth exploratory analysis on our translated results via topic modeling, uncovering several significant historical events.

## introduction historical records are invaluable sources of information on the lifestyle and scientific records of our ancestors. humankind has learned how to handle social and political problems by learning from the past. the historical records also serve as the evidence of intellectual accomplishment of humanity over time. given such importance, there has been a great deal of nationwide efforts to preserve these historical records. for instance, unesco protects world heritage sites, and experts from all around the world have been converting and restoring historical records in a digital form for long-term preservation. a representative example is the google books library project foot_0 . however, despite the importance of the historical records, it has been challenging to properly utilize the records for the following reasons. first, the nontrivial amounts of the documents are partially damaged and unrecognizable due to unfortunate historical events or environments, such as wars and disasters, as well as the weak durability of paper documents. these factors result in difficulties to translate and understand the records. second, as most of the records are written in ancient and outdated languages, non-experts are difficult to read and understand them. thus, for their in-depth analysis, it is crucial to recover the damaged parts and properly translate them into modern languages. to address these issues existing in historical records, we formulate them as the task of language modeling, especially for the recovery and neural machine translation, by leveraging the advanced neural networks. moreover, we apply topic modeling to the translated historical records to efficiently discover the important historical events over the last hundreds of years. in particular, we utilize two representative korean historical records: the annals of the joseon dynasty and the diaries of the royal secretariat (hereafter we refer to them as ajd and drs respectively). these records, which contain 50 million and 243 million characters respectively, are recognized as the largest historical records in the world. considering their high value, unesco recognized them as the memory of the world. 2,3 these two historical corpora contain the contents of five hundred years from the fourteenth century to the early twentieth century. in detail, ajd consists of administrative affairs with national events, and drs contains events that occurred around the kings of the joseon dynasty. these corpora are valuable as they contain diverse information including international relations and natural disasters. in addition, the contents of the records are objective since the writing rules are strict that political intervention, even from the kings, is not allowed by their independent institution. although drs contains a much larger amount of information than ajd, only 10-20% of drs has been translated into the modern korean language by a few dozens of experts for the last twenty years. the complete translation of drs is currently expected to additionally take more than 30-40 years if only human experts continue to translate them. applying the neural machine translation models into the historical records contains several issues. first, the pre-trained models for chinese are not suitable to drs and ajd, mainly because of the differences between hanja and the chinese language. in the past, korean historiographers borrowed the chinese character to write the sentences spoken by koreans. as a result, diverse characters had been moderated or created, and considerable grammatical differences exist between the chinese language and hanja. furthermore, several parts of those records are damaged and require restoration as shown in fig. 2 . therefore, these damaged parts should be restored in order to translate them correctly. in order to address these issues, we propose overall, we propose a novel multi-task approach to restore the damaged parts and translate the records into a modern language. afterward, we extract the meaningful historical topics from the world's largest historical records as shown in fig. 1 . this study makes the following contributions: ## related work this work broadly incorporates three different tasks: document restoration, machine translation, and document analysis. therefore, this section describes studies related to the restoration of damaged documents, neural machine translation, and the analysis of historical records. ## proposed methods this section describes a multi-task learning approach based on the transformer networks to effectively restore and translate the historical records. the overview of our model is shown in fig. 3 . ajd and drs datasets consist of hanja sentences @xmath0, . . . , h n } and korean sentences @xmath1, . . . , k n }, where each korean sentence is translated from its corresponding hanja sentence. here, the hanja represents the chinese characters borrowed to write the korean language in the past. especially, drs contains additional hanja sentences @xmath2, . . . , h m } that are not translated yet. hence, we have in total m hanja sentences in the hanja corpus such that ĥ = h ∪ h and n korean sentences in the korean corpus k. considering the properties of ajd and drs, we design a multi-task learning approach with document restoration and machine translation, based on the transformer networks. as shown in fig. 3 , our model consists of embedding and output layers for hanja and korean, and three transformer modules: the shared encoder, the restoration encoder, and the translation decoder. the restoration encoder is an encoder for the restoration task. the translation decoder is used for translating hanja sentences into modern korean sentences, and the shared encoder is used for both the restoration and translation tasks. by sharing the encoder module for both tasks, the shared encoder is trained with a large-scale corpus, i.e., the hanja-korean paired dataset and the additional unpaired hanja dataset. the parameter sharing technique assists the model to learn rich information from the hanja corpus. we apply the cross-layer parameter-sharing technique in the same manner as used in albert @xcite , which shares the attention parameters for each transformer encoder and decoder modules to reduce the model size and the inference time. ## experiments this section first describes our datasets and experimental settings. ## experimental results this section describes the results of the performances of our model for restoration and translation, followed by qualitative examples of each task as well as topic modeling results. ## conclusions in this paper, we proposed a novel approach to translate and restore the historical records of the joseon dynasty by formulating the multi-task learning task based on the self-attention mechanism. our approach significantly increases the translation quality by learning the rich contents in large documents. we anticipate these tasks are the first steps towards translating the ancient korean historical records into modern languages such as english. furthermore, the model effectively predicts the original words from the damaged parts of the documents, which is an essential step for restoaring the damaged documents. results from text mining approaches show that our approaches have the potential in supporting analysts in effectively exploring the large volume of historical documents. we also expect researchers from diverse domains can explore documents and discover historical findings such as astronomical phenomena and undiscovered international affairs, with no special domain knowledge. as future work, we will also leverage the transfer learning approach to translate historical documents into other languages, such as english or french. we also plan to apply knowledge graphbased machine learning approaches, e.g. knowledge graph embedding and graph neural networks, to discover historical events and relations.