Some Trials on Ancient M odern C hinese Translation (2023)

Abstract: In this study, we explored various neural machine translation techniques for the task of translating ancient Chinese into modern Chinese. Our aim was to find an effective method for achieving accurate and reliable translation results. After experimenting with different approaches, we discovered that the method of concatenating adjacent sentences yielded the best performance among all the methods tested.

## introduction chinese characters are the writing system of chinese and are considered one of the oldest written languages in the world. according to verifiable records, over 3000 years ago, chinese characters had already developed a mature writing system, including oracle bone inscriptions. while some characters have been retained throughout the subsequent development process, the expression forms and meanings of ancient chinese and modern chinese differ significantly. ancient chinese often features rare words that are not commonly found in modern chinese, and the grammar structures also vary. consequently, reading ancient chinese poses difficulties for modern individuals, often necessitating the expertise of professionals to translate it into modern chinese. neural machine translation has already demonstrated remarkable performance in various bilingual translation tasks. however, there has been limited exploration of the existing advanced neural machine translation technology in the domain of monolingual translation from ancient chinese to modern chinese. this relatively specialized field has received little attention in terms of developing neural machine translation technology. in this evahan 2023 competition, we are participating in the task of translating ancient chinese into modern chinese. the training data of this task is extracted from the twenty-four histories (recording the history from the pre qin period to the ming dynasty), which was finished by the research group of the national social science foundation of china major project "research on the construction and application of cross-language knowledge base of ancient chinese classics" (project no. :21&zd331) in this paper, we begin by introducing the methods employed in our study, including data augmentation, fine-tuning of pre-trained models, and attention mechanisms such as group attention. next, we provide details on our training setting, including the use of pre-trained models, data segmentation, vocabulary construction, and the division of training and validation sets. we also present the experimental results, including the performance of different methods on evaluation metrics such as bleu and chrf. lastly, we discuss the outcomes of our attempts, highlighting the effectiveness of certain techniques, the limitations of others, and the potential for further improvements in low-resource translation tasks. ## method in this section, we will describe some methods that we have tried. data augmentation data augmentation is an essential technique in machine learning tasks to increase the size and diversity of the training data. in our case, we have observed that adjacent sentences in the training data are typically in the same article. therefore, a straightforward data augmentation method we employ is concatenating adjacent sentences to form longer sentences. we denote the concatenation of no more than k sentences as k-cat. tune training translation models can be seen as training classification models, as both tasks involve categorizing text. however, the challenge arises from the presence of numerous categories, leading to a long tail problem. to address this issue, we draw inspiration from the approach described in @xcite . in our methodology, we augment the trained model by adding log p (x) to the bias term of the final classification layer. here, p (x) represents the prior distribution of token x, which is derived from the distribution observed in the training set. subsequently, we continue training the model, while keeping the non-classification layers frozen. group attention group attention techniques @xcite , such as group attention and combine attention, were applied to data augmented with concatenated sentences. group attention focuses exclusively on information within the same sentence during attention calculations, while combine attention combines group attention with traditional global attention. finetune pre-trained model to further enhance the model's performance, we conducted fine-tuning using the masked lm technique inspired by bert @xcite . the pretrained model was trained for 50 epochs on the training dataset. subsequently, the fine-tuned model was loaded and trained on the 3-cat and 4-cat data. ## conclusion in conclusion, among all the methods explored in our experiments, the data augmentation technique of directly concatenating sentences proved to be the most effective. however, as the number of concatenated sentences increased, the improvement in performance became less significant. this suggests that simply adding more repetitive data does not necessarily lead to better results. it also indicates that the potential of the model may not be fully utilized with only 0.3m parallel sentences. therefore, for low-resource translation tasks such as translating from ancient chinese to modern chinese, data augmentation methods should be the primary approach to consider.