Pre-trained Model In A ncient- C hinese-to- M odern- C hinese Machine Translation (2023)

Abstract: This paper presents an analysis of the pre-trained Transformer model Neural Machine Translation (NMT) for the Ancient-Chinese-to-Modern-Chinese machine translation task.

## introduction the transformer model @xcite has demonstrated exceptional performance in various natural language processing tasks, including machine translation. one key aspect that has propelled the transformer's success is the utilization of pre-training techniques, such as the popular bert @xcite ) model. by pre-training on large-scale corpora, bert captures rich linguistic representations and context, allowing for more effective transfer learning. in this study, we incorporate a pre-trained transformer model, chinese-roberta-wwmext @xcite into our nmt system, enabling it to leverage the wealth of linguistic knowledge encoded in the pre-training process. by fine-tuning the pre-trained model on translation-specific data, we aim to exploit the benefits of both pre-training and task-specific learning. ## related work the currently most commonly used bert model @xcite is pre-trained on generaldomain text using universal language representation. while the model exhibits strong generality, its performance is easily constrained when applied to natural language processing tasks involving domain-specific texts. due to the inherent differences in grammar, semantics, and pragmatics between ancient chinese and other languages, there are significant deviations in features. it is challenging to achieve the same performance level as in general corpora. therefore, the direct application of bert in projects related to ancient chinese does not yield ideal results. anchibert @xcite is a pre-trained model specifically designed for ancient chinese texts. it "reads" a total of 39.5 million characters of ancient chinese, including historical records, prose, ancient poetry, and couplets, spanning thousands of years. the downstream tasks of anchibert include comprehension and generation of ancient chinese texts. the paper suggests that to use anchibert for text generation in ancient chinese, a framework based on the transformer model can be adopted. the encoder part of the framework utilizes an-chibert, while the decoder part uses the original transformer model's decoder with randomly initialized parameters. in the same year, dongbo wang et al. employed high-quality, validated full-text corpus from the qing dynasty's qianlong period edition of the extensive series "siku quanshu" as an unsupervised training set. they continued training a bert model based on the bert structure, incorporating a large amount of ancient chinese texts. this led to the development of the siku bert @xcite pre-trained language model specifically tailored for intelligent processing tasks related to ancient chinese. by directly using the pre-trained model as initialization parameters, not only did the model possess stronger generalization capabilities and faster convergence speed, but it also required only a small amount of labeled data for fine-tuning, significantly improving the performance of natural language processing tasks while avoiding overfitting. the siku bert pre-trained language model and anchibert pre-trained model introduced the idea of transfer learning in low-resource machine translation research. this can provide a theoretical and practical foundation for the text generation project in ancient chinese. ## approach we employed a standard transformer-based nmt architecture with the pre-trained model, chinese-roberta-wwm-ext @xcite as the encoder and a randomly initialized decoder, depicted in figure 1 .chinese-roberta-wwm-ext @xcite , as an advanced language model specifically designed for processing and understanding the chinese language, has demonstrated exceptional performance on a wide range of chinese nlp benchmarks, surpassing previous state-of-the-art models in tasks such as text classification, sentiment analysis, and natural language understanding. during the training process, we employed a strategy where the encoder parameters were frozen, and only the decoder parameters were updated. at the same time, we adopted the technology of joined dictionary. thanks to the similarity between ancient chinese and modern chinese, joined vocabulary simplifies the process of aligning words between ancient chinese and modern chinese. by using a joined dictionary, identical words only require one embedding vector, reducing the number of model parameters, memory consumption, and computational overhead, thereby enhancing training efficiency. training details: the model was implemented on the top of fairseq toolkit 1 . the dropout rate was set to 0.3. we set weight decay to 1e -4 to overcome over-fitting. we used adam @xcite to optimize the model parameters, with β 1 = 0.9 and β 2 = 0.98. (a) loss (b) ppl figure 2: loss and ppl ## results during local test, we get the following results. figure 1 shows the loss during training and the change of perplexity(ppl), both of which decrease steadily. figure 2 shows the changes in the bleu @xcite figure 4: short example1 that require expansion based on context. although this does not significantly impact the overall comprehension of the translated sentences, it does emphasize the importance of incorporating longer contextual information. for short sentences, relying solely on the source-to-target sentence pairs may not suffice to effectively translate referential information. for longer sentences, as shown in figure 6 , we have found that our model is generally able to maintain consistency in conveying meaning. although the accuracy of the translation may slightly decrease with increasing sentence length, there are instances where certain vocabulary may not be rendered with utmost precision. nonetheless, overall comprehension of the sentences can still be achieved. on some vocabulary, our model translates more detail. also, the model demonstrates a high level of proficiency in accurately recognizing and classifying proper nouns. its advanced language processing capabilities enable it to effectively identify and distinguish names of specific people, places, organizations, and other entities. in the official evahan2023 foot_1 test set, the best model achieves the bleu score of 22.05. ## conclusion in this study, we explored the application of pre-trained transformer models in ancient-chinese-to-modern-chinese machine translation. by incorporating the chinese-robertawwm-ext model as the encoder in our nmt system, we aimed to leverage the rich linguistic representations and contextual knowledge captured through pre-training. our findings and experimental results shed light on the effectiveness of pre-training techniques for improving translation quality in this specific language pair. by leveraging pre-training techniques and adopting a joined dictionary approach, we achieved moderately satisfactory results, exploring the way for ancient-chinese-to-modern-chinese machine translation.