Translating A ncient C hinese to M odern C hinese at Scale: A Large Language Model-based Approach (2023)

Abstract: Recently, the emergence of large language models (LLMs) has provided powerful foundation models for a wide range of natural language processing (NLP) tasks. However, the vast majority of the pre-training corpus for most existing LLMs is in English, resulting in their Chinese proficiency falling far behind that of English. Furthermore, ancient Chinese has a much larger vocabulary and less available corpus than modern Chinese, which significantly challenges the generalization capacity of existing LLMs. In this paper, we investigate the Ancient-Chinese-to-Modern-Chinese (A2M) translation using LLMs including LLaMA and Ziya. Specifically, to improve the understanding of Chinese texts, we explore the vocabulary expansion and incremental pre-training methods based on existing pre-trained LLMs. Subsequently, a large-scale A2M translation dataset with 4M pairs is utilized to finetune the LLMs.Experimental results demonstrate the effectiveness of the proposed method, especially with Ziya-13B, in translating ancient Chinese to modern Chinese. Moreover,we deeply analyze the performance of various LLMs with different strategies, which we believe can benefit further research on LLM-based A2M approaches.

## introduction ancient chinese plays a crucial role in carrying the invaluable heritage of traditional chinese culture. however, ancient chinese expresses in a significantly different way compared with modern chinese, which hinders the understanding of ancient chinese books by non-experts. therefore, automatic ancient-chinese-to-modern-chinese (a2m) translation is essential to the preservation of traditional chinese culture. existing neural machine translation methods mainly adopted a sequence-tosequence paradigm, evolving from architectures based on recurrent neural networks @xcite , to convolutional neural networks @xcite , and to transformer @xcite . although great industrial and academic success has been achieved in the neural machine translation area, the a2m translation @xcite @xcite is still quite under-explored. with the emergence of large language models (llms) @xcite , they have rapidly been applied to a wide variety of natural language processing (nlp) tasks, exhibiting high generalization and reasoning capacities. although there have been studies @xcite that attempt to use llms for ancient chinese, their model sizes are limited. to this end, we propose to solve the a2m translation problem using llms with large-scale parameters and datasets. specifically, the model architecture is based on llama @xcite and its variants (e.g., ziya @xcite ). furthermore, owing to the lack of chinese texts in the pre-training corpus of llama, we align the chinese understanding ability of llama with english using vocabulary extension and incremental pre-training following recent works @xcite @xcite . after that, a large-scale a2m translation dataset with 4m pairs is employed to fine-tune the llm, so as to transfer its general capacity to the specific a2m translation task. the experimental results demonstrate the effectiveness of our method, exhibiting 29.68% bleu-4 on the testing set of the evahan2023 competition dataset @xcite . ## methodology in this section, we present our approach for transferring the knowledge of a pre-trained llm (e.g., llama @xcite ) using english pre-training corpus to the task of translating ancient chinese to modern chinese. as depicted in figure 1 , our approach involves three main steps, i.e., vocabulary extension, incremental pretraining, and large-scale finetuning. in the following sections, we will give detailed descriptions of these steps. ## conclusion in this paper, we propose a novel approach to address the ancient-chineseto-modern-chinese (a2m) translation task using large language models (llms). specifically, based on existing pre-trained llms, the proposed method involves vocabulary extension, incremental pre-training, and large-scale finetuning. the experimental results demonstrate the effectiveness of our method on the a2m translation task. moreover, the ablation study highlights the importance of vocabulary extension and incremental pre-training for llms to improve their understanding of low-resource languages. we believe that our findings can benefit further research on llm-based a2m approaches and contribute to the preservation of traditional chinese culture.