Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data (2024)

Abstract: In this paper, we propose a two-phase training approach where pre-trained large language models are continually pre-trained on parallel data and then supervised fine-tuned with a small amount of high-quality parallel data. To investigate the effectiveness of our proposed approach, we conducted continual pre-training with a 3.8B-parameter model and parallel data across eight different formats. We evaluate these methods on thirteen test sets for Japanese-to-English and English-to-Japanese translation. The results demonstrate that when utilizing parallel data in continual pre-training, it is essential to alternate between source and target sentences. Additionally, we demonstrated that the translation accuracy improves only for translation directions where the order of source and target sentences aligns between continual pre-training data and inference. In addition, we demonstrate that the LLM-based translation model is more robust in translating spoken language and achieves higher accuracy with less training data compared to supervised encoder-decoder models. We also show that the highest accuracy is achieved when the data for continual pre-training consists of interleaved source and target sentences and when tags are added to the source sentences.

## introduction in machine translation, transformer encoderdecoder models @xcite , such as @xcite , mt5 @xcite , and mbart @xcite predominate. the emergence of pre-trained large language models (llms) composed solely of the transformer decoder, such as gpt series @xcite , has prompted the development of pre-trained llms, including, palm @xcite , and llama @xcite . when translating with these llms, it is common to use in-context few-shot learning. according to @xcite , gpt-3 demonstrates comparable or superior accuracy to wmt-best for high-resource languages. furthermore, as reported by @xcite , gpt-4's 5-shot surpasses wmt-best's accuracy in most translation directions. however, @xcite noted that in 8-shot scenarios, relatively small-scale llms (e.g., 7b parameters) exhibit lower accuracy than supervised encoder-decoder models. therefore, it is necessary to investigate methods capable of achieving translation accuracy equivalent to existing translation models with relatively small-scale llms. on the other hand, in models such as bert @xcite and roberta @xcite , which consist solely of transformer encoders, the effectiveness of continual pre-training, where pretrained models are further trained on task-specific data such as classification to improve the accuracy of the task, has been reported @xcite . in the context of llms, continual pre-training has been reported to transfer models primarily pre-trained in english, such as llama, to other languages @xcite . additionally, when building llm-based translation models, the effectiveness of conducting continual pre-training with either monolingual data, parallel data, or both, followed by supervised fine-tuning, has been reported, mainly when basing the model on primarily english pre-trained models such as llama-2 @xcite @xcite . although those recent publicaion in the context of llms are closely related to our study, this paper presents research conducted independently of those latest llm-based translation studies such as @xcite ; @xcite ; @xcite . this paper proposes a two-phase training approach: continual pre-training on parallel data crawled from the web and supervised fine-tuning using a small amount of high-quality parallel data created by professional translators. to comprehen-sively investigate methods for improving translation accuracy through continual pre-training, we conduct continual pre-training across eight data formats for japanese-to-english (ja ⇒ en) and english-to-japanese (en ⇒ ja) translations using a 3.8b-parameter llm. we evaluate the translation accuracy on 13 test sets. our paper's novelty compared to @xcite ; @xcite ; @xcite lies in the following aspects. parallel data in pre-training from scratch when pre-training llms from scratch, it is expected to use monolingual data. however, some reports incorporating bilingual data, such as parallel data, into the pre-training dataset can enhance the accuracy of downstream tasks. @xcite show that incorporating parallel data into pre-training 1b and 8b parameter llms enhances translation accuracy in zero-and five-shot. separate studies further show that including parallel data in the pre-training of the encoder-decoder model also improved performance in downstream multilingual and cross-lingual tasks @xcite . llms-based translation models @xcite demonstrated that fine-tuning 15 multilingual llms using qlora for french-to-english translation surpasses the accuracy of both incontext few-shot learning and models trained from scratch. conversely, @xcite demonstrated that models predominantly pre-trained on english data, such as llama-2, suffer reduced translation accuracy when translating into non-english target languages. addressing this issue, they introduced alma, a method that employs fine-tuning monolingual data in the first stage, followed by supervised fine-tuning with a small quantity of high-quality parallel data in the second stage. furthermore, there exists a report on improving translation accuracy by employing contrastive preference optimization (cpo) for the second stage of supervised fine-tuning in alma @xcite . in addition, the effectiveness of utilizing monolingual and parallel data in the first stage has been reported @xcite . llm-based translation models have only been evaluated on test data from the wmt general machine translation task @xcite and @xcite . therefore, their effectiveness compared to conventional supervised encoder-decoder models has not been sufficiently validated across various types of data. additionally, the impact of continual pre-training data on translation accuracy remains unclear. our study aims to address these two points. ## continual pre-training and supervised fine-tuning with parallel data we introduce a two-phase training to enhance the accuracy of translation of llms. in the first phase, we perform continual pre-training using parallel data crawled from the web, such as paracrawl @xcite . then, in the second phase, we conduct supervised fine-tuning with a small amount of high-quality parallel data. in llm fine-tuning, the importance of data quality has been reported @xcite . however, it has also been reported that parallel data crawled from the web may have low data quality @xcite . therefore, we used data created by professional translators as high-quality parallel data. ## experiments we conduct experiments on two nvidia rtx a6000 gpus. due to severely limited com-putational resources, we use a 3.8b parameters llm, rinna/bilingual-gpt-neox-4b (rinna-4b) foot_0 , which is already pre-trained on japanese and english data, totaling 524b tokens, with 173b tokens in japanese and 293b tokens in english. since rinna-4b has undergone sufficient pre-training from scratch on monolingual data for both japanese and english, as stated in section 3, we believe that continual pre-training with monolingual data is unnecessary. given that the model we employ is pretrained on japanese and english, we experiment with japanese-to-english and english-to-japanese translation tasks. all experiments utilizing rinna-4b are conducted using the open-source huggingface transformers library. foot_1 4.1 dataset ## conclusion we propose a two-phase training approach comprising continual pre-training with interleaved source and target sentence data, followed by supervised fine-tuning using a small amount of high-quality parallel data. our investigation comprehensively explores methods for enhancing translation accuracy through continual pre-training across eight data formats. evaluation across 13 test sets reveals that models trained with continual pre-training followed by supervised fine-tuning outperform those supervised fine-tuned solely on parallel data. furthermore, we observe variations in language direction accuracy improvement during continual pretraining based on the order of source and target sentences. we also demonstrate that llm-based translation models are more robust in translating sentences containing spoken language, and achieve higher accuracy with less training data, compared to traditional encoder-decoder models. additionally, augmenting source sentences with tags or using prefixes yields higher accuracy than simple concatenation. in larger llms than the rinna-4b model we utilized, such as llama-2 7b and 13b, lora enables training with fewer computational resources. lora has been reported to be effective in translation tasks @xcite . therefore, it is essential to experiment with lora in the future to determine if similar results can be achieved and to investigate if similar results can be obtained with other llms.