A Novel Paradigm Boosting Translation Capabilities of Large Language Models (2024)

Abstract: This paper presents a study on strategies to enhance the translation capabilities of large language models (LLMs) in the context of machine translation (MT) tasks. The paper proposes a novel paradigm consisting of three stages: Secondary Pre-training using Extensive Monolingual Data, Continual Pre-training with Interlinear Text Format Documents, and Leveraging Source-Language Consistent Instruction for Supervised Fine-Tuning. Previous research on LLMs focused on various strategies for supervised fine-tuning (SFT), but their effectiveness has been limited. While traditional machine translation approaches rely on vast amounts of parallel bilingual data, our paradigm highlights the importance of using smaller sets of high-quality bilingual data. We argue that the focus should be on augmenting LLMs’ cross-lingual alignment abilities during pre-training rather than solely relying on extensive bilingual data during SFT. Experimental results conducted using the Llama2(CITATION)model, particularly on Chinese-Llama2(CITATION) after monolingual augmentation, demonstrate the improved translation capabilities of LLMs. A significant contribution of our approach lies in Stage2: Continual Pre-training with Interlinear Text Format Documents, which requires less than 1B training data, making our method highly efficient. Additionally, in Stage3, we observed that setting instructions consistent with the source language benefits the supervised fine-tuning process. Experimental results demonstrate that our approach surpasses previous work and achieves superior performance compared to models such as NLLB-54B(CITATION) and GPT3.5-text-davinci-003, despite having a significantly smaller parameter count of only 7B or 13B. This achievement establishes our method as a pioneering strategy in the field of machine translation.

## introduction translation capabilities of large language models (llms) @xcite @xcite in machine translation (mt) tasks have been explored extensively in previous research @xcite @xcite @xcite . however, achieving significant improvements in translation quality through supervised fine-tuning (sft) strategies has proven challenging. traditionally, machine translation relies on vast amounts of parallel bilingual data, but sft only requires a small amount of high-quality bilingual data, highlighting a crucial distinction. it is a naive approach to consider using vast quantities of parallel bilingual data during sft. however, experiments have shown that increasing the data volume yields limited improvements in quality and even leads to performance degradation on certain test sets. thus, the question arises: are extensive parallel bilingual data useless in sft, or are they being misused? in this paper, we propose a novel training paradigm, consisting of three stages, to boost the translation capabilities of llms. our contributions include refining the training strategy for downstream tasks and emphasizing the enhancement of llms' cross-lingual alignment abilities during pretraining. these contributions address the limitations observed in previous research. our training paradigm comprises the following stages: stage 1: continual pre-training using extensive monolingual data. consistent with previous findings @xcite , we validate the effectiveness of monolingual data augmentation. specifically, we perform sft on a chinese-llama2 @xcite model, which undergoes monolingual data augmentation, thereby demonstrating improved translation capabilities. stage 2: continual pre-training with sentence-aligned parallel data. we construct interlinear text format from sentence-aligned bilingual parallel data and utilize them for continual pre-training of llms. experimental results demonstrate the critical importance of this stage, resulting in a significant improvement in translation quality, particularly for english-other translations. stage 2 stands as a pivotal contribution in our paper, requiring less than 1b training data, thereby enhancing training efficiency. stage 3: leveraging source-language consistent instruction for supervised fine-tuning. in sft, we discover that using instruction aligned with the source language of the translation notably improves performance. leveraging sourcelanguage consistent instructions during sft yields significant enhancements. in summary, we introduce a three-stage training paradigm, highlighting the effectiveness of secondary pre-training, continual pre-training with interlinear text format documents, and leveraging source-language consistent instruction for supervised fine-tuning. these contributions address the limitations observed in previous research and pave the way for improved translation quality. ## a new training recipe we propose an innovative training strategy to enhance the translation capabilities of large language models. as shown in figure 1 , our approach consists of three stages: (1) secondary pre-training using extensive monolingual data, (2) continual pre-training with interlinear text format documents, and (3) leveraging source-language consistent instruction for supervised fine-tuning. ## results and analysis as shown in ## conclusions in this study, we have introduced a novel paradigm for enhancing the translation capabilities of large language models in machine translation tasks. our three-stage approach, including secondary pretraining using extensive monolingual data, continual pre-training with interlinear text format documents, and leveraging source-language consistent instruction for supervised fine-tuning, addresses the limitations of previous strategies and offers notable improvements in translation quality. we emphasize the significance of pre-training stages in enhancing llms' cross-lingual alignment abilities and the effectiveness of using a smaller but high-quality set of bilingual data during supervised fine-tuning. notably, stage2, which involves continual pre-training with interlinear text format documents, stands out as a highly efficient method, requiring minimal training data. furthermore, aligning the instructional setting with the source language during supervised fine-tuning, as observed in stage3, yields positive effects. the findings from this paper contribute to advancing the field of machine translation and offer valuable insights for optimizing the translation capabilities of large language models. future research can explore additional language pairs, alternative data augmentation techniques, and different pre-training strategies to further refine our proposed paradigm.