E3: Optimizing Language Model Training for Translation via Enhancing Efficiency and Effectiveness (2024)

Abstract: “In the field of Natural Language Processing (NLP), Large-scale Language Models (LLMs) havedemonstrated exceptional capabilities across a variety of tasks, including question answering,classification, and particularly, natural language understanding. The integration of neural ma-chine translation with LLMs presents significant potential, transforming the paradigms of cross-lingual communication and information exchange. This study investigates the foundational as-pects of LLMs’ translation abilities and identifies effective training methodologies to equip themwith multilingual capacities. We specifically explore the optimal timing for introducing trans-lation capabilities to LLMs via supervised tasks, considering the inherent bilingual nature ofmachine translation. Key questions explored include whether it is more beneficial to integratemultiple languages during the pre-training or supervised fine-tuning (SFT) stages, how varia-tions in language ratios influence LLMs’ translation abilities, and whether longer or shorter textsare more effective for training these models. This research conducts a thorough investigationby training multiple LLMs from scratch with parameter scales in the billions and enhances therobustness of our findings by upgrading the language capabilities of pre-trained open-sourcemodels with parameter scales reaching tens of billions. The aim is to provide a detailed analysisthat elucidates the complexities of augmenting machine translation capabilities within LLMs.”

## introduction the large language model (llm) has proven itself capable across a broad spectrum of tasks, including named entity extraction, text classification, and document understanding. moreover, the llm has demonstrated the ability to address highly complex tasks using the innovative chain of thoughts (cot) methodology, as pioneered by zhang et al. @xcite . among the various capabilities of the llm, this paper focuses on one particular area that has significantly intrigued the authors: its exceptional proficiency in machine translation. this proficiency promises transformative potential for the field, with extensive implications for cross-lingual communication and the broader dissemination of information. the evolutionary trajectory of large language models (llms) has undergone significant cyclical transformations, each contributing substantially to their growth and capabilities. the journey began with the introduction of the transformer model for neural machine translation, a foundational breakthrough as described by @xcite . building on this, bert @xcite leveraged the transformer's encoder to achieve exceptional performance in text classification and cloze tests. following the advent of bert, a series of influential models emerged, notably gpt @xcite , which is characterized by its unique decoder-only architecture. furthermore, models like t5 @xcite and mt5 @xcite , with their comprehensive encoder-decoder frameworks, have continuously expanded the boundaries of what is possible with llms. most of these models are either directly based on the transformer architecture or represent substantial enhancements to it, reaffirming the critical role of the transformer in the advancement of the field. this progression of models, each augmenting the work of its predecessors, has driven llms to their current state of remarkable performance and vast potential in natural language processing (nlp). this paper aims to delve deeper into these evolving aspects of natural language processing, particularly focusing on the translation proficiency of pre-trained language models (plms). our objectives are threefold: to explore the foundational aspects of translation efficiency in plms, to investigate the impact of diverse training methodologies and data ratios on plms' translation capabilities, and to propose effective strategies for training plms to enhance their translation performance. we explore several pertinent questions in the context of neural machine translation using llms: ## machine translation in the large language model (llm) era the exploration of machine translation driven by language models (lms) has been revitalized by the advent of large language models (llms), building on a rich history of significant research. pioneering studies such as @xcite integrated large-scale statistical language modeling into machine translation, laying the groundwork for subsequent advancements. this approach was refined by @xcite , who tailored lms to improve the translation of complex phrases. the shift to neural language models, notably researched by @xcite ) @xcite , emphasized their potential to enhance translation accuracy and fluency. lee (2020) @xcite effectively bridged theoretical research with practical applications, particularly in quality estimation tasks, demonstrating the real-world impact of these developments. in more recent studies, @xcite examined the superiority of extra-large pre-trained language models (xlplms) over smaller counterparts in domain-specific translation tasks, providing vital insights into model selection for specific contexts. concurrently, @xcite introduced multi-stage prompting, an innovative method that utilizes pre-trained lms for translation, illustrating the continued evolution in lm applications in translation tasks. @xcite extended this investigation, exploring the capability of llms in translating a diverse array of languages and thus contributing to our understanding of the scalability and adaptability of llms in multilingual translation scenarios. ## conclusion in this study, we conducted an extensive series of comparative experiments using five models trained from scratch, each with a parameter count of 1.3 billion, alongside several continued-training scenarios on open-source large language models (llms) of 13 billion parameters. our key findings reveal that integrating translation tasks during the pre-training phase of llms consistently enhances performance in handling longer text sequences, especially evident in context-aware translation tasks. additionally, our results indicate that strategically segmenting translation training tasks from the base model training into distinct phases can significantly expedite the overall training process of the model. while variations in language ratios were observed to affect the convergence speed and training duration of the base model, they did not substantially impact the performance metrics of downstream translation tasks. this dynamic interaction between pre-training strategies, task integration, and language ratio effects underscores the potential for optimizing both the efficiency and effectiveness of neural machine translation systems within large-scale language models. future efforts will focus on employing under-resourced languages to investigate methods for bridging 'zero-shot' language translation challenges.