MT 2: Towards a Multi-Task Machine Translation Model with Translation-Specific In-Context Learning (2023)

Abstract: Sentence-level translation, document-level translation, translation memory, and terminology constrained translation play an important role in machine translation. Most of the previous work uses separate models or methods to solve these tasks, which is not conducive to knowledge transfer of different tasks and increases the complexity of system construction. In this work, we explore the potential of pre-trained language model in machine translation tasks and propose a Multi-Task Machine Translation (MT2) model to integrate these translation tasks. We design a novel translation-specific In-Context Learning (ICL) paradigm for model training, in which all of the translation tasks can be modeled as context-learning tasks that integrate contextual information for performance improvement. Specifically, we propose a retrieval and alignment method to obtain a large scale context-enhancement training data, then we train the model in an in-context learning manner. Furthermore, we adopt two context-dependent training strategies to encourage the model to better understand and utilize contextual information for translation. Extensive experiments on translation memory, terminology constrained translation, document-level translation, and few-shot domain-adaptation tasks demonstrate the superior performance of our model, verifying the effectiveness of our proposed approach.

## introduction translation memory, terminology constrained translation, and document-level machine translation play an important role in machine translation. most previous studies employ separate models or methods to handle these tasks, such as introducing additional context encoders @xcite and hierarchical attention to modeling context information @xcite in documentlevel translation task, or changing the model structure @xcite @xcite to integrate translation memory sentences. this hampers the transfer of knowledge across diverse tasks and increases to the intricacy of system development. recently, large language models (llms) have gained significant attention in the field of natural language processing. there are several work that are interested in the application of llms in machine translation and have conducted many evaluation work with llms, such as gpt3.5 and bloom-176b @xcite @xcite . these work find that even the currently best-performed llm, chatgpt, still lags behind the powerful supervised models in many cases @xcite . meanwhile, some work has shown that the llms can perform better when given appropriate prompts, which shows the in-context learning ability of large models @xcite @xcite . these work mainly focus on evaluating and how to stimulate the performance of large language models, however, we focus on training a translation-specific model to enhance in-context translation capability. in this work, we explore training a unified model to address multiple machine translation tasks. we propose a multi-task machine translation (mt2) model, which follows the in-context learning paradigm to tackle the context-learning translation tasks that integrate contextual information for performance improvement, such as translation memory, terminology constrained translation, and document-level machine translation et al. we propose a two-stage training method, where the first stage focuses on training the general translation ability, and the second stage aims to improve its context-modeling ability with contextenhancement data and training strategies. especially, we propose a method based on retrieval and alignment to obtain a large-scale contextenhancement training data, where the retrieved similar sentences and aligned translation pairs are used as contexts for improving translation. moreover, we adopt two context-dependent training strategies to enhance the context-modeling ability of the model, which encourage the model to better understand and utilize the knowledge transmitted through the context. we give some examples to compare our model with the chatgpt, and it is shown in figure 1 . we conduct extensive experiments on multiple translation tasks, including translation memory, terminology constrained translation, document-level translation, and domain adaptation tasks. with the help of in-context learning paradigm, the translation quality and term translation accuracy achieve significant improvement. in addition, the model can perform domain-specific translation directly by concatenating few in-domain examples without updating parameters. our contributions can be summarized as follows: ## conclusion in this work, we propose a multi-task machine translation model with translation-specific incontext learning, which can address multiple context-learning type translation tasks with the incontext learning paradigm. we propose a method based retrieval and alignment to construct the context-enhancement data and then train the model with in-context learning paradigm in two stages. in addition, we also adopt two context-dependent training strategies to make model better understand and utilize contextual information for performance improvement. finally, we conduct experiments on terminology constrained translation, documentlevel translation, translation memory and few-shot domain-adaptation tasks to evaluate the effectiveness of our approach. we will extend the model to a multilingual translation system and explore instruction transfer learning in the future.