Exploring Multilingual Pretrained Machine Translation Models for Interactive Translation (2023)

Abstract: Pre-trained large language models (LLM) constitute very important tools in many artificial intelligence applications. In this work, we explore the use of these models in interactive machine translation environments. In particular, we have chosen mBART (multilingual Bidirectional and Auto-Regressive Transformer) as one of these LLMs. The system enables users to refine the translation output interactively by providing feedback. The system utilizes a two-step process, where the NMT (Neural Machine Translation) model generates a preliminary translation in the first step, and the user performs one correction in the second step–repeating the process until the sentence is correctly translated. We assessed the performance of both mBART and the fine-tuned version by comparing them to a state-of-the-art machine translation model on a benchmark dataset regarding user effort, WSR (Word Stroke Ratio), and MAR (Mouse Action Ratio). The experimental results indicate that all the models performed comparably, suggesting that mBART is a viable option for an interactive machine translation environment, as it eliminates the need to train a model from scratch for this particular task. The implications of this finding extend to the development of new machine translation models for interactive environments, as it indicates that novel pre-trained models exhibit state-of-the-art performance in this domain, highlighting the potential benefits of adapting these models to specific needs.

## introduction machine translation (mt) has become an integral part of modern communication, facilitating cross-border communication and interaction among people from different linguistic backgrounds. however, the effectiveness of mt depends heavily on the quality of the translation models and the techniques used for training them. recently, pre-trained multilingual mt models such as mbart (multilingual bidirectional auto-regressive transformer) @xcite , mt5 @xcite , and xlm @xcite have emerged as powerful tools that can achieve state-of-the-art performance on various benchmark datasets. now, we can obtain high-quality translations for a specific task or domain by fine-tuning them with a not large training data, which is very effective in using them in low-resource settings. however, even with the high performance of these pre-trained models, there are still challenges in achieving accurate and fluent translations for all languages and domains @xcite . interactive machine translation (imt), which combines human intelligence with mt, has been proposed as a potential solution to address these challenges and ensure consistently high-quality translations @xcite . imt systems allow users to actively participate in the translation process by providing feedback to the machine, which generates a new translation that corrects the previous error. this process repeats until the machine generates a perfect translation, and the user validates it. in this paper, we explore the use of mbart in an imt environment. we aim to investigate and compare the effectiveness of using a pre-trained model like mbart, which assesses stateof-the-art results in a large set of translation tasks, with models we train from scratch for a specific domain using the opennmt-py toolkit @xcite . to achieve this, we design and implement an imt system with a prefix-based protocol @xcite ) that integrates mbart. in this protocol, firstly described by @xcite , further developed by @xcite ; @xcite ; @xcite , the user only corrects at each iteration the first error that he finds from left to right. this process repeats until the user validates the machine-generated translation. we also fine-tune mbart to compare our results with the two versions of it, with and without the fine-tuning. to compare the different models, we use different evaluation metrics that help us evaluate the effort the user has to perform during the translation session. our results showed that although the mbart models obtained higher quality translation than ours, the user effort results are similar. we need to fine-tune the mbart model on the specific domain to achieve a better effort reduction. in order to achieve optimal translation quality, our models should produce the most accurate translations possible on the initial attempt. however, when operating in an imt environment, the model must adapt to user feedback. based on our findings, it can be concluded that while pre-trained models generally achieve better translation quality, training a mt model from scratch for a specific domain can lead to better generalization, which produces significant benefits in this field. our work presents several contributions to the field of imt. specifically, our contributions are as follows: the rest of the paper is organized as follows. in section 2, we provide a brief overview of related work in pre-trained language models and imt. section 3 describes our proposed approach in detail, including the architecture of the imt system and the feedback mechanism. in section 4, we present the experimental framework, and in section 5, we discuss the results obtained. finally, we conclude the paper in section 6 and discuss potential future directions for this research. ## related work in this work, our primary focus is to investigate whether pre-trained models, which have shown significant success in various tasks such as translation, can also be used effectively for imt. training a mt model from scratch for a specific domain can be time-consuming and challenging to obtain a suitable dataset. therefore, evaluating whether pre-trained models can achieve similar results in this field as in mt tasks would be beneficial. although mt and imt tasks are similar, the goal of the first is to obtain the most accurate translation, while the second tries to obtain a perfect translation with minimal user interaction, so we need a model that can generalize well more than one that performs better translations. this paper uses the multilingual pre-trained model mbart to compare its performance with our models trained from scratch for the specific task. there are other pre-trained multilingual models, such as mt5 @xcite , xlm, @xcite deltalm @xcite , or xglm @xcite , that we could have used for our purpose. as we have used, other people are trying to use these models for new tasks that were not initially thought. @xcite used to resolve math word problems, @xcite to summarize persian texts, @xcite generated poetry with them, and @xcite implemented it in the speech translation field. apart from pre-trained multilingual models, the advent of large language models has prompted research into their utilization for specific tasks, including translation. these models have undergone extensive training on large-scale multilingual datasets, enabling them to capture linguistic patterns and translations across multiple languages @xcite @xcite . in some scenarios, the translations produced by these models exhibit such remarkable quality that they pose a competitive challenge to the existing state-of-the-art translation models @xcite . the task in which we have employed and compared mbart is that of imt. this field has been under investigation since @xcite , with the first appearance of the prefix approach, and has continued to evolve ever since. numerous research branches have emerged, exploring different techniques to reduce human effort in translation. @xcite proposed a fresh approach to the behavior of translators, transitioning from correcting at the prefix level to enabling the validation of multiple segments within a single translation. other techniques aim to minimize human effort more directly, such as optimizing the utilization of user mouse actions @xcite or implementing a confidence measurement system to provide an initial evaluation of the translation @xcite . additional techniques take advantage of the ability of the imt system to guarantee perfect translations, utilizing them to enhance the translation model through active and online learning techniques @xcite @xcite . some frameworks like casmacat @xcite and transtype @xcite add a large set of these innovations in the same workplace. commercial environments like lilt and unbabel can also use interactive machine translation. in the upcoming section, we will explore the framework of prefix-based imt, which will provide a deeper understanding of how we have tailored the mbart model for this specific task. ## imt framework first, it is essential to examine the neural machine translation (nmt) framework to elucidate the modifications undertaken to adapt it for imt systems. the nmt framework, introduced by @xcite , has demonstrated its efficacy and power in recent years. its impact and effectiveness have been widely recognized in the field of mt @xcite . given the sentence x j 1 = x 1 , ..., x j from the source language x, to get the translation with the highest probability ŷ î 1 = ŷ1 , ..., ŷî from the target language y , the fundamental equation of the statistical approach to nmt would be: we have implemented this framework to be compatible with models from both opennmt-py toolkit @xcite , which we trained from scratch, and huggingface library @xcite , from where we obtained the mbart model checkpoint. we have developed an imt system in which the translation model interacts with a simulated user to generate translations. the simulated user detects the first error by comparing the generated translation with the reference word-by-word. section 4.4 of the paper describes the simulation process in more detail. ## experimental framework this section provides a comprehensive account of our experimental procedures, beginning with an overview of the evaluation metrics used to assess our proposal. we then describe the corpora utilized to train and test our models and outline the specific training procedures employed for our machine translation systems. finally, we describe the user simulation process in detail. ## results in order to perform our comparison, we evaluated the three different models in a prefix-based imt system. we aim to see a reduction in human effort when using the mbart models. in this case, we evaluate the human effort with two different metrics, the wsr, and the mar, each evaluating a different kind of effort-the effort performed by typing the words and the one by moving the mouse. in order to achieve a more precise evaluation of the models, we will consider that the effort required to type a word exceeds that of moving the mouse. furthermore, in a professional setting with real translators, there may be instances where mouse actions are performed using the keyboard, diminishing their implication. regarding the effort derived from keyboard usage, evaluated through the wsr and assumed to be more important than mouse usage, the mbart models achieve the best results. except for the en-de language pair experiment, the mbart ft model has successfully reduced the effort compared to our baseline model, suggesting that proper fine-tuning of mbart can yield better results in the field of interactive machine translation than training a model from scratch. it is also worth noting that when comparing the opennmt-py model with mbart, in cases where the target language was english, mbart achieved superior results without requiring fine-tuning. this fact aligns with the findings reported in @xcite , where the best results were