The TALP - UPC Participation in WMT 21 News Translation Task: an m BART -based NMT Approach (2021)

Abstract: This paper describes the submission to the WMT 2021 news translation shared task by the UPC Machine Translation group. The goal of the task is to translate German to French (De-Fr) and French to German (Fr-De). Our submission focuses on fine-tuning a pre-trained model to take advantage of monolingual data. We fine-tune mBART50 using the filtered data, and additionally, we train a Transformer model on the same data from scratch. In the experiments, we show that fine-tuning mBART50 results in 31.69 BLEU for De-Fr and 23.63 BLEU for Fr-De, which increases 2.71 and 1.90 BLEU accordingly, as compared to the model we train from scratch. Our final submission is an ensemble of these two models, further increasing 0.3 BLEU for Fr-De.

## introduction monolingual data is usually more abundant than parallel data as it does not need any human processing. neural machine translation (nmt) has focused traditionally on parallel data between languages and monolingual data as back-translation @xcite . this method consists of translating a monolingual corpus with an nmt system and training the model using the synthetic data. alternatively, in recent years, pre-trained models have been proposed using monolingual data as a pre-training step with self-supervised learning, before performing task-specific fine-tuning. an example of this approach is bert @xcite , which is a transformer model @xcite , pre-trained on masked language modeling and next sentences prediction on a large unlabeled corpus. while bert is used primarily for classification tasks, bart @xcite has been proposed for sequence-to-sequence tasks. bart is a transformer encoder-decoder, pre-trained as a denoising autoencoder (dae) on monolingual unlabeled text. since bart is pre-trained on mono-lingual data, an additional encoder should be introduced during fine-tuning to obtain a bilingual nmt system. mbart overcomes this restriction by being pre-trained on multilingual denoising. mbart @xcite ) is liable to finetuning on several translation directions in order to obtain a multilingual nmt system. our participation to the news translation task at wmt focuses on translating between german (de) and french (fr) in both directions, de-fr and fr-de. to accomplish this, we employ a pre-trained mbart model, and more specifically mbart50 @xcite , which is pre-trained with 50 languages. we fine-tune the mbart50 on both translation directions to obtain a single multilingual model for the task. to measure the importance of the pre-training step, we additionally train a transformer with the same architecture and hyperparameters but randomly initialized. our experiments show that the fine-tuned mbart50 can achieve better translation quality in both directions, with improvements of 2.71 for de-fr and 1.9 for fr-de. apart from fine-tuning a pre-trained model, our approach also includes extensive filtering of a large bilingual corpus to ensure high-quality training data. finally, we have considered ensembling the fine-tuned mbart50 and the trained-from-scratch transformer for our submission. this ensembling has resulted in bleu scores of 31.69 and 23.93 for de-fr and fr-de accordingly. the rest of this paper is organized as follows: in section 2 we describe the background techniques this work builds upon, multilingual nmt and mbart. in section 3 we present the datasets we used for training and the techniques applied for filtering them. in section 4 we provide the system description along with the implementation details and section 5 involves the results of our experiments. finally, in section 6 we discuss the conclusions of this work and present possible directions for further research. ## data in this section we introduce the datasets used for training our systems and we go through the data filtering process that was applied in each one of them. ## system description in this section we are going to describe the two main steps of our submission, fine-tuning of a pretrained with the provided data and ensemble of pre-trained and not pre-trained models. ## experiments and results multilingual fine-tuning. our first hypothesis is that the use of pre-trained models could improve translation performance. therefore, we compare our system with a baseline system with the same vocabulary and parameter configuration with random initialization to measure the impact of the translation step. tables 2 and 3 show the translation results for german-french and french-german translation directions, respectively. results show that fine-tuning of pre-trained models improves bleu ∆bleu baseline 28.98 -mbart50 31.69 2.71 +ensemble 31.69 0.00 ## conclusions this work describes the talp-upc system for the wmt 2021 shared news translation task for french-german and german-french. experimental results show that pre-trained models help improve translation performance in this kind of scenario, even for high-resource language pairs with millions of parallel sentences available, with 2.71 points for german-french translation direction and 1.90 points for french-german. results also show that ensembling of pre-trained and randomly initialized models can lead to minor performance improvements (up to 0.3 bleu) but not consistently on both tested languages. in future work, better results may be obtained by combining fine-tuned pre-trained models with traditional back translation. both techniques would benefit from the additional monolingual in two different aspects of the nmt model, initialization and additional training on the monolingual data provided.