Comparing Multilingual NMT Models and Pivoting (2022)

Abstract: Following recent advancements in multilingual machine translation at scale, our team carried out tests to compare the performance of multilingual models (M2M from Facebook and multilingual models from Helsinki-NLP) with a two-step translation process using English as a pivot language. Direct assessment by linguists rated translations produced by pivoting as consistently better than those obtained from multilingual models of similar size, while automated evaluation with COMET suggested relative performance was strongly impacted by domain and language family.

## materials and methods for this research, we set out to compare the performance of our company's pivoting system with open-source pre-trained multilingual models. for the pivoting system, we used general-purpose models trained to handle the different content types we have historically received in our translation work. these models were trained with between ten and thirty million sentence pairs, for fifty epochs or until the early stopping criterion was met (no improvement in validation set perplexity for 6 successive validation checkpoints). we used the transformer-base architecture with guided alignment using alignment from fast align @xcite , and to limit potential confounding factors we use english as the pivot language for all language pairs. we chose to compare our system with two m2m-100 systems (the 480 million and the 1.2 billion parameter models) @xcite and the multilingual models from helsinki-nlp @xcite . while there are other pre-trained multilingual sota models such as mt5 that could be fine-tuned for the downstream task of multilingual translation @xcite , we believe that the m2m-100 and helsinki-nlp models were easily accessible and ready to be used with no further fine-tuning. moreover, since all these systems were released around the same time, there is no published or reliable research to suggest that one model outperforms the rest. we selected seven language pairs for which we received requests in the past year but for which we had no direct bilingual model. these were the following: we also carried out a quantitative comparison for danish-spanish and swedish-french, but since we could not find linguists available for the human evaluation, we do not include the results for these two pairs of languages. in our experiments we used data from two different sources to avoid biases and compare performance across multiple domains. the first set of data was drawn from our company's previous human translation work (with care being taken to ensure that none of the data had been seen by the models during training). although the data involved a wide variety of content types, we consider these test data to be "in domain" for our engines as they were sampled from essentially the same distribution as our training data. the second set of data was extracted from leipzig's corpora collection @xcite . this collection includes monolingual corpora for 291 languages. being a monolingual database, we can be quite confident that none of those texts were used for the training of any of the engines we were comparing. we extracted text from the news domain and from the most recent year available for each source language. these test data were considered to be "out-of-domain" for our engines. since no reference translations were available for any of the input sentences, we performed automated, reference-free evaluation using comet, which was unbabel's submission for the wmt 2020 quality estimation shared task @xcite . the reason behind this decision was that this model ended on the top 5 of best models in all tasks and language pairs but one. moreover, it can be used for document-level assessment, it is easily accessible, it can be run on gpu, and it offers a command to compare multiple systems with statistical testing. additionally, we also engaged human linguists to carry out blinded direct assessment (da) for each language pair. ordinarily we would commission multiple linguists for each language pair to mitigate the effects of bias and human error. however, for these less common translation directions, only one linguist was available per language pair. nevertheless, we consider these scores reliable as the linguists were selected from our pool of certified translators for the language pair. this means that the annotators were not simply bilingual speakers, but held translation certification and actively performed translation tasks in this language pair. each linguist scored 200 segments chosen at random (100 from the in-domain data and 100 from the out-of-domain data) using a scale from 0 to 100. linguists were instructed to score the segments based on the general quality of the mt output -how well it represented the main message of the input sentence -rather than small errors which would be more heavily penalized when evaluating human translations. the scoring criteria provided to the linguists were as below: statistical significance for automated metrics was calculated using the bootstrap t-test from comet @xcite , and statistical significance for human da was determined using unpaired t-test with @xmath0.01 considered statistically significant. ## results the results of the human and automated evaluations are presented in tables 1 through 3 below. in every case, human evaluation favored the translation from the pivot system, often by a large margin. this was true for both test sets as well as the overall scores. the difference was more pronounced for language pairs from different families than for language pairs where both the source and target were european languages (average difference of 10.99 in the overall scores for fr-ja, fr-zh, and ## discussion in this study we compared translations from different models using human da and automated evaluation with the comet quality estimation model. we tested model performance using a combination of data sampled from the same distribution as our training data (in-domain) and news data (which were out-of-domain for our models used in the pivot system). single-blind human da showed a clear preference for the translations obtained through pivoting, while automated evaluation with comet was less conclusive: the domain of the content and whether or not the source and target languages belonged to the same language family appeared to have a significant effect on the scores. beyond translation quality, as a translation company we must also take other aspects into consideration. while these fall outside the scope of this work, there are many other relevant factors, such as: contrary to our intuitions prior to undertaking this study, our results suggest that pivoting is a reasonable choice for language pairs where no direct model exists, at least in terms of translation quality. the strength of the conclusions are limited by the relatively small sample size, and we anticipate these results will need to be revisited as multilingual models become more capable. moreover, fine-tuning other pre-trained multilingual models such as mt5 and comparing those with the pivoting approach could lead to different conclusions. further research is needed to more comprehensively weigh the advantages and disadvantages of replacing multiple bilingual models with a single multilingual model.