Disentangling the Roles of Target-side Transfer and Regularization in Multilingual Machine Translation (2024)

Abstract: Multilingual Machine Translation (MMT) benefits from knowledge transfer across different language pairs. However, improvements in one-to-many translation compared to many-to-one translation are only marginal and sometimes even negligible. This performance discrepancy raises the question of to what extent positive transfer plays a role on the target-side for one-to-many MT. In this paper, we conduct a large-scale study that varies the auxiliary target-side languages along two dimensions, i.e., linguistic similarity and corpus size, to show the dynamic impact of knowledge transfer on the main language pairs. We show that linguistically similar auxiliary target languages exhibit strong ability to transfer positive knowledge. With an increasing size of similar target languages, the positive transfer is further enhanced to benefit the main language pairs. Meanwhile, we find distant auxiliary target languages can also unexpectedly benefit main language pairs, even with minimal positive transfer ability. Apart from transfer, we show distant auxiliary target languages can act as a regularizer to benefit translation performance by enhancing the generalization and model inference calibration.

## introduction multilingual machine translation (mmt) enables a single model to translate among multiple language pairs by joint training @xcite . the improvements in translation quality, especially for low-resource languages, are generally attributed to transfer learning @xcite @xcite . however, mmt suffers from a performance gap where the gains for oneto-many translation are not as substantial as for many-to-one translation @xcite @xcite @xcite . empirical studies @xcite also show little to no benefit for one-to-many translation compared to their bilingual baselines, leading to the hypothesis that positive transfer does not occur on the target-side @xcite . the challenge of knowledge transfer for one-tomany translation is attributed to the inherent characteristics of translating into distinct target languages. the necessity for target language-specific representations in the translation process hinders knowledge transfer as transfer learning prefers languageinvariant representations @xcite . on the other hand, @xcite and @xcite list the increasing amounts of source language data and regularization induced by multiple target languages as possible reasons for the observed benefits in massively mmt scenarios. nevertheless, the extent to which positive knowledge transfer occurs on the target-side still remains unclear. furthermore, a comprehensive analysis of the interplay between different factors, i.e., knowledge transfer, source data size, and regularization, for one-to-many translation is lacking. this hinders the optimization of mmt performance. to understand the impact of knowledge transfer, we conduct comprehensive controlled experiments with varying target languages along two dimensions, i.e., linguistic similarity and corpus size. we select a set of bilingual out-of-english translation tasks, e.g., english into german, as main language pairs. subsequently, we add different auxiliary target language pairs to each main language pair, considering variations in auxiliary language families, written scripts, data sizes, and the number of target languages. our experimental results show a consistent positive correlation between the improvements and their translation task relatedness, i.e., increasing the amounts of similar target languages encourages more positive knowledge transfer for the main language pair than distant ones. these findings confirm the existence of knowledge transfer on the target-side and also clearly show factors that influence target-side transfer, i.e., target data size, number of translation tasks, and linguistic similarity. the performance differences induced by various target languages also indicate their varying transfer ability. apart from knowledge transfer, we find distant auxiliary target languages can also yield substantial improvements, even with minimal transfer ability. instead of transferring similar linguistic knowledge, we show that distant auxiliary target languages exhibit strong regularization abilities improving translation performance.to understand why language regularization plays a role, we show it benefits translation performance by reducing generalization errors and improving inference calibration. with introducing distant auxiliary target languages, the translation model is implicitly calibrated so that the confidences of its predictions are more aligned with the accuracies of its predictions. in this paper, we show the interplay between knowledge transfer and regularization which is visualized in figure 1 . we observe that languages that are similar to the target language, in this example belarusian, tend to benefit the target language by mostly transferring knowledge and only act as a regularizer to a very limited extent. the inverse holds for distant auxiliary languages. overall, our paper provides a more comprehensive understanding of one-to-many translation, from the perspectives of target-side transfer and regularization. first, we show how positive knowledge transfer occurs on the target-side, by varying the linguistic similarity and data size of the auxiliary target language. second, we point out the importance of regularization in one-to-many translation, by showing its effectiveness on generalization ability and inference confidence calibration. ## background in this section, we discuss some background on transfer learning and regularization in mmt. ## experimental setting model. we follow the setup of the transformer base model @xcite . more details on model hyperparameters can be found in appendix b. data. we choose three main language pairs (lps) in different language families and written scripts: english-into-german (@xmath0, englishinto-russian (@xmath1, and english-into-spanish (@xmath2. the training data for the main language pairs @xmath3, @xmath4, and @xmath5, wmt14, and wmt22, respectively. we randomly sample 100k and 1m sentence pairs from each language pair respectively to mimic ## target-side transfer in this section, we aim to estimate empirically whether target-side transfer occurs in mmt. to achieve this, we select three main language pairs, mimicking a low-resource direction: @xmath6, @xmath7, @xmath8: @xmath9. we train each main language pair with different auxiliary target languages to investigate the target-side transfer in multilingual machine translation for influencing main language pairs. we include variations in the auxiliary target language pairs, with changes in lin-3 nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1 guistic similarity, data size, and the total number of target tasks. ## language regularization the previous section shows low-and mediumresource translation tasks benefit from language regularization. in this section, we aim to further investigate the effectiveness of language regularization in one-to-many mt from two angles: generalization ability (section 5.1) and model calibration (section 5.2). in the end, we provide a simple but effective way to improve machine translation performance with the help of language regularization (section 5.3). ## conclusion in this work, we disentangle the roles of knowledge transfer and language regularization in oneto-many mmt. in contrast to previously held assumptions, we show that target-side knowledge transfer does play an important role in one-to-many mmt, influenced by several dominant factors: auxiliary target data size, linguistic similarity, and the number of auxiliary target tasks. this finding also shows that the increased amount of source data does not explain all transfer. future work can leverage this information to encourage different language pairs to have similar word representations to achieve maximum positive transfer. surprisingly, we find that using a small amount of linguistically distant auxiliary target data acts as an effective regularizer resulting in translation performance gains. this form of language regularization shows its effectiveness by benefiting generalization ability and inference calibration. our findings on language regularization provide a simple but effective way to boost the translation performance of real-world low-and medium-resource language pairs, especially when similar target languages are not available. future work can further explore the optimization of multilingual training by leveraging distant auxiliary data.