Causes and Cures for Interference in Multilingual Translation (2023)

Abstract: Multilingual machine translation models can benefit from synergy between different language pairs, but also suffer from interference. While there is a growing number of sophisticated methods that aim to eliminate interference, our understanding of interference as a phenomenon is still limited. This work identifies the main factors that contribute to interference in multilingual machine translation. Through systematic experimentation, we find that interference (or synergy) are primarily determined by model size, data size, and the proportion of each language pair within the total dataset. We observe that substantial interference occurs mainly when the model is very small with respect to the available training data, and that using standard transformer configurations with less than one billion parameters largely alleviates interference and promotes synergy. Moreover, we show that tuning the sampling temperature to control the proportion of each language pair in the data is key to balancing the amount of interference between low and high resource language pairs effectively, and can lead to superior performance overall.

## introduction multilingual machine translation models can benefit from transfer between different language pairs (synergy), but may also suffer from interference @xcite @xcite . while there are methods to reduce interference and achieve better performance @xcite @xcite , such approaches are often compute intensive, and do not always work @xcite . in this work, we demonstrate that interference in multilingual translation largely occurs when the model is very small compared to the abundance of training data, and that the simple principled approach of enlarging the model and tuning the data sampling temperature provides a consistent solution to the interference problem that can even promote synergy. this work methodically deduces the most simple ways of reducing interference in multilingual translation. we begin by inquiring what are the dominant factors that may interfere with learning to translate a particular language pair of focus @xmath0, in the context of learning a multilingual translation model with many different language pairs. controlled experiments show that besides model size and number of @xmath1, the main factor that correlates with the level of interference is the proportion of focus pair examples (@xmath2. surprisingly, aspects like language similarity or number of translation directions have a much smaller effect. in model and data scaling experiments, we observe that interference mainly occurs in extreme parameter poverty, when the language pair of focus is data-rich, but has to "share" a crowded parameter space with large quantities of other data. enlarging the model to standard model sizes in machine translation literature alleviates interference and even facilitates synergy. for context, given a language pair of 15m sentence pairs that accounts for 20% of the total training data (75m), we observe severe levels of interference with 11m-and 44m-parameter transformers, but no interference when scaling the model to 176m parameters (the "big" model of @xcite ) and significant synergy with 705m parameters. interestingly, when the model is large enough, we find that increasing the amount of non-focus data to a certain point can further increase synergy. finally, given the evidence that data sizes and ratios strongly correlate with interference, we experiment with a natural lever that controls the proportion of each dataset in the overall mix in the simplest way: sampling temperature. indeed, we find that calibrating the distribution of language pairs via temperature can substantially reduce the amount of interference in both high-and lowresource language pairs. our results demonstrate the importance of tuning the temperature hyperparameter in multitask training, and suggest that previously reported accounts of severe interference in multilingual translation models might stem from suboptimal hyperparameter configurations. ## measuring interference we assume a common multilingual translation setup that involves l language pairs @xmath3, where the source is always the same language s (english), and the target language t varies (english-to-many), or vice versa (many-to-english). the overall training data is a union of these training subsets, we note their sizes by d @xmath4. sampling a training example x follows the distribution: where t is the temperature hyperparameter @xcite . @xmath5, 0 < @xmath6, and @xmath7. we mostly focus on the english-to-many setting in which interference is more apparent. 1 we define interference as a negative interaction between different translation directions in a multilingual translation model. it is measured for a specific translation direction @xmath8: negative values of i @xmath9, while positive values indicate synergy. ## experimental setup models we train encoder-decoder transformer @xcite throughout our experiments. we use the original foot_1 transformer-base and transformer-big variants, as well as a smaller and a larger versions by adjusting the width of the architecture ( ## conclusion this work examines the dominant factors that influence interference in multilingual machine translation. namely, the model size, the amount of parallel data for the focus language pair, and the proportion of examples from the focus language pair with respect to the total data seen during training. while specialized multitask techniques are sometimes demonstrated on small transformer models, we find that a standard baseline model of 176m parameters reduces the interference problem significantly, and further scaling up results in synergy among the different language pairs. we further demonstrate the importance of tuning the temperature at which different language pairs are sampled during training; while existing literature largely relies on high temperatures, which indeed improve low-resource performance in parameter-poor settings, larger models benefit from a more natural distribution that reflects the raw training data. these simple strategies for addressing interference call into question the necessity and perhaps even the validity of recentlyproposed complex anti-interference methods and reaffirm the tried-and-true method of increasing model capacity to accommodate for higher data diversity.