Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance (2023)

Abstract: Multilingual Neural Machine Translation (MNMT) facilitates knowledge sharing but often suffers from poor zero-shot (ZS) translation qualities. While prior work has explored the causes of overall low zero-shot translation qualities, our work introduces a fresh perspective: the presence of significant variations in zero-shot performance. This suggests that MNMT does not uniformly exhibit poor zero-shot capability; instead, certain translation directions yield reasonable results. Through systematic experimentation, spanning 1,560 language directions across 40 languages, we identify three key factors contributing to high variations in ZS NMT performance: 1) target-side translation quality, 2) vocabulary overlap, and 3) linguistic properties. Our findings highlight that the target side translation quality is the most influential factor, with vocabulary overlap consistently impacting zero-shot capabilities. Additionally, linguistic properties, such as language family and writing system, play a role, particularly with smaller models. Furthermore, we suggest that the off-target issue is a symptom of inadequate performance, emphasizing that zero-shot translation challenges extend beyond addressing the off-target problem. To support future research, we release the data and models as a benchmark for the study of ZS NMT.

## introduction multilingual neural machine translation (mnmt) has shown great potential in transferring knowledge across languages, but often struggles to achieve satisfactory performance in zero-shot (zs) translations. prior efforts have been focused on investigating causes of overall poor zero-shot performance, such as the impact of model capacity @xcite , initialization @xcite @xcite , and how model forgets language labels can affect zs performance @xcite . in contrast, our work introduces a fresh perspective within zero-shot nmt: the presence of high variations in the zero-shot performance. this phenomenon suggests that certain zs translation directions can closely match supervised counterparts, while others exhibit substantial performance gaps. we recognize this phenomenon holds for both english-centric systems and systems going beyond english-centric data (e.g.: m2m-100 models). this raises the question: which factors contribute to variations in the zero-shot translation quality? through systematic and comprehensive experimentation involving 1,560 language directions spanning 40 languages, we identify three key factors contributing to pronounced variations in zeroshot nmt performance: 1) target side translation capacity, 2) vocabulary overlap, and 3) linguistic properties. more importantly, our findings are general regardless of the resource level of languages and hold consistently across various evaluation metrics, spanning word, sub-word, character, and representation levels. drawing from our findings, we offer potential insights to enhance zero-shot nmt. our investigation begins by assessing the impact of supervised translation capability on zero-shot performance variations. this is achieved by decomposing the unseen zs direction (@xmath0. for instance, @xmath1: @xmath2. our findings show the target side translation quality significantly impacts zs performance and can explain the variations the most. surprisingly, the source side translation quality has a very limited impact. moreover, our analysis demonstrates the substantial impact of linguistic properties, i.e., language family and writing system, in elucidating the variations in zero-shot performance. figure 1 highlights this conclusion by showing the much stronger zs performance of resource-rich zs directions with similar linguistic properties compared to other directions. intriguingly, our investigation also shows that the impacts of linguistic properties are more pronounced for smaller models. this suggests that larger models place less reliance on linguistic similarity when engaging in zs translation, expanding more insights upon prior research about the impact of model capacity on zs nmt @xcite . furthermore, we found the language pair with higher vocabulary overlap consistently yields better zero-shot capabilities, suggesting a promising future aspect to improve the zs nmt. in addition, while @xcite asserts the off-target issue as a primary cause that impairs the zero-shot capability, we conclude that the off-target issue is more likely to be a symptom of poor zero-shot translation qualities rather than the root cause. this is evident by small off-target rates (smaller than 5%) not necessarily resulting in high zs capabilities. lastly, we argue that prior research on zero-shot nmt is limited by focusing only on the 1% of all possible zs combinations @xcite @xcite or prioritizing resource-rich language pairs @xcite @xcite @xcite . to overcome these limitations, we create the ec40 mnmt dataset for training purposes and utilize multi-parallel test sets for fair and comprehensive evaluations. our dataset is the first of its kind considering real-world data distribution and diverse linguistic characteristics, serving as a benchmark to study zs nmt. ## variations in zero-shot nmt does pre-training matter? @xcite ; @xcite ; @xcite have shown that pre-trained seq2seq language models can help alleviate the issue of forgetting language ids often observed in transformer models trained from scratch, leading to improvements in zero-shot performance. however, our results show an interesting finding: when the mnmt model size matches that of the pre-trained model, the benefits of pre-chrf++ en-cent. sup. zs en-cent. sup. zs averaged performance mt-large 27.7 11.9 6.2 52.5 34.1 23.2 m2m100-419m 24.5 15.7 7.4 49.2 39.3 25.8 m2m100-1.2b 29.0 18.9 9.9 52.9 42.8 29.5 ## factors in the zero-shot variations we investigate factors that might contribute to variations in zero-shot directions: 1) english translation capability 2) vocabulary overlap 3) linguistic properties 4) off-target issues. for consistency, we use our best model (mt-large) in the following analyses, unless mentioned otherwise. for simplicity, we denote the zero-shot direction as @xmath5, where src and tgt represent the source and target language respectively. in this chapter, we present all analysis results using spbleu and provide results based on other metrics in the appendix for reference. ## from causes to potential remedies in this section, we summarize our findings and offer insights, building upon the previous observations. enhance target side translation we identified that the quality of target side translation (@xmath7. to this end, future research should explore more reliable approaches to enhance the target side translation capability. one practical promising direction is the use of back-translation @xcite focusing more on improving out-of-english translations. similarly, approaches like multilingual regularization, sampling, and denoising are worth exploring to boost the zero-shot translation directions. focus more on distant pairs we recognize that distant language pairs constitute a significant percentage of all zero-shot directions, with 61% involving different scripts and 81% involving different language families in our evaluations. our anal-ysis reveals that, especially with smaller models, distant pairs exhibit notably lower zero-shot performance compared to closer ones. consequently, enhancing zero-shot performance for distant pairs is a key strategy to improve overall capability. an unexplored avenue for consideration involves multisource training @xcite using romanization @xcite , with a gradual reduction in the impact of romanized language. encourage cross-lingual transfer via vocabulary sharing furthermore, we have consistently observed that vocabulary overlap plays a significant role in explaining zero-shot variation. encouraging greater cross-lingual transfer and knowledge sharing via better vocabulary sharing has the potential to enhance zero-shot translations. previous studies @xcite have shown promising results in improving multilingual translations by augmenting multilingual vocabulary sharing. additionally, cross-lingual pre-training methods utilizing multi-parallel dictionaries have demonstrated improvements in word alignment and translation quality @xcite . ## conclusion in this work, we introduce a fresh perspective within zero-shot nmt: the presence of high variations in the zero-shot performance. we recognize that our investigation of high variations in zero-shot performance adds an important layer of insight to the discourse surrounding zero-shot nmt, which provides an additional perspective than understanding the root causes of overall poor performance in zero-shot scenarios. we first show the target side translation quality significantly impacts zero-shot performance the most while the source side has a limited impact. furthermore, we conclude higher vocabulary overlap consistently yields better zero-shot performance, indicating a promising future aspect to improve zero-shot nmt. moreover, linguistic features can significantly affect zs variations in the performance, especially for smaller models. additionally, we emphasize that zero-shot translation challenges extend beyond addressing the off-target problem. we release the ec-40 mnmt dataset and model checkpoints for future studies, which serve as a benchmark to study zero-shot nmt. in the future, we aim to investigate zero-shot nmt from other views, such as analyzing the discrepancy on the representation level.