How Far can 100 Samples Go? Unlocking Zero-Shot Translation with Tiny Multi-Parallel Data (2024)

Abstract: Zero-shot translation aims to translate between language pairs not seen during training in Multilingual Machine Translation (MMT) and is widely considered an open problem. A common, albeit resource-consuming, solution is to add as many related translation directions as possible to the training corpus. In this paper, we show that for an English-centric model, surprisingly large zero-shot improvements can be achieved by simply fine-tuning with a very small amount of multi-parallel data. For example, on the EC30 dataset, we obtain up to +21.7 ChrF++ non-English overall improvements (870 directions) by using only 100 multi-parallel samples while preserving English-centric translation quality. This performance exceeds M2M100 by an average of 5.9 ChrF++ in the involved non-English directions. When investigating the size effect of fine-tuning data on translation quality, we found that already a small, randomly sampled set of fine-tuning directions is sufficient to achieve comparable improvements. The resulting non-English performance is close to the complete translation upper bound. Even in a minimal setting—fine-tuning with only one single sample—the well-known off-target issue is almost completely resolved, explaining parts—but not all—of the observed improvements in translation quality.

## introduction the zero-shot capability shown by multilingual machine translation (mmt) @xcite is of considerable significance, particularly in the context of translating between low-resource or distant language pairs. however, even for systems trained on large-scale data, the zero-shot performance is still far from sufficient (tan and monz, 2023), especially when scaling up the number of involved languages. substantial efforts (zhang et al., et al., 2023) have been dedicated to improving the zero-shot capabilities of models trained on readily available, predominantly english-centric corpora. to fully cover translation directions, freitag and firat (2020) propose mining multi-parallel (multiway aligned) samples to extend the training set from english-centric to a complete multilingual one as shown in figure 1-(b ). non-english translation quality in this setting indeed increases substantially. however, such a setting is far away from real-world practice when scaling up. as shown in freitag and firat (2020), to solely extend the training set from bilingual aligned to all languages (6way) involved in their case, the amount of available data drops from 123m to 10k, which is insufficient. to reflect worldwide translation needs, fan et al. @xcite build and open-source a training dataset covering 100 languages through industry-scale mining. in addition to english-centric data, supervised data for thousands of bridge language pairs is mined and included, organized based on language families. the mmt model trained on the resulting data, m2m100, exhibits clear improvements in many non-english directions. this work drives a simple but resource-consuming solution for real-world demand: mining as much training data as possible to bridge non-english language pairs, at least at the language family level. in this paper, we take a step back and again look at the readily available english-centric model. we empirically show that the zero-shot ability of an english-centric model can be easily unlocked via fine-tuning with a tiny amount of multi-parallel data, which is much simpler and more efficient than the extensive training data mining and bridging done by earlier work. e.g., after fine-tuning the english-centric system trained on the ec30 dataset with 100 multi-parallel samples from ntrex (federmann et al., 2022), we observed 21.7 chrf++ zero-shot improvements across 870 language directions. this performance surpasses m2m100 (fan et al., 2021) by 5.9 chrf++ (see given the high efficiency and practicality, we encourage the community to consider fine-tuning with tiny readily available multi-parallel data, like ntrex @xcite , as a strong base-line for zero-shot translation. our findings challenge some prevailing assumptions about mmt systems based on the clear evidence proposed in this paper that: 1) the off-target issue is likely overestimated and can be easily handled, and 2) the potential of single-language-centric (like english) mmt models is substantial and often overlooked. we hope these insights will inspire new discussions and explorations within the community. ## related work the zero-shot translation capability of mmt is associated with multilingualism, following the hypothesis of universal representation or interlingua. this paper focuses on a data-centric approach for comprehensively improving zero-shot performance. we empirically show that a well-trained english-centric model can be easily boosted for overall zero-shot capability via fine-tuning with minimal data, even if only covering a small portion of translation directions (10%). this allows us to leverage multi-parallel data, which is hard to obtain in large quantities, resulting in a highly efficient and practical solution to overall zero-shot translation. we note that maillard et al. (2023) also show that integrating small high-quality data (6k samples) into the training corpus can have a big impact on low-resource translation systems, especially when combined with back translation @xcite . however, we contend that the reasons for the effectiveness differ: 1) as shown in section 3.6, the substantial improvements persist when using data built from the training set, where the influence from domain or quality level is eliminated, and 2) our method can work with extremely minimal fine-tuning data (100 or even a single sample). ## experiments in this paper, we propose a simple approach that largely improves overall zero-shot translation quality, i.e., fine-tuning an english-centric model in multiple directions with sentence pairs constructed from small, readily available multi-parallel datasets, like ntrex @xcite . we refer to such a process as "fine-tuning with multiparallel data" or "multi-parallel fine-tuning". ## conclusion in this paper, we show that the zero-shot performance of an english-centric mmt model can be easily boosted by a tiny amount of multi-parallel data. on ec30, +21.7 chrf++ average gains can be achieved by fine-tuning using 100 samples from ntrex, meanwhile preserving the english-centric performance, see section 3.4. more surprisingly, we show that fine-tuning on a small portion (10%) of directions can achieve comparable improvements to full-direction fine-tuning, see section 3.5, which are even close to the ideal but impractical upper-bound model, see section 3.6. in terms of using language tags, we show that fine-tuning can address the two-tag model's performance degradation in zero-shot directions (wu et al., 2021). moreover, the final performance substantially surpasses that of the one-tag model across multiple metrics, see section 3.4. we also question earlier findings (zhang et al., 2020; yang et al., 2021; sennrich et al., 2023; chen et al., 2023) that consider the off-target issue as a challenging problem for mmt. this paper shows that the off-target issue can be easily addressed by fine-tuning with even a single sample. moreover, we also observe clear gains on the already on-target translations after fine-tuning with a few samples, implying that the off-target issue is not the primary source, but more like a symptom, of low zero-shot quality, see section 4.1. by decoupling the impacts of multi-parallel signals, we demonstrate that fine-tuning data does not have to be multi-parallel to achieve significant improvements, see section 4.2. however, we still recommend using multi-parallel data, as it is already sufficient, albeit small-scale, and convenient for building fine-tuning data in each involved direction. lastly, we shed some light on the impact of different types of signals from the fine-tuning data on the final performance, see section 4.3. given the clear advantages of our proposed method, we encourage the community to consider fine-tuning as a strong baseline for zero-shot translation, especially in the two-tag setting. furthermore, we suggest that the community reconsider the real bottlenecks of mmt systems in light of the evidence presented in this paper: 1) the off-target issue is overestimated and can be easily managed, and 2) the potential of single-language-centric (like english) mmt models, is substantial and often overlooked.