Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation (2020)

Abstract: Zero-shot neural machine translation is an attractive goal because of the high cost of obtaining data and building translation systems for new translation directions. However, previous papers have reported mixed success in zero-shot translation. It is hard to predict in which settings it will be effective, and what limits performance compared to a fully supervised system. In this paper, we investigate zero-shot performance of a multilingual EN<->FR,CS,DE,FI system trained on WMT data. We find that zero-shot performance is highly unstable and can vary by more than 6 BLEU between training runs, making it difficult to reliably track improvements. We observe a bias towards copying the source in zero-shot translation, and investigate how the choice of subword segmentation affects this bias. We find that language-specific subword segmentation results in less subword copying at training time, and leads to better zero-shot performance compared to jointly trained segmentation. A recent trend in multilingual models is to not train on parallel data between all language pairs, but have a single bridge language, e.g. English. We find that this negatively affects zero-shot translation and leads to a failure mode where the model ignores the language tag and instead produces English output in zero-shot directions. We show that this bias towards English can be effectively reduced with even a small amount of parallel data in some of the non-English pairs.

## introduction zero-shot translation has first been introduced by @xcite and refers to the ability of a multilingual nmt model to translate between all its source and target languages, even those pairs for which no parallel data was seen in training. in the simplest setting, all parameters in the network are shared between the different languages and the translation is guided only by special tags to indicate the desired output language @xcite . while this capability is attractive because it is an alternative to building n 2 dedicated translation systems to serve n languages, performance on zero-shot pairs tends to lag behind pivot translation. recent papers, such as @xcite , @xcite and @xcite , have suggested training techniques to improve the generalization to unseen language pairs, but performance varies considerably across settings. in this paper, we examine in detail the behavior of the multilingual model proposed by @xcite on zero-shot translation directions. our experiments show the following: overall, we observe improvements of 8.1 bleu (15.9→24.0) on 6 zero-shot directions with simple changes to the multilingual training setup. ## related work our experiments are based on the multilingual model proposed by @xcite : a single model is trained on multiple language pairs with a standard encoder-decoder architecture, all parameters in the network are shared for all languages, including the vocabulary. an artificial target language token determines the output language. we prefix this special token to the source sentence as in @xcite . the major advantage of this model lies in its simplicity, since it does not require changing the architecture or training objective. several recent studies have explored approaches to improve generalization to zero-shot language pairs, for example through semi-supervised training @xcite @xcite or alignment of encoder representations @xcite . our study is concerned with data conditions that enable zero-shot generalization for multilingual nmt, specifically preprocessing and data settings. while initial work used separate encoders and decoders for different languages @xcite , sharing of encoder and decoder parameters was established by @xcite ; @xcite and has since been widely adopted. @xcite use a shared subword segmentation model across languages, and this strategy is followed by later work (e.g. @xcite . @xcite do not share embeddings across languages, but use language-specific codes. we will show that both strategies cause errors. in terms of data settings, the number of languages involved in multilingual models has increased from 3-4 @xcite to over 100 @xcite . the most popular setup are english-centric datasets, where the model is trained on translations between english and a number of other languages. a multiway parallel corpus between 5 languages has been provided for the iwslt17 multilingual task @xcite . results on this dataset show strong zero-shot generalization, close or even exceeding the supervised condition @xcite , but multi-way parallel corpora are only available in small amounts and specific domains, so we investigate alternatives to english-centric models that do not rely on multi-way parallelism. ## data and models following @xcite , our baseline setup is english-centric. for training, we use 5 million parallel sentences per language pair for english↔{french,czech,german,finnish} from wmt @xcite . for all zero-shot language pairs, we sample test sets from opus @xcite , see to indicate the target language, we prefix a language tag on the source side (e.g. <2en>). following @xcite , we segment all data with a byte-pair encoding model trained jointly on the training data in all five languages @xcite , with a threshold of 32k bpe operations. all our systems are base transformers @xcite implemented in sockeye @xcite , trained with early stopping based on bleu on a development set that consists in equal parts of parallel sentences from all trained translation directions. see appendix a and b for training details. ## baseline experiments bleu foot_0 on zero-shot pairs is relatively unstable, see fig. 1 : while bleu on the trained pairs increases steadily during training (dashed lines), performance on unseen language pairs fluctuates considerably, as also observed by @xcite . furthermore, multiple training runs result in relatively large differences in bleu on the zero-shot directions. across three training runs, average bleu varies up to 0.24 points on trained language pairs (standard deviation: 0.12), but up to 6.28 bleu on zero-shot pairs (standard deviation: 3.14) -see as an alternative to zero-shot translation, we report results obtained via pivot translation through english (e.g. german-english-czech). on our data set, this approach works better than zero-shot translation. pivot translation is stable across training runs, with a standard deviation of 0.19. for model a), we remove any potential vocabulary overlap by adding a language identifier to each subword. for instance, consider the preposition in in german and english: instead of one token in, the network vocabulary has an entry for in#de# and an additional entry for in#en#. this corresponds to the language-specific coding introduced by @xcite . for model b), we split words with the same language-specific bpe models as for a), but we allow vocabulary overlap, i.e. homographic forms in different languages are represented by a single entry in the network's vocabulary. this results in a vocabulary size of ∼50k for model a), whereas for model b), the vocabulary amounts to a total of ∼36k subwords. model b), with language-specific bpe and overlapping vocabularies, represents a compromise between a fully shared representation and fully language-specific coding. we hypothesize that allowing some vocabulary overlap helps aligning the representation between sentences with the same 3 this essentially means that the strict evaluation gives us a more realistic estimate of the translation quality we can expect if the source and target language do not happen to share word forms, e.g. languages in different scripts. ## multi-bridge models a common issue in zero-shot translation is output in the wrong language. previous work has addressed this with semi-supervised training @xcite @xcite . we explore whether the recent trend to train english-centric models is to blame for this behavior. in most cases, the model will wrongly produce english output in zero-shot directions, since for all non-english languages, english was the only target language seen in training. we suspect that adding even a small amount of parallel data in pairs without english will improve the results in ## comparison to back-translation and encoder alignment previous work on the zero-shot generalization of multilingual nmt systems has proposed backtranslation or changes to the training objective to improve translation in unsupervised directions. while we consider our proposed solutions on the data side to be complementary, and easier to adopt widely, we still want to discuss the question how our solutions compare to previous work, and whether they can be combined. ## conclusions we analyze the importance of shared subwords in multilingual models and find that language-specific bpe segmentation helps to reduce the amount of untranslated segments in zero-shot directions. furthermore, we explore whether the tendency to produce the wrong output language can be attributed to using english as the only bridge language, and show that even with a small amount of additional training data in non-english language pairs, generalization to unseen translation directions improves as the model is less likely to produce output in the wrong language. compared to previous work, the methods we propose are easier to implement, since they only concern data collection and pre-processing, and result in higher gains for zero-shot directions. they are also compatible in principle with approaches that introduce new training objectives or model modifications, and we report best results when finetuning a multi-bridge model with back-translation for zero-resource translation directions. for future work, we are interested in testing the effects of subword regularization @xcite on zero-shot translation performance, and scaling multi-bridge setups to massively multilingual settings.