Improving Zero-Shot Translation by Disentangling Positional Information (2021)

Abstract: Multilingual neural machine translation has shown the capability of directly translating between language pairs unseen in training, i.e. zero-shot translation. Despite being conceptually attractive, it often suffers from low output quality. The difficulty of generalizing to new translation directions suggests the model representations are highly specific to those language pairs seen in training. We demonstrate that a main factor causing the language-specific representations is the positional correspondence to input tokens. We show that this can be easily alleviated by removing residual connections in an encoder layer. With this modification, we gain up to 18.5 BLEU points on zero-shot translation while retaining quality on supervised directions. The improvements are particularly prominent between related languages, where our proposed model outperforms pivot-based translation. Moreover, our approach allows easy integration of new languages, which substantially expands translation coverage. By thorough inspections of the hidden layer outputs, we show that our approach indeed leads to more language-independent representations.

## introduction multilingual neural machine translation (nmt) system encapsulates several translation directions in a single model @xcite . these multilingual models have been shown to be capable of directly translating between language pairs unseen in training @xcite . zero-shot translation as such is attractive both practically and theoretically. compared to pivoting via an intermediate language, the direct translation halves inference-time computa-encoder a big cat encoder un gato grande h 1 h 2 h 3 h 1 h 3 h 2 figure 1 : an example of language-specific encoder outputs as a results of the strong positional correspondence to input tokens (even assuming the word embeddings are cross-lingually mapped). despite the potential benefits, achieving highquality zero-shot translation is a challenging task. prior works @xcite @xcite have shown that standard systems tend to generate poor outputs, sometimes in an incorrect target language. it has been further shown that the encoder-decoder model captures spurious correlations between language pairs with supervised data @xcite . during training, the model only learns to encode the inputs in a form that facilitates translating the supervised directions. the decoder, when prompted for zero-shot translation to a different target language, has to handle inputs distributed differently from what was seen in training, which inevitably degrades performance. ideally, the decoder could translate into any target language it was trained on given an encoded representation independent of input languages. in practice, however, achieving a language-agnostic encoder is not straightforward. in a typical transformer encoder @xcite , the output has a strong positional correspondence to input tokens. for example in the english sentence in figure 1 , encoder outputs h 1,2,3 correspond to "a", "big", "cat" respectively. while this property is essential for tasks such as sequence tagging, it hinders the creation of languageindependent representations. even assuming that the input embeddings were fully mapped on a lexical level (e.g. "cat" and "gato" have the same embedding vector), the resulting encoder outputs are still language-specific due to the word order differences. in this light, we propose to relax this structural constraint and offer the model some freedom of word reordering in the encoder already. our contributions are as follow: ## disentangling positional information zero-shot inference relies on a model's generalizability to conditions unseen in training. in the context of zero-shot translation, the input should ideally be encoded into an language-agnostic representation, based on which the decoder can translate into any target language required, similar to the notion of an interlingua. nevertheless, the ideal of "any input language, same representation" cannot be easily fulfilled with a standard encoder, as we have shown in the motivating example in figure 1 . we observe that the encoder output has a positional correspondence to input tokens. formally, given input token embeddings (x 1 , . . . , x n ), in the encoder output (h 1 , . . . , h n ), the i-th hidden state h i mostly contains information about x i . while this structure is prevalent and is indeed necessary in many tasks such as contextual embedding and sequence tagging, it is less suitable when considering language-agnostic representations. as a sentence in different languages are likely of varying lengths and word orders, the same semantic meaning will get encoded into different hidden state sequences. there are two potential causes of this positional correspondence: residual connections and encoder self-attention alignment. we further hypothesize that, by modifying these two components accordingly, we can alleviate the positional correspondence. specifically, we set one encoder layer free from these constraints, so that it could create its own output ordering instead of always following a one-to-one mapping with its input. ## experimental setup our experiments cover high-and low-resource languages and different data conditions. we choose an english-centered setup, where we train on x ↔ en parallel data, and test the zero-shot translation between all non-english languages. this scenario is particularly difficult for zero-shot translation, as half of the target-side training data is in english. indeed, recent works @xcite have outlined downsides of the englishcentered configuration. nevertheless, intrigued by the potential of covering n 2 translation directions by training on 2n directions, we still explore this scenario. ## results our approach substantially improves zero-shot translation quality, as summarized in in the very challenging case of pmindia (row 5), while removing residual does improve the zeroshot performance, the score of 2.3 indicates that the outputs are still far from being useful. nonetheless, we are able to remedy this by further regularization as we will present in subsection 4.1. contrary to the large gains by removing residual connections, the attention query modification is not effective when combined with residual removal. this suggests that the primary source of positionspecific representation is the residual connections. moreover, by contrasting row 2 and 3 of ## discussions and analyses to see beyond bleu scores, we first analyze how much position-and language-specific information is retained in the encoder hidden representations before and after applying our approaches. we then study circumstances where zero-shot translation tends to outperform its pivoting-based counterpart. lastly, we discuss the robustness of our approach to the impact of different implementation choices. ## related work initial works on multilingual translation systems already showed some zero-shot capability @xcite . since then, several works improved zero-shot translation performance by controlling or learning the level of parameter sharing between languages @xcite . recently, models with full parameter sharing have gained popularity, with massively multilingual systems showing encouraging results @xcite @xcite . besides advantages such as compactness and ease of deployment, the tightly-coupled model components also open up new questions. one question is how to form language-agnostic representations at a suitable abstraction level. in this context, one approach is to introduce auxiliary training objectives to encourage similarity between the representations of different languages @xcite . in this work we took a different perspective: instead of introducing additional objectives, we relax some of the pre-defined structure to facilitate language-independent representations. another line of work on improving zero-shot translation utilizes monolingual pretraining @xcite or synthetic data for the zero-shot directions by generated by backtranslation @xcite . with both approaches, the zero-shot directions must be known upfront in order to train on the corresponding languages. in comparison, our adaptation procedure offers more flexibility, as the first training step remains unchanged regardless of which new language is later finetuned on. this could be suitable to the practical scenario of later acquiring data for the new language. our work is also related to adaptation to new languages. while the existing literature mostly focused on adapting to one or multiple supervised training directions @xcite @xcite @xcite , our focus in this work is to rapidly expand translation coverage via zero-shot translation. while our work concentrates on an englishcentered data scenario, another promising direction to combat zero-shot conditions is to enrich available training data by mining parallel data between non-english languages @xcite . on a broader scope of sequenceto-sequence tasks, @xcite enforced encoder-decoder modularity for speech recognition. the goal of modular encoders and decoders is analogous to our motivation for zero-shot translation. ## conclusion in this work, we show that the positional correspondence to input tokens hinders zero-shot translation. specifically, we demonstrate that: 1) the encoder outputs retain word orders of source languages; 2) this positional information reduces cross-lingual generalizability and therefore zero-shot translation quality; 3) the problems above can be easily alleviated by removing the residual connections in one middle encoder layer. with this simple modification, we achieve improvements up to 18.5 bleu points on zero-shot translation. the gain is especially prominent in related languages, where our proposed model outperforms pivot-based translation. our approach also enables integration of new languages with little parallel data. similar to interlingua-based models, by adding two translation directions, we can increase the translation coverage by 2n language pairs, where n is the original number of languages. in terms of model representation, we show that the encoder outputs under our proposed model are more languageindependent both on a sentence and token level.