Adapting to Non-Centered Languages for Zero-shot Multilingual Translation (2022)

Abstract: Multilingual neural machine translation can translate unseen language pairs during training, i.e. zero-shot translation. However, the zero-shot translation is always unstable. Although prior works attributed the instability to the domination of central language, e.g. English, we supplement this viewpoint with the strict dependence of non-centered languages. In this work, we propose a simple, lightweight yet effective language-specific modeling method by adapting to non-centered languages and combining the shared information and the language-specific information to counteract the instability of zero-shot translation. Experiments with Transformer on IWSLT17, Europarl, TED talks, and OPUS-100 datasets show that our method not only performs better than strong baselines in centered data conditions but also can easily fit non-centered data conditions. By further investigating the layer attribution, we show that our proposed method can disentangle the coupled representation in the correct direction.

## introduction training multilingual neural machine translation (mnmt) system requires enormous number of parameters and resources, but the zero-shot translation, namely translating unseen language pairs during training, has shown the potential to simplify the mnmt @xcite . @xcite has shown that adding language tokens, e.g. <en>, at the beginning of a sentence allows the model to build cross-linguistic representation by treating the token as translation instruction specifying target language. however, the zero-shot translation is always unstable. one possibility causing the instability of zero-shot translation is spurious correlation @xcite . the target linguistic representation captured by the model is directly and strictly dependent on encoded source linguistic information instead of learning specific representations for source and target language, then combining independent linguistic representations to generate results. prior works @xcite @xcite @xcite indicated that the spurious correlation is caused by the centered data condition in which multilingual data is constructed by bridging a central language, e.g. english, to other non-centered languages. the central language will dominate the representation in the mnmt model to degenerate the information specific to non-centered languages since multilingual data comprises a set of bilingual data constructed by coupling non-centered languages with the central language. however, the non-centered data condition without any central language is also unstable in zero-shot translation. 2 therefore, simply attributing the instability of zeroshot translation to the central language cannot fit all cases of zero-shot translation. we move the perspective from the domination of the central language to the weakness of noncentered languages. the problems of zero-shot translation could be attributed to the strict dependence of non-centered languages. specifically, a non-centered language would strictly depend on another language as a strongly related language pair to prohibit learning robust and independent translation instructions for zero-shot translation. under this hypothesis, the centered data condition is a special case of this description, because all noncentered languages depend on the central language. in this light, a key to improving zero-shot translation is disentangling non-centered languages from the strict dependence which is built in training. specifically, we model extra language-specific (ls) components @xcite @xcite adapting to non-centered languages in a mixing shared and ls information mode @xcite , our objective is to enhance the weak representations for assisting the balance of crosslinguistic representation in shared information container to improve the quality of translation @xcite ; @xcite . furthermore, the mixing mode can decrease the complexity of ls modeling since we treat the representation space of the mnmt model as the combination of shared and ls information, and we no longer build the independent representation space for each language @xcite . in this motivation, we propose a simple, lightweight yet effective method to augment feedforward network of transformer @xcite by ls components adapting to non-centered languages. our contributions are as follows: ## related work initially, @xcite laid the foundation of zero-shot translation which endorses training the mnmt model under the centered data condition and put forward the thinking about the instability of data conditions on zero-shot translation. on this basis, @xcite also showed that the performance of zero-shot translation is sensitive to parameters for initialization, which is another cause of instability. in this paper, we systematically described this instability (section 5.1) and tested it experimentally. in the early stages, @xcite pointed out that increasing corpus size can effectively improve zero-shot translation. however, the spurious correlation @xcite means that unreasonably increasing training data could degenerate the zero-shot translation due to strict dependence. since then, the concern of the centered data condition was started to be discussed by several different strategies. @xcite ; @xcite augmented the training data to make all languages interconnected, which will result in an excessive increase in training costs. @xcite explored incrementally training the mnmt model by monolingual data, and @xcite ; @xcite generated synthetic data for the zero-shot directions by backtranslation. these methods transformed the zero-shot task to zeroresource task. another line of work on improving zero-shot translation is to adjust the learning of representations in the mnmt model. lu et al. (2018 our work is based on ls modeling which is the heuristic variation of mixture-of-experts model @xcite , because it aims to build extra components as experts to directionally improve linguistic features. @xcite and @xcite built ls encoder or decoder, but multi-encoder/decoder architecture has too many parameters. @xcite divided neural cells into ls parts and @xcite divided ls subnets from the model, but these methods limited the learning capacity. @xcite and @xcite added ls adapters on the end of encoder and decoder and fine tuned for ls representations. @xcite ; @xcite explored the paradigm of constructing ls components to assist the shared information. however, extra components always increase the cost of modeling significantly when languages existed too much. the investigation about the importance of ls information specified to target language @xcite @xcite enlightens us to limit the improving ls information in the decoding process to achieve lightweight ls modeling. ## central language aware multilingual neural machine translation we employ transformer @xcite as the backbone to construct our architecture. consider a set of m languages @xmath0, l 2 , . . . , l m }, we assign the first language l 1 as the central language l c . the non-centered set is the subset of l, that is l ′ = {l 2 , l 3 , . . . , l m }. we follow prior works @xcite @xcite to assign english as the center of multilingual data. given the original input sequence of symbol representation to the encoder @xmath1, x 2 , ..., x i and the output sequence generated by decoder @xmath2, y 2 , ..., y j , we follow the method of @xcite to insert the language token at the beginning of x as translation instruction. therefore, the actual input sequence is x ′ = (l, x), and we model the translation of x ′ to y with transformer. we only build ls layers (lsls) parallel with the feed-forward network (ffn) layers in the decoder of transformer, and keep the self-attention and cross-attention mechanism fixed. the ffn of transformer consists of two fully connected neural networks with a relu activation function in between: where h is the input vector, w indicates parameter matrices for projections, and b indicates bias parameter matrices. lsls are a series of neural networks specified to l ′ . each lsl is similar to ffn in architecture but can be relatively light in inner size: based on the piecewise function eq.( 3 ), the role of central language will be abandoned in non-centered data conditions, namely the case of @xmath3. we illustrate the architecture of cll in figure 1 : the cll is a component, including lightweight lsls and ffn, to replace the original role of ffn in each decoder layer of transformer. compared to the mixture of experts which is the generalization of the gating mechanism @xcite , a deterministic route specific to language replaces the gate in cll. for convenience, we use fcll (full cll) to indicate that the model in which all decoder layers are constructed in the form of our proposed architecture. we introduce a variation named sd that constructs cll in a single decoder layer among all layers of transformer, namely single-disentangled cll. inspired by the work of @xcite , we remove the residual connection of ffn in a middle encoder layer to weaken the linguistic features of encoding. to keep the balance between weakening encoding and improving decoding, we empirically build cll in the middle decoder. specifically, given n encoder and decoder layers of transformer, we remove the residual connection of the ffn in the encoder and replace the ffn with cll in the decoder at n/2 + 1 th layer of both networks. our experiments (section 4.3) empirically show that sd has comparable performance with fcll in small-scale datasets, although sd is more parameter-efficient than fcll ( ## conclusion in this work, we supplement the theory of zeroshot translation with the strict dependence of noncentered languages, and we describe the instability of zero-shot translation. to counteract the influence of the dependence, we proposed a simple yet effective method that employs ls modeling by adapting to non-centered languages. our analysis based on layer attribution demonstrated that ls information is conducive to disentangling the coupled model representation. our experiments on various datasets and different data conditions show that our proposed method outperforms in performance and complexity.