Sparse and Decorrelated Representations for Stable Zero-shot NMT (2020)

Abstract: Using a single encoder and decoder for all directions and training with English-centric data is a popular scheme for multilingual NMT. However, zero-shot translation under this scheme is vulnerable to changes in training conditions, as the model degenerates by decoding non-English texts into English regardless of the target specifier token. We present that enforcing both sparsity and decorrelation on encoder intermediate representations with the SLNI regularizer (Aljundi et al., 2019) efficiently mitigates this problem, without performance loss in supervised directions. Notably, effects of SLNI turns out to be irrelevant to promoting language-invariance in encoder representations.

## introduction as massive language pairs are supported in recent works in neural machine translation (nmt) @xcite , obtaining training data becomes more of an issue. due to limited availability of parallel corpora, datasets for multilingual nmt are in many cases englishcentric-english is either on the source side or the target side-, or at least missing several pairs among the supported language set. this leads to a conspicuous need for a model to support zero-shot translation, which is to translate between language pairs for which no parallel training data exists. a popular scheme for multilingual nmt is to have one encoder and one decoder shared across all trained directions, and prepend a reserved token to the source text to indicate the target language. this model is capable of zero-shot translation; setting the target token which was unpaired with the source at training time still works @xcite @xcite . however, while being parameter-efficient, an exposure bias arises when trained with english-centric data; as non-english languages are always trained to be translated into english, they are wrongly decoded into english for zero-shot directions @xcite . in fact, zero-shot nmt under this scheme is extremely sensitive to hyperparmeters including batch size, dropout, and weight initialization @xcite . fixing the hyperparameters favorable to zero-shot directions would not be desirable, however, if such conditions hurt performance on supervised directions. we utilize the sparse coding through local neural inhibition (slni) @xcite regularizer to make the representations more robust to hyperparameters. slni was originally suggested as a continual learning technique by enforcing representation sparsity and decorrelation. here, we deviate from its previous use and focus on its single-stage effects during joint multitask training of multiple language pairs. we present that enforcing representation sparsity and decorrelation together stabilizes zero-shot performance across various training conditions, without hurting performance on supervised directions. meanwhile, @xcite noted that regularizing the model to be language-invariant empirically alleviates degeneration. they aligned non-english latent representations to english by minimizing the cosine distance between parallel instances. @xcite built a universal encoder on both source and pivot languages, so that the encoder can deal with zero-shot directions in the way it handles pivot-target data. @xcite were on the simliar track by learning language-invariant features, though via regularizing the decoder. we also utilize a regularizer, but its effects turn out to be irrelevant to making language-invariant representations (see 5.2 for details). with a batch of n inputs and 1 ≤ i, @xmath0, the layer representation adapting slni to transformers. slni was originally applied to toy datasets in the vision domain and rather simple models. here, we adopt it to the real-world language domain and to transformers @xcite . 1 we apply slni on the encoder-side. 2 outputs of every layer normalization (after both self-attention and position-wise feed-forward sublayers) are subject to regularization. 3 unlike images, inputs for nmt have time dimension. we flatten the batch and time dimensions into n , so that the representations are regularized at the token level. ## conclusion without a specifically adjusted training condition, a single encoder-decoder model trained with englishcentric data suffers from exposure bias in such target language specifier tokens are ignored. we resolve this problem with the slni regularizer which enforces sparse and decorrelated representations. we show its effects as a silver bullet technique to preserve performance over all language pairs, both zero-shot and supervised. the ground for this success seems to be orthogonal to previous studies, proposing a new context to be incorporated for a more complete picture of robust zero-shot nmt.