Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation (2023)

Abstract: This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST). Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default. However, Xu et al. (2019) has revealed that PreNorm carries the risk of overfitting the training data. Based on this, we hypothesize that PreNorm may overfit supervised directions and thus have low generalizability for ZST. Through experiments on OPUS, IWSLT, and Europarl datasets for 54 ZST directions, we demonstrate that the original Transformer setting of LayerNorm after residual connections (PostNorm) consistently outperforms PreNorm by up to 12.3 BLEU points. We then study the performance disparities by analyzing the differences in off-target rates and structural variations between PreNorm and PostNorm. This study highlights the need for careful consideration of the LayerNorm setting for ZST.

## introduction multilingual neural machine translation (mnmt) enables translation between unseen language pairs, i.e., zero-shot translation (zst) @xcite . prior studies have explored techniques such as language tags @xcite , residual connections @xcite , and novel training objectives @xcite @xcite @xcite @xcite for improving zst. they primarily used the transformer architecture @xcite , which has two variations depending on the position of layer normalization (layernorm) @xcite , namely, prenorm (applied at the input of layers) @xcite and postnorm (applied after residual connections), as shown in fig. 1 . as previous studies showed that prenorm can result in more stable training and faster convergence compared to postnorm for mnmt @xcite , most zst works @xcite @xcite use prenorm as the default setting following those mnmt studies. however, @xcite revealed that prenorm carries the risk of overfitting the training data. we thus hypothesize that in a multilingual scenario, prenorm may overfit supervised directions and have poor zst generalizability. we systematically explore prenorm and postnorm's effect on zst to verify this. using the opus, iwslt, and europarl datasets and a total of 54 zst directions, we show that postnorm consistently outperforms prenorm by up to 12.3 bleu points. following previous work, we also evaluate different language tag @xcite and residual connection @xcite settings, as they have been shown to impact zst but we observe that postnorm continues to be superior thereby lending credibility to our hypothesis. to better understand the performance differences, we introduce a novel analysis approach called layer-wise language recognition (llr), which tracks the off-target rates for each encoder and decoder layer by training token-level classifiers to recognize the source or target language. this analysis shows that prenorm is more sensitive to language tag settings than postnorm, negatively impacting zst performance. additionally, by examining the unraveled view of prenorm (fig. 1 ) inspired by @xcite , we reveal structural flaws in prenorm for zst. our analysis demonstrates that the order of layernorm and selfattention/feed-forward network in prenorm is the main factor affecting its zst performance. given the prevalent use of prenorm as the default setting in zst baselines and frameworks such as fairseq @xcite foot_0 and ten-sor2tensor @xcite , our study emphasizes the importance of careful consideration in the layernorm setting for zst. nguyen and salazar (2019) have explored the impacts of normalization and initialization choices on supervised low-resource nmt settings, however, we delve deeper and focus on the significance of the positioning of layernorm for zero-shot nmt. we expect this to complete the understanding of layernorm's role in multilingualism, particularly in the context of zero-shot translation. ## experiments and results we evaluate the performance of prenorm and post-norm for zst on various datasets and language pairs. we then analyze the off-target rates and structural discrepancies between prenorm and post-norm to understand performance differences. ## conclusion in this paper, we comprehensively explored the effects of layernorm on zst performance. our results demonstrate that postnorm consistently outperforms prenorm for zst, regardless of the language tag and residual connection settings used. through in-depth analysis of off-target rates and structural flaws in the prenorm model, we were able to identify the underlying factors that contribute to the performance discrepancy. our study suggests that care should be taken when selecting the layernorm setting for zst in future research.