Layer-Wise Multi-View Learning for Neural Machine Translation (2020)

Abstract: Traditional neural machine translation is limited to the topmost encoder layer’s context representation and cannot directly perceive the lower encoder layers. Existing solutions usually rely on the adjustment of network architecture, making the calculation more complicated or introducing additional structural restrictions. In this work, we propose layer-wise multi-view learning to solve this problem, circumventing the necessity to change the model structure. We regard each encoder layer’s off-the-shelf output, a by-product in layer-by-layer encoding, as the redundant view for the input sentence. In this way, in addition to the topmost encoder layer (referred to as the primary view), we also incorporate an intermediate encoder layer as the auxiliary view. We feed the two views to a partially shared decoder to maintain independent prediction. Consistency regularization based on KL divergence is used to encourage the two views to learn from each other. Extensive experimental results on five translation tasks show that our approach yields stable improvements over multiple strong baselines. As another bonus, our method is agnostic to network architectures and can maintain the same inference speed as the original model.

## introduction neural machine translation (nmt) adopts the encoder-decoder paradigm to model the entire translation process @xcite . specifically, the encoder finds a multi-layer representation of the source sentence, and the decoder queries the topmost encoding representation to produce the target sentence through a cross-attention mechanism @xcite . however, such overreliance on the topmost encoding layer is problematic in two aspects: (1) prone to over-fitting, especially when the encoder is under-trained, such as in low-resource tasks @xcite ; (2) it cannot make full use of representations extracted from lower encoder layers, which are syntactically and semantically complementary to higher layers @xcite . researchers have proposed many methods to make the model aware of various encoder layers besides the topmost to mitigate this issue. almost all of them resort to the adjustment of network structure, which can be further divided into two categories. the first is to merge the feature representations extracted by distinct encoder layers before being fed to the decoder @xcite @xcite . the differences between them lie in the design of the merge function: through self-attention @xcite , recurrent neural network @xcite , or tree-like hierarchical merge @xcite . moreover, the second makes each decoder layer explicitly align to a parallel encoder layer @xcite or all encoder layers @xcite . however, the above methods either complicate the original model @xcite @xcite or limit the model's flexibility, such as requiring the number of the encoder layers to be the same as the decoder layers @xcite . instead, in this work, we propose layer-wise multi-view learning to address this problem from the perspective of model training, without changing the model structure. our method's highlight is that only the training process is concerned, while the inference speed is guaranteed to be the same as that of the standard model. the core idea is that we regard the off-the-shelf output of each encoding layer as a view for the input sentence. therefore, it is straightforward and cheap to construct multiple views during a standard layer-by-layer encoding process. further, in addition to the output of the topmost encoder layer used in standard models (refer to the primary view), we also incorporate an intermediate encoder layer as the auxiliary view. we feed the two views to a partially shared decoder for independent predictions. an additional regularization loss based on prediction consistency between views is used to encourage the auxiliary view to mimic the primary view. thanks to the co-training on the two views, the gradients during back-propagation can simultaneously flow into the two views, which implicitly realizes the knowledge transfer. extensive experimental results on five translation tasks (@xmath0, iwslt'14 @xmath1, wmt'17 @xmath2, wmt'16 @xmath3, and wmt'16 @xmath4. in particular, we have achieved new state-of-the-art results of 10.8 bleu on @xmath5.23 bleu on iwslt'14 @xmath6. further analysis shows that our method's success lies in the robustness to encoding representations and dark knowledge @xcite provided by consistency regularization. ## approach in this section, we will take the transformer model @xcite as an example to show how to train a model by our multi-view learning. we first briefly introduce transformer in § 2.1, then describe the proposed multi-view transformer model (called mv-transformer) and its training and inference in detail in § 2.2. finally, we discuss why our method works in § 2.3. see figure 1 for an overview of the proposed approach. ## conclusion we studied to incorporate different encoder layers through multi-view learning in neural machine translation. in addition to the primary view from the topmost layer, the proposed model introduces an auxiliary view from an intermediate encoder layer and encourages the transfer of knowledge between the two views. our method is agnostic to network architecture and can maintain the same inference speed as the original model. we tested our method on five translation tasks with multiple strong baselines: transformer, deep transformer, and dynamicconv. experimental results show that our multi-view learning method can stably outperform the baseline models. our models have achieved new state-of-the-art results in @xmath7. biao zhang, ivan titov, and rico sennrich. 2019. improving deep transformer with depth-scaled initialization and merged attention. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 898-909, hong kong, china, november. association for computational linguistics.