Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation (2021)

Abstract: Non-Autoregressive machine Translation (NAT) models have demonstrated significant inference speedup but suffer from inferior translation accuracy. The common practice to tackle the problem is transferring the Autoregressive machine Translation (AT) knowledge to NAT models, e.g., with knowledge distillation. In this work, we hypothesize and empirically verify that AT and NAT encoders capture different linguistic properties of source sentences. Therefore, we propose to adopt multi-task learning to transfer the AT knowledge to NAT models through encoder sharing. Specifically, we take the AT model as an auxiliary task to enhance NAT model performance. Experimental results on WMT14 En-De and WMT16 En-Ro datasets show that the proposed Multi-Task NAT achieves significant improvements over the baseline NAT models. Furthermore, the performance on large-scale WMT19 and WMT20 En-De datasets confirm the consistency of our proposed method. In addition, experimental results demonstrate that our Multi-Task NAT is complementary to knowledge distillation, the standard knowledge transfer method for NAT.

## introduction neural machine translation (nmt), as the state-ofthe-art machine translation paradigm, has recently been approached with two different sequence decoding strategies. the first type autoregressive translation (at) models generate output tokens one by one following the left to right direction @xcite , but it is often criticized for its slow inference speed @xcite . the second type non-autoregressive translation (nat) models adopt a parallel decoding algorithm to produce output tokens simultaneously @xcite @xcite , but the translation quality of it is often inferior to auto-regressive models @xcite . many researchers have investigated the collaboration between at and nat models. for instance, encoder-nad-ad @xcite leverages nat models to improve the performance of at. specifically, their method inserts a nat decoder between the conventional at encoder and decoder to generate coarse target sequences for the final autoregressive decoding. a line of research @xcite @xcite holds the opinion that the lack of contextual dependency on target sentences potentially leads to the deteriorated performance of nat models. to boost the nat translation performance, many recent works resort to the knowledge transfer from a well-trained at model. typical knowledge transfer methods include sequence-level knowledge distillation with translation outputs generated by strong at models @xcite , word-level knowledge distillation with at decoder representations @xcite , and fine-tuning on at model by curriculum learning @xcite , etc. in this work, we first verify our our hypothesis that at and nat encoders -although they belong to the same sequence-to-sequence learning task -capture different linguistic properties of source sentences. we conduct our verification by evaluating the encoder on a set of probing tasks (conneau @xcite for at and nat models. further, by leveraging the linguistic differences, we then adopt a multi-task learning framework with a shared encoder (i.e., multi-task nat) to transfer the at model knowledge into the nat model. specifically, we employ an additional at task as the auxiliary task of which the encoder parameters are shared with the nat task while parameters of the decoder are exclusive. since many works @xcite suggest that the weights for each task are critical to the multi-task learning, in this work, the multi-task weight assigned to the at task is dynamically annealed from 1 to 0. we name this scheme importance annealing. we empirically show the benefit of importance annealing in both directions of the original wmt14 english⇔german dataset. further with knowledge distillation, our proposed multi-task nat achieves significant improvements on wmt14 english⇔german and wmt16 english⇔romanian datasets. this confirms the effectiveness of our proposed model on machine translation tasks. our contributions are as follows: to verify our hypothesis that at and nat encoders capture different linguistic properties of source sentences and can thereby complement each other, we probe the linguistic knowledge @xcite that embedded in the at and nat encoders on a set of tasks to investigate to what extent an encoder captures the linguistic properties. we present the detail for each probing tasks in appendix b. moreover, in appendix c, we also provide a qualitative investigation to capture the difference between high-dimensional representations of at and nat encoders from another perspective. the at and nat models referred to in the following experiments are transformer and mask-predict. we train the models on the wmt14 english⇒german dataset, and the details of the experiments are introduced in the appendix. probing tasks probing tasks @xcite can quantitatively measure the linguistic knowledge embedded in the model representation. we follow @xcite to set model configurations. the experimental results are depicted in ## approach in this section, we introduce that our shared encoder structure between at and nat models under the multi-task learning framework. multi-task nat given the at and nat models under the standard encoder-decoder structure, we employ the hard parameter sharing method @xcite to share their encoder parameters. therefore, as shown in figure 1 , the proposed model multi-task nat consists of three components: shared encoder, at decoder, and nat decoder. their parameters are jointly optimized towards minimizing the multi-task loss function, as introduced in the next section. multi-task framework the loss function of the proposed multi-task nat l at iteration step t is defined as the weighted sum of at loss and nat loss: ## experiment we conducted experiments on two widely used wmt14 english⇔german and wmt16 english⇔romanian benchmark datasets, which consist of 4.5m and 610k sentence pairs, respectively. we applied bpe @xcite with 32k merge operations for both language pairs. the experimental results are evaluated in case-sensitive bleu score @xcite . we use transformer @xcite ) as our baseline autoregressive translation model and the mask-predict @xcite as our baseline non-autoregressive model. we integrate the transformer decoder into the mask-predict to implement the proposed multi-task nat model. for λ t , we use the annealing scheme described in section 3. since the major nat architecture of our method is exactly the mask-predict model, any established decoding latency results @xcite for mask-predict can also be applied to ours. all of the parameters are randomly initialized for a fair comparison with the mask-predict. more training details are introduced in appendix a. ## conclusion and future work in this paper, we have presented a novel multitask learning approach for nat model with a hard parameter sharing mechanism. experimental results confirm the significant effect of the proposed multi-task nat model, which shows the complementary effects of multi-task learning to the knowledge distillation method. based on our multi-task nat, there are many promising directions for future research. for example, 1) decoder interaction: knowledge distillation in an online fashion between at and nat decoders; 2) share-all framework: shared-encoder and shared-decoder with two decoding strategies, and the model can dynamically choose the optimal decoding strategy during model inference. 3) data manipulation strategies: such as data rejuvenation @xcite , lexical frequency discrepancy @xcite .