Self-Improvement of Non-autoregressive Model via Sequence-Level Distillation (2023)

Abstract: Although Non-autoregressive Transformer (NAT) models have achieved great success in terms of fast inference speed, this speedup comes with a performance drop due to the inherent multi-modality problem of the NAT model. Previous works commonly alleviate this problem by replacing the target side of the raw data with distilled data generated by Autoregressive Transformer (AT) models. However, the multi-modality problem in the distilled data is still significant and thus limits further improvement of the NAT models. In this paper, we propose a method called Sequence-Level Self-Distillation (SLSD), which aims to generate distilled data by the NAT model itself, eliminating the need for additional teacher networks. Furthermore, SLSD can adapt to different NAT models without precise adjustments since the self-distilled data is generated from the same types of NAT models. We conduct extensive experiments on WMT14 EN \leftrightarrow DE and WMT16 EN \leftrightarrow RO and choose four classic NAT models as the backbones to validate the generality and effectiveness of SLSD. The results show that our approach can consistently improve all models on both raw data and distilled data without sacrificing the inference speed.

## introduction non-autoregressive transformer (nat) models @xcite have significantly improved inference speed compared to autoregressive transformer (at) models @xcite . nat models make the conditional independence assumption in the output space and generate the entire sequence in parallel. however, this speedup comes at the cost of translation quality due to the multimodality problem @xcite , where the data distribution typically includes numerous potential translations of the same sequence. unlike at models, which generate subsequent tokens conditioned * corresponding author on the previous ones, nat models lack contextual dependency and are therefore more likely to generate sequences with mixed modalities, leading to a decline in performance. one way to improve the ability of nat models to handle complex data is by enhancing their capacity @xcite @xcite . latent alignment models @xcite relax the alignment restriction by marginalizing out all monotonic latent alignments using the connectionist temporal classification (ctc) loss @xcite . recently, @xcite proposed the directed acyclic transformer (dat) that can capture multiple translation modalities simultaneously by modeling several decoding paths. despite the success of these two models, they struggle to handle non-monotonic alignments in machine translation @xcite , which are also common in multi-modality problems. another common approach to alleviate the multi-modality problem is modifying the target sequence @xcite . one standard method is to simplify raw data with sequence-level knowledge distillation @xcite , which replaces the target side of training data with the output of at models. however, at distilled data still has two issues. first, the multi-modality issue in distilled data still remains @xcite . second, @xcite find that higher-quality distilled data does not necessarily improve the performance of nat models generally. weaker nat models achieve the best performance on distilled data generated by smaller at, while stronger nat models prefer the data generated by stronger at. this phenomenon suggests that, when simplifying data for training a specific nat model, high quality may not be the first consideration, but rather making the data more suitable for nat to learn. considering the aforementioned problems, some recent works focus on modifying the distillation strategy to generate more adaptive distilled data for nat models. for instance, @xcite construct the distilled data in five steps via re-ranking and filtering the data using an at model, and @xcite generate multi-reference distilled data with multiple at models. however, these works introduce a redundant pipeline and only achieve limited performance improvements. therefore, most existing works only consider knowledge distillation as a necessary data processing technique, rather than producing adaptive data for nat models to learn better. in this paper, we propose a simple yet effective approach to generate distilled data that is more adaptive for nat models to learn, named sequence-level self-distillation (slsd). inspired by @xcite and @xcite , which suggest that targets with a higher likelihood of the nat models have fewer problems with multi-modality, we search for selfdistilled targets in the space of the output distributions produced by the nat models. we define a score function to select the self-distilled targets to ensure the quality of the self-distilled data. in the slsd framework, the nat model itself produces the distilled data, eliminating the need for additional teacher networks. furthermore, it can adapt to the training processes of different nat models without requiring precise adjustments. to validate the effectiveness and generality of the proposed slsd approach, we conduct extensive experiments on four machine translation benchmarks: wmt14 en↔de and wmt16 en↔ro. we chose five classic nat methods as the baseline methods, including vnat @xcite , cmlm @xcite , glat @xcite , ctc, and dat. the results show that the slsd approach can consistently improve all models for different translation directions on both raw and distilled data. additionally, under the same training strategy, the model fine-tuned on self-distilled data can achieve better performance than the raw data and distilled data generated by at. the experiments indicate that self-distilled data can mitigate the multi-modality problem in raw and distilled data, thereby improving the performance of the nat models. our code is released at https://github.com/bluezeros/slsd_nat . the major contributions of our paper are summarized as follows: ## preliminary in this section, we first briefly describe the task formulation and then introduce three types of nat models. the machine translation task can be formally defined as a sequence-to-sequence generation problem. given the target language sequence @xmath0, y 2 , ..., y t } and source language sequence @xmath1, x 2 , ..., x s }, the non-autoregressive models assume conditional independence between the output tokens and factorize the output probabilities as p θ (y|x)= t @xmath2, where θ represents the parameters of the nat models. ## method in this section, we first explain the motivation of the proposed slsd framework. then we describe the process of generating self-distilled targets in two steps: the sampling of the self-distilled targets and the selection of the self-distilled targets. finally, we will discuss the training details of the slsd framework. ## conclusion in this paper, we introduced a simple yet effective method, slsd, to generate distilled data using nat models themselves. this approach can greatly reduce the multi-modality problem in the data, and consistently improves the performance of four types of nat models across all datasets. ablation experiments verify that self-distilled data is better suited for nat models to learn from, compared to the distilled data generated by at models.