A Study of Non-autoregressive Model for Sequence Generation (2020)

Abstract: Non-autoregressive (NAR) models generate all the tokens of a sequence in parallel, resulting in faster generation speed compared to their autoregressive (AR) counterparts but at the cost of lower accuracy. Different techniques including knowledge distillation and source-target alignment have been proposed to bridge the gap between AR and NAR models in various tasks such as neural machine translation (NMT), automatic speech recognition (ASR), and text to speech (TTS). With the help of those techniques, NAR models can catch up with the accuracy of AR models in some tasks but not in some others. In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all? (2) Why techniques like knowledge distillation and source-target alignment can help NAR models. Since the main difference between AR and NAR models is that NAR models do not use dependency among target tokens while AR models do, intuitively the difficulty of NAR sequence generation heavily depends on the strongness of dependency among target tokens. To quantify such dependency, we propose an analysis model called CoMMA to characterize the difficulty of different NAR sequence generation tasks. We have several interesting findings: 1) Among the NMT, ASR and TTS tasks, ASR has the most target-token dependency while TTS has the least. 2) Knowledge distillation reduces the target-token dependency in target sequence and thus improves the accuracy of NAR models. 3) Source-target alignment constraint encourages dependency of a target token on source tokens and thus eases the training of NAR models.

## introduction non-autoregressive (nar) models @xcite @xcite , which generate all the tokens in a target sequence in parallel and can speed up inference, are widely explored in natural language and speech processing tasks such as neural machine translation (nmt) @xcite @xcite @xcite , automatic speech recognition (asr) @xcite ) and text to speech (tts) synthesis @xcite . however, nar models usually lead to lower accuracy than their autoregressive (ar) counterparts since the inner dependencies among the target tokens are explicitly removed. several techniques have been proposed to alleviate the accuracy degradation, including 1) knowledge distillation @xcite @xcite , 2) imposing source-target alignment constraint with fertility @xcite , word mapping @xcite , attention distillation @xcite and duration prediction @xcite . with the help of those techniques, it is observed that nar models can match the accuracy of ar models for some tasks @xcite , but the gap still exists for some other tasks @xcite . therefore, several questions come out naturally: (1) why the gap still exists for some tasks? are some tasks more difficult for nar generation than others? (2) why the techniques like knowledge distillation and source-target alignment can help nar generation? the main difference between ar and nar models is that nar models do not consider the dependency among target tokens, which is also the root cause of accuracy drop of nar models. thus, to better understand nar sequence generation and answer the above questions, we need to characterize and quantify the target-token dependency, which turns out to be non-trivial since the sequences could be of different modalities (i.e., speech or text). for this purpose, we design a novel model called conditional masked prediction model with mix-attention (comma), inspired by the mix-attention in @xcite and the masked language modeling in @xcite : in comma, (1) the prediction of one target token can attend to all the source and target tokens with mix-attention, and 2) target tokens are randomly masked with varying probabilities. comma can help us to measure target-token dependency using the ratio of the attention weights on target context over that on full (both source and target) context when predicting a target token: bigger ratio, larger dependency among target tokens. we conduct a comprehensive study in this work and obtain several interesting discoveries that can answer previous questions. first, we find that the rank of the target-token dependency among the three tasks is @xmath0: asr has the largest dependency while tts has the smallest. this finding is consistent with the accuracy gap between ar and nar models and demonstrates the difficulty of nar generation across tasks. second, we replace the target sequence of original training data with the sequence generated by an ar model (i.e., through knowledge distillation) and use the new data to train comma; we find that the targettoken dependency is reduced. smaller target-token dependency makes nar training easier and thus improves the accuracy. third, source-target alignment constraint such as explicit duration prediction @xcite or implicit attention distillation @xcite also reduces the target-token dependency, thus helping the training of nar models. the main contributions of this work are as follows: ## comma in this section, we analyze the token dependency in the target sequence with a novel conditional masked prediction model with mix-attention (comma). we first introduce the design and structure of comma, and then describe how to measure the target token dependency based on comma. ## study on the difficulty of nar generation in this section, we aim to find out why the gap still exists for asr and nmt tasks, while in tts, nar can catch up with the accuracy of ar model. we also analyze the causes of different difficulties for different tasks. we start from evaluating the accuracy gap between ar and nar models for nmt, asr and tts, and then measure the token dependency based on our proposed comma. ## study on knowledge distillation in the current and next sections, we investigate why some techniques can help nar generation from the aspect of target token dependency. we only analyze knowledge distillation and attention alignment techniques which are widely used in nar, but we believe our analysis method can be applied to other nar techniques, such as iterative refinement @xcite , fine-tuning from an ar model @xcite ) and so on. most existing nar models @xcite @xcite @xcite rely on the technique of knowledge distillation, which generates the new target sequence given original source sequence from a pre-trained ar model and trains the nar model for better accuracy. in this section, we first conduct experiments to verify the accuracy improvements of knowledge distillation. next, based on our proposed comma, we analyze why knowledge distillation could help nar models. ## study on alignment constraint without the help of target context, nar models usually suffer from ambiguous attention to the source context, which affects the accuracy. recently, many works have proposed a variety of approaches to help with the source-target alignment of nar models, which can improve the estimation of the soft alignment in attention mechanism model. for example, @xcite constrain the kl-divergence of the encoder-to-decoder attention distributions between the ar and nar models. @xcite predict the fertility of the source tokens to approximate the alignments between target sequence and source sequence. @xcite convert the source token to target token with phrase table or embedding mapping for alignments. @xcite predict the duration (the number of mel-spectrograms) of each phoneme. in this section, we first study the effectiveness of alignment constraint for nar models, and then analyze why alignment constraint can help the nar models by observing the changes of token dependency based on our proposed comma. ## related works several works try to analyze and understand nar models on different tasks. we discuss these analyses from the two aspects: knowledge distillation and source-target alignment constraint. knowledge distillation knowledge distillation has long been used to compress the model size @xcite @xcite @xcite or transfer the knowledge of teacher model to student model @xcite , and soon been applied to nar models @xcite @xcite @xcite @xcite to boost the accuracy. some works focus on studying why knowledge distillation works: @xcite provide some insights into the mechanisms of knowledge distillation by studying the special case of linear and deep linear classifiers and find that data geometry, optimization bias and strong monotonicity determine the success of distillation; @xcite argue that the success of kd is also due to the regularization of soft targets, which might be as important as the similarity information between categories. however, few works have studied the cause of why knowledge distillation benefits nar training. recently, @xcite investigate why knowledge distillation is important for the training of nar model in nmt task and find that knowledge distillation can reduce the complexity of data sets and help nar model to learn the variations in the output data. @xcite explore the causes of the poor performance of the nar model by observing the attention distributions and hidden states of nar model. @xcite presents some experiments and analysis to prove the necessity for multiple iterations generation for nat. they also investigate the effectiveness of knowledge distillation in different task and make the assumption that teacher model can essentially clean the training data so that the distilled nar model substantially outperforms nar model trained with raw data. attention alignment constraint previous work pointed out that adding additional alignment knowledge can improve the estimation of the soft alignment in attention mechanism model. for example, @xcite uses the viterbi alignments of the ibm model 4 as an additional knowledge during nmt training by calculating the divergence between the attention weights and the statistical alignment information. compared with ar model, the attention distributions of nar model are more ambiguous, which leads to the poor performance of the nar model. recent works employ attention alignment constraint between the well-trained ar and nar model to train a better nar model. @xcite leverages intermediate hidden information from a well-trained ar-nmt teacher model to improve the nar-nmt model by minimizing kldivergence between the per-head encoder-decoder attention of the teacher and the student. @xcite choose the encoder-decoder attention head from the ar-tts teacher as the attention alignments to improve the performance of the nar model in tts. ## conclusion in this paper, we conducted a comprehensive study on nar models in nmt, asr and tts tasks to analyze several research questions, including the difficulty of nar generation and why knowledge distillation and alignment constraint can help nar models. we design a novel comma and a metric called attention density ratio to measure the dependency on target context when predicting a target token, which can analyze these questions in a unified method. through a series of empirical studies, we demonstrate that the difficulty of nar generation correlates on the target token dependency, and knowledge distillation as well as alignment constraint reduces the dependency of target tokens and encourages the model to rely more on source context for target token prediction, which improves the accuracy of nar models. we believe our analyses can shed light on the understandings and further improvements on nar models.