POS - C onstrained P arallel D ecoding for N on-autoregressive G eneration (2021)

Abstract: The multimodality problem has become a major challenge of existing non-autoregressive generation (NAG) systems. A common solution often resorts to sequence-level knowledge distillation by rebuilding the training dataset through autoregressive generation (hereinafter known as “teacher AG”). The success of such methods may largely depend on a latent assumption, i.e., the teacher AG is superior to the NAG model. However, in this work, we experimentally reveal that this assumption does not always hold for the text generation tasks like text summarization and story ending generation. To provide a feasible solution to the multimodality problem of NAG, we propose incorporating linguistic structure (Part-of-Speech sequence in particular) into NAG inference instead of relying on teacher AG. More specifically, the proposed POS-constrained Parallel Decoding (POSPD) method aims at providing a specific POS sequence to constrain the NAG model during decoding. Our experiments demonstrate that POSPD consistently improves NAG models on four text generation tasks to a greater extent compared to knowledge distillation. This observation validates the necessity of exploring the alternatives for sequence-level knowledge distillation.

## introduction unlike autoregressive generation (ag) that generates tokens step-by-step, non-autoregressive generation (nag) parallelly generates all tokens in one time step and thus the inference could be significantly speeded up @xcite @xcite . despite the computational advantage of nag, it has faced the multimodality problem @xcite caused by the conditionally independent decoding. a typical example of the problem is illustrated in figure 1 , where either of "thank you." and "many thanks." is the correct translation (i.e., generation modes). in this example, a mixed mode "many you." / "thank thanks." will be generated by nag. it is because the conditional dependence among target words will be broken in parallel decoding. a typical manifestation is that words are usually missing (e.g., "many you.") and repeating (e.g., "thank thanks.") in nag's sentences. to solve this problem, the key is helping nag models to deal with various generation modes. to date, one of the most widely used solutions is sequence-level knowledge distillation @xcite which aims to reduce the generation modes of the raw data @xcite . taking machine translation as an example, the knowledge distillation based methods rebuild the target sequence in the training set by employing an ag model to translate the training samples. the assumption is that the target sentences generated by one ag model tend to have less modality. despite the success of the above studies, there are still two major limitations: (1) most existing works mainly focus on machine translation where the performance of ag is generally assumed to be better than nag. clearly, such a solution will degrade the performance of nag on the task where the ag model cannot obtain a better result. as demonstrated in our experiments (see § 4.5), there are a number of such tasks beyond the assumption like text summarization and story ending generation. to overcome the aforementioned limitations, we explore to alleviate the multimodality problem in a different manner. in short, we aim to constrain nag generation modes in the inference stage, rather than directly reducing generation modes in the training stage. more specifically, our basic idea is that the linguistic structure of the target sentence could be helpful to alleviate the multimodality problem. in this paper, we show that the part-of-speech (pos) sequence, one of most simple solutions in modeling the linguistic structure @xcite , could effectively verify our idea and show promising performance in four different tasks. in more details, the proposed pos-constrained parallel decoding (pospd) trains a pos predictor to obtain pos tags of target sequences. in the inference stage, pospd constrains nag models to choose the final outputs that satisfy the pre-specified pos sequence. as the pos predictor with a shallow decoder is separately trained, our pospd could act as a plug-and-play method to assistant nag models with negligible extra time. meanwhile, it also shows the speed advantage of our method even considering the time cost in building the pos dataset, since pos tagging is much faster than sentence generating due to the small pos dictionary. to conduct a comprehensive empirical evaluation, we examine the generalizability of pospd by applying it to two widely-used nag models (i.e., cmlm and disco) over four text generation tasks, including text summarization, story ending generation, question generation, and machine translation. experiments demonstrate that pospd significantly and consistently improves the two nag models and beats the sequence-level knowledge distillation with a considerable performance gap. the main contributions of this work could be summarized as follows: ## related works in this section, we first analyze related works on alleviating the multimodality problem. then, we review some representative works which introduce the linguistic structure into some text generation scenarios. ## methodology in this section, we elaborate our pospd for the nag model. to ease of presentation, we start from a toy example to illustrate the overview of pospd in § 3.1, and then give a detailed explanation of the implementation in § 3.2. after that, we present the training details of pospd in § 3.3. ## experiments in this section, we use multiple text generation datasets to comprehensively evaluate the effectiveness and efficiency of the proposed pospd. for an extensive comparison, we compare our pospd with the sequence-level knowledge distillation, and provide detailed analyzes in alleviating the multimodality problem and the time cost in dataset building. ## conclusion in this paper, we revisit the role of the knowledge distillation in alleviating the multimodality problem of nag. in brief, we experimentally reflect that the basic assumption of these knowledge distillation methods, the ag model is superior to nag model, does not always hold for all text generation tasks. to alleviate the multimodality problem, we show a different solution by incorporating linguistic structure into nag. extensive experiments demonstrate that our pospd significantly and consistently improves the nag models in effectiveness and computational efficacy. as we tentatively give a successful implementation of leveraging one of the simplest linguistic structures to benefit the nag models in inference, such paradigm deserves a closer and more detailed exploration. thus in the future, we will investigate to make the nag models enjoy the benefits of incorporating diverse and abundant linguistic structures in a more superior way. in addition, our experimental results suggest that future work might need to consider wider ranges of generation tasks instead of only machine translation when assessing the performance of nag models.