Non-Autoregressive Sequence Generation (2022)

Abstract: Non-autoregressive sequence generation (NAR) attempts to generate the entire or partial output sequences in parallel to speed up the generation process and avoid potential issues (e.g., label bias, exposure bias) in autoregressive generation. While it has received much research attention and has been applied in many sequence generation tasks in natural language and speech, naive NAR models still face many challenges to close the performance gap between state-of-the-art autoregressive models because of a lack of modeling power. In this tutorial, we will provide a thorough introduction and review of non-autoregressive sequence generation, in four sections: 1) Background, which covers the motivation of NAR generation, the problem definition, the evaluation protocol, and the comparison with standard autoregressive generation approaches. 2) Method, which includes different aspects: model architecture, objective function, training data, learning paradigm, and additional inference tricks. 3) Application, which covers different tasks in text and speech generation, and some advanced topics in applications. 4) Conclusion, in which we describe several research challenges and discuss the potential future research directions. We hope this tutorial can serve both academic researchers and industry practitioners working on non-autoregressive sequence generation.

## target audience this tutorial targets those audiences who work on 1) neural sequence generation (e.g., neural machine translation, etc.); 2) natural language and speech processing; 3) deep learning and artificial intelligence in general. some prerequisites for the attendees are: part i introduction (∼ 20 minutes) 1.1 problem definition 1.2 evaluation protocol 1.3 multi-modality problem part ii methods (∼ 90 minutes) 2.1 model architectures 2.1.1 fully nar models 2.1.2 iteration-based nar models 2.1.3 partially nar models 2.1.4 locally ar models 2.1.5 nar models with latent variables 2.2 objective functions 2.2.1 loss with latent variables 2.2.2 loss beyond token-level 2.3 training data 2.4 learning paradigms 2.4.1 curriculum learning 2.4.2 adversarial training 2.4.3 self-supervised pre-training 2.5 inference methods and tricks part iii applications (∼ 50 minutes) 3.1 text generation 3.1.1 neural machine translation 3.1.2 text summarization 3.1.3 text error correction 3.1.4 automatic speech recognition 3.2 speech generation 3.2.1 text to speech 3.2.2 voice conversion 3.3 advanced topics in applications 3.3.1 advanced length prediction 3.3.2 alignment (duration vs attention) 3.3.3 target token dependency 3.3.4 relationship with streaming part iv open problems, future directions, q&a (∼20 minutes) 5 how the tutorial includes other people's work we organize our tutorial content from a broad view of non-autoregressive sequence generation, spanning from basic methods to applications, which cover diverse work in this area, most of which are other people's work. ## diversity considerations methods we introduce the methods of nonautoregressive sequence generation in a comprehensive and diverse view, covering model architectures, objective functions, training data, learning paradigms, and additional tricks. these methods are general and not limited to specific languages or domains. applications we introduce a variety of nonautoregressive sequence generation tasks, spanning from the text (e.g., neural machine translation, text error correction) to speech (e.g., text to speech, voice conversion). instructors we are from different institutions (facebook and microsoft) and work on diverse topics in machine learning, nlp, and nonautoregressive sequence generation. audiences due to the diversity in the methods and applications of our tutorial and the tutorial instructors, we can attract audiences interested in diverse sequence generation tasks and modalities (text and speech) and from both academia and industry. ## reading list please see the citations in section 1. for participants interested in reading important studies before this tutorial, we recommend the following basic papers: (1) the typical ar model (transformer) @xcite ; (2) the vanilla nar model @xcite ; (3) the typical iterationbased nar model @xcite ; (4) a study on nar models for both text and speech tasks @xcite . ## ethics statement non-autoregressive sequence generation can improve the inference speed of various sequence generation tasks in text and speech. unfortunately, this technology may be misused to generate deepfake content @xcite such as mimicking one's writing style or speaking style. however, great attempts have been made to detect the deepfake content @xcite , which can minimize or avoid its potential negative impact.