latent-GLAT : Glancing at Latent Variables for Parallel Text Generation (2022)

Abstract: Recently, parallel text generation has received widespread attention due to its success in generation efficiency. Although many advanced techniques are proposed to improve its generation quality, they still need the help of an autoregressive model for training to overcome the one-to-many multi-modal phenomenon in the dataset, limiting their applications. In this paper, we propose GLAT, which employs the discrete latent variables to capture word categorical information and invoke an advanced curriculum learning technique, alleviating the multi-modality problem. Experiment results show that our method outperforms strong baselines without the help of an autoregressive model, which further broadens the application scenarios of the parallel decoding paradigm.

## introduction non-autoregressive transformer (nat, @xcite introduce a parallel decoding paradigm with higher decoding efficiency (> 10×) than autoregressive models @xcite @xcite . unlike autoregressive models, nat models impose conditional independence assumptions in words to support parallel decoding of sentences during inference. it attracts many researchers to explore nat in machine translation @xcite @xcite ) and text-to-speech tasks @xcite . amount of researchers devoted themselves to improve the nats' inferior generation quality. such as modeling word inter-dependencies by curriculum learning @xcite or iterative refinements mechanism @xcite , introducing latent variables to decompose target sentences and serve as the springboard for decoding @xcite @xcite , and introduce inductive bias for models' training @xcite . the most successful method is the glancing transformer (glat, @xcite , which trains the nat model by sampling partial target words as inputs to predict the remaining target words, explicitly building dependencies between the observed and unobserved words. @xcite employ glat to achieve impressive results on the translation task of wmt21 foot_1 , even outperforming many strong autoregressive translation systems in bleu score @xcite . although existing nat models achieve competitive results compared to autoregressive models in translation tasks, it is not negligible that they still need the help of an autoregressive transformer (at, @xcite as a teacher for training, i.e., sequence-level knowledge distillation @xcite . a well-recognized explanation is a multi-modality problem @xcite : each input may have multiple valid outputs in datasets, which will prevent nat models from learning to organize consistent outputs. training with the outputs of an at can directly bypass the multi-modal phenomenon in the dataset, effectively improving the models' performances. however, training nat models by knowledge distillation are limited. first, it needs to train an extra at model, which inevitably enlarges the training cost. second, it is hard to promise that the teacher (or at) model can be accurate enough in all text generation settings, which will become the bottleneck for its student nat model. therefore, training a model from scratch without the help of an at model is still an open and interesting problem. in this paper, we propose latent-glat, which can directly learn from the raw dataset. it alleviates the multi-modality problem following a divide-andconquer spirit, introducing a small set of discrete latent variables to capture the target word categorical information and divide the origin goal into latent variables modeling and sentence reconstruction. first, the categorical information may have fewer multi-modality phenomena than the original words, thus can be learned directly without the help of knowledge distillation. second, the word categorical information is informativeness to the sentence reconstruction. we can extend glancing training with these discrete latent variables for modeling the sentence, encouraging the model to build dependencies on word categorical information rather than words, which works more robustly. experiment results on wmt14, quora, and dai-lydialog datasets show that latent-glat achieves remarkable improvements over several strong baselines, verifying the effectiveness of latent-glat. more impressively, latent-glat even outperforms autoregressive models in quora and dailydialog datasets, further validating our motivation for removing knowledge distillation. in-depth analyses indicate that the introduced discrete latent variables are helpful to alleviate the multi-modality problem and are necessary for performance improvement. ## background for a sequence-to-sequence task of predicting sequence @xmath0, y 2 , • • • , y m ) given its input sequence @xmath1, x 2 , • • • , x n ), the classical autoregressively factorization decomposes the p(y |x) with a series of conditional probability: although such factorization achieved great success in previous studies @xcite @xcite , they predict each word foot_2 based on the prefix words, which may suffer from the issues of error accumulation and slow decoding during inference. non-autoregressive transformer. to tackle the above problems, @xcite firstly propose non-autoregressive transformer (nat), introduc-ing a non-autoregressive factorization as: ## proposed method: latent-glat in this section, we present latent-glat. latent-glat follows latent transformer models @xcite but introduces glancing training @xcite with the discrete latent variables. our intuitions are as follows: first, compared to the words, the introduced discrete latent variables may have fewer modes than words and be informative to determine the modes of the sentences. in such a case, we can directly learn the discrete latent variables by the glancing transformer @xcite , keeping competitive inference efficiency. more importantly, we can employ the latent variables to invoke glancing training for modeling the target sentences, which is informative enough to reduce the multi-modality problem of original sentences. besides, glancing at latent variables also works robustly due we can obtain the latent variables during inference. ## experiments we conduct experiments on several generation tasks, including machine translation, paraphrase generation, and dialog generation. ## conclusion we propose latent-glat, which can be directly trained without the help of knowledge distillation. specifically, we employ discrete latent variables to capture the word categorical information and divide the original goal into the latent variables modeling and word prediction tasks. then, we learn each task with the glancing training and encourage the model to build dependencies on the latent variables, which have fewer modes than the words and are also informative for modeling the target sentences. experiments results on machine translation, paraphrase generation, and dialogue generation tasks validate the effectiveness of our latent-glat.