Continuous Language Generative Flow (2021)

Abstract: Recent years have witnessed various types of generative models for natural language generation (NLG), especially RNNs or transformer based sequence-to-sequence models, as well as variational autoencoder (VAE) and generative adversarial network (GAN) based models. However, flow-based generative models, which achieve strong performance in image generation due to their invertibility and exact density estimation properties, have been less explored for NLG. In this paper, we propose a flow-based language generation model by adapting previous flow generative models to language generation via continuous input embeddings, adapted affine coupling structures, and a novel architecture for autoregressive text generation. We also apply our framework to Sequence-to-Sequence generation, including text- and video-based Question Generation (QG) and Neural Machine Translation (NMT), and data augmentation for Question Answering (QA). We use our language flow model to provide extra input features for QG and NMT, which achieves improvements over the strong QG baselines on SQuAD and TVQA and NMT baseline on WMT16. We also augment QA data with new context by injecting noise to the latent features of the language flow and show this augmentation leads to a large performance improvement from strong baselines on SQuAD and TVQA.

## introduction several generative models have been proposed for language generation, including sequence-tosequence models based on rnns @xcite and transformers @xcite , as well as variational autoencoders (vaes) to generate diverse texts @xcite , plus generative adversarial networks (gans) @xcite to improve intended semantic fidelity. another line of the generative model, normalizing flow @xcite , is widely explored in computer vision and representation learning but less explored for nlg tasks. flow models have been shown to be capable of improving probability density estimation, including variational inference @xcite and exact density estimation @xcite . generative flow is one type of flow model and first proposed by @xcite ; @xcite . taking advantage of its invertible structure, it can perform an exact density estimation of the input distribution. thus, during generation, we can sample from its latent space and then generate new examples through its invertible decoder. generative flow shows strong performance on image generation, attribute manipulation, and latent space inference @xcite . considering these successful applications, we conjecture that the flow model should also have strong potential to be adapted for language generation tasks. therefore, in this paper, we introduce a continuous language generative flow model that can deal with discrete language data in continuous latent space. we propose two variants, the non-autoregressive and autoregressive models, and show that they both can perform well on density estimation tasks. we follow the architecture of one previous generative flow model, @xcite , but make adaptions for language generation tasks. we first employ glove word embeddings @xcite to map the discrete token sequence to a continuous embedding matrix. furthermore, we utilize two components: time-dimension permutation and affine coupling with rnn or transformer non-linearity functions, which allow interaction between words in a se-quence and better contextualizes language semantics. overall, these proposed components help generate texts in a non-autoregressive manner. however, even though the non-autoregressive model has attracted a lot of research attention because of its fast generation speed, it still hardly surpasses the generation quality of autoregressive models @xcite . therefore, to make our language flow model learn language generation in a stronger autoregressive manner, we change the flow model's affine coupling and permutation to a uni-directional structure, i.e., each timestep can only attend to previous timesteps. in this way, we enable our model to perform text generation autoregressively. some recent works have developed density estimation models targeted on character-level discrete data (discreteflow @xcite ) and explored using the flow architecture as an extra data encoder that provides latent features to support nonautoregressive text generation (flowseq @xcite ). while our work shares some similar characteristics, we explore different directions: (1) dis-creteflow develops a modulus calculation method to process discrete data. instead, we use word embedding to transform the discrete input tokens to continuous features, which is simple yet effective. for density estimation, we compare the negative likelihoods of our models against a baseline lstm model. for qg, we use the non-autoregressive flow model to provide extra input features for a standard encoder-decoder text generation model. we show that it can significantly improve a baseline qg model for both squad and tvqa on both automatic and human evaluation metrics. aided by our flow model, we achieve strong improvements over a transformer baseline in the neural machine translation experiment. in addition to improving language generation quality, we also use the proposed autoregressive flow model for data augmentation. for this, we focus on generating diverse textual contexts for qa tasks. in particular, we inject noise into the latent features of our flow models (encoded from ground-truth contexts) and then generate new contexts from the noise-injected features. experiments show that the generated contexts can be either a varied expression of the same subject or paraphrasing the original context, but, mostly keep the answerability of the original question (see examples in overall, we have two contributions: (1) we propose two continuous language generative flow model variants that have better density estimation abilities than an lstm baseline model, and can perform non-autoregressive and autoregressive generation respectively; (2) our language flow model largely improves qg, nmt, and data augmentation for qa tasks. ## language generative flow in this section, we first review the generative flow model proposed in previous works @xcite . then, following it, we propose two variants of our continuous language generative flow model. ## experimental setup we follow @xcite to split the development set of squadv1.1 @xcite into two halves and show the result on the test split. we generally follow previous work on evaluation metrics. for density estimation, we use negative log-likelihood (nll) for comparison and bits per dimension to regularize the negative loglikelihood loss, formulated as l m log(2) , where m represents the dimension of input. we evaluate qg by bleu4 @xcite , meteor @xcite , rouge-l @xcite , and amazon mturk human evaluation. we use the bleu score to evaluate nmt. we use accuracy to evaluate the tvqa qa model and em (exact match) and f1 score to evaluate the squad qa model. we replicate zhang and bansal (2019)'s baseline qg model. we use the stage model with model tvqa subtitle squad article bi-lstm -7.31 -1.27 att-c 0.68 -2.01 rnn-c 0.50 -0.37 att-s -8.02 -17.12 rnn-s -8.35 -17.17 att-ar -9.62 -17.12 rnn-ar -9.63 -17.26 glove embeddings developed by @xcite as the tvqa qa baseline and use bert as the squad qa baseline. see appendix a for more experiment/reproducibility details. ## discussion we show some sample questions generated by our non-autoregressive and autoregressive flow models in while variational autoencoder has the issue of ignoring latent space @xcite , our models do not suffer from this issue. we introduced two types of language generation models in the paper: (1) the autoregressive flow model (used in data augmentation tasks) and (2) the model that uses flow latent features as extra input (e.g., for qg tasks). our autoregressive flow model's decoder is the inverted version of its encoder with the same weights, so it ensures the decoder uses the latent features. when we use flow latent features as extra inputs, it significantly improves qa performance ( ## conclusion we have proposed a language generative flow model with non-autoregressive and autoregressive variants. the non-autoregressive flow model achieves strong performance on density estimation and helps improve question generation and machine translation by providing additional useful latent features to the decoder. moreover, the autoregressive variant largely improves question answering by generating new contexts with noise injection.