Controlled Text Generation Using Dictionary Prior in Variational Autoencoders (2022)

Abstract: While variational autoencoders (VAEs) have been widely applied in text generation tasks, they are troubled by two challenges: insufficient representation capacity and poor controllability. The former results from the posterior collapse and restrictive assumption, which impede better representation learning. The latter arises as continuous latent variables in traditional formulations hinder VAEs from interpretability and controllability. In this paper, we propose Dictionary Prior (DPrior), a new data-driven prior that enjoys the merits of expressivity and controllability. To facilitate controlled text generation with DPrior, we propose to employ contrastive learning to separate the latent space into several parts. Extensive experiments on both language modeling and controlled text generation demonstrate the effectiveness of the proposed approach.

## introduction as one of the representative deep generative models, variational autoencoders (vaes) @xcite have been widely applied in text generation tasks, such as dialog generation @xcite , machine translation @xcite @xcite and poetry generation @xcite . despite the success, vaes still suffer from two challenges: insufficient representation capacity and poor controllability. the challenge of insufficient representation capacity in variational models arises from two aspects. one is the posterior collapse, a notorious issue that generally exists in vaes especially serious in autoregressive text generation @xcite , which leads to degenerate local optimums during the training of vaes @xcite . another is the restrictive assumption for priors and variational another challenge of vaes is poor controllability. the challenge is rooted in the continuous latent variables that hinder vaes from interpreting the discrete attributes like sentiments or topics @xcite . thus it is difficult to generate text with user-specified attributes, as the examples in in this paper, we follow the practice of learning discrete representations and propose a new datadriven prior that enjoys the merits of expressivity and controllability. specifically, we deploy a set of learnable vectors and interpolate the learnable vectors to form the prior, which we call dictionary prior (dprior). each learnable vector is dubbed an atom in the dictionary. to facilitate generative models with dprior, dual-form kl-divergence @xcite is employed to make the prior distribution spanned by dictionary atoms approximate the posterior distribution. our dprior is modelagnostic and could be combined with pre-trained models such as bert/gpt to enrich posterior representations @xcite . to enforce controllability to the dprior, we separate the dictionary atoms into several disjoint subsets according to the natural language attributes. then, we propose to employ contrastive learning to incorporate the attribute information, which can cluster different subsets of dictionary atoms into different semantic subspaces. we demonstrate the superiority of dprior against recent vae variants on the language modeling task. we also validate our dprior in controlled text generation where dprior shows its effectiveness over several advanced counterparts. the main contributions of this paper can be summarized as: ## related work controlled text generation controlled text generation is a task aiming to generate realistic sentences with desired attributes, e.g., sentiments or topics. most efforts for controlled text generation are based on conditional pre-trained language models @xcite . ctrl @xcite employs a gpt-2 like pre-trained language model and trains it from scratch on a large corpus containing various control codes. subsequently, controlled generation is accomplished by using the control codes as prompting words. pplm @xcite seeks to avoid the further training process and combines the gpt-2 model with several simple attribute classifiers whose gradients can update the latent representations. another line of work tries to explore limited labeled data via learning latent representations @xcite . @xcite propose to approach controlled text generation by learning disentangled latent representations including independent content and attribute parts. in this paper, we learn entangled latent representations and approach controlled text generation by separating prior space into several parts. ## methodology in this section, we first review the basics of deep generative models in section 3.1, then introduce dictionary prior (dprior) in section 3.2 which is built on a set of learnable vectors. we further approach controlled text generation in section 3.3. the overall illustration of our proposed approach is shown in figure 1 . more details will be explained in the following sections. ## experiments in this section, we apply dprior model to two tasks: (i) language modeling, where dprior shows its advantage in expressive prior in comparison with state-of-the-art vae methods. (ii) controlled text generation, where dprior shows its superiority in controllability with desired attributes. we also conduct a series of analyses and visualizations to shed more light on the proposed approach. ## conclusion in this paper, we propose the dictionary prior (dprior), a new data-driven prior that enjoys the merits of expressivity and controllability. the proposed prior deploys a set of learnable vectors dubbed as dictionary atoms and interpolate the atoms to form the prior. we apply dual-form kldivergence to make the prior distribution spanned by dictionary atoms approximate the posterior distribution. contrastive learning is further deployed to the disjoint dictionary subsets to enable controllability and interpretability. empirical results on benchmark datasets demonstrate the superiority of our approach in both language modeling and controlled text generation. nevertheless, the proposed approach has limitations. while the gaussian distribution employed in standard vaes has an infinite support region, the support region of dprior is finite as it corresponds to the convex hull of the dictionary atoms. therefore, future work considers extending our framework to the more general infinite support region. we will also apply dprior to more text generation tasks like poetry generation @xcite and machine translation @xcite .