VOLTA : Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder (2024)

Abstract: The natural language generation domain has witnessed great success thanks to Transformer models. Although they have achieved state-of-the-art generative quality, they often neglect generative diversity. Prior attempts to tackle this issue suffer from either low model capacity or over-complicated architectures. Some recent methods employ the VAE framework to enhance diversity, but their latent variables fully depend on the input context, restricting exploration of the latent space. In this paper, we introduce VOLTA, a framework that elevates generative diversity by bridging Transformer with VAE via a more effective cross-attention-based connection, departing from conventional embedding concatenation or summation. Additionally, we propose integrating InfoGAN-style latent codes to enable input-independent variability, further diversifying the generation. Moreover, our framework accommodates discrete inputs alongside its existing support for continuous inputs. We perform comprehensive experiments with two types of Transformers on six datasets from three different NLG tasks to show that our approach can significantly improve generative diversity while maintaining generative quality.

## introduction the rapid advancement of natural language generation (nlg) has been propelled by the remarkable success of transformer models, including the notable series of gpt models @xcite @xcite @xcite , t5 @xcite , opt @xcite , and llama @xcite , etc. while they have demonstrated unparalleled proficiency in autoregressive text generation @xcite @xcite , they predominantly focus on learning to reassemble text from large corpora with high generative quality. however, the pursuit of generative diversity remains a critical yet underexplored context atop the main building's gold dome is a golden statue of the virgin mary. immediately in front of the main building and facing it, is a copper statue of christ with arms upraised with the legend "venite ad me omnes". next to the main building is the basilica of the sacred heart. immediately behind the basilica is the grotto •••••• q what type of statue is on the main building? a golden statue of the virgin mary q what is the name of the copper statue on the main building? a a copper statue of christ with arms upraised with ••• q what is next to the main building? a grotto variational autoencoder (vae) (kingma and welling, 2014) offers a framework addressing the low-diversity issue. by encoding inputs into lowerdimensional latent variables, vae introduces the opportunity to diversify the decoding process: perturbing these latent variables allows generated sentences to deviate from annotated ones, thereby enhancing diversity. however, prior attempts like info-hcvae @xcite , utilizing lstmbased vaes, inherit limitations associated with lstms. while transformers have emerged as the mainstream network, integrating them into the vae framework poses challenges due to the parallelized self-attention mechanism. more precisely, this complexity arises from inserting a bottleneck layer of latent variables between transformer layers, as the embeddings of the entire sequence pass through the model simultaneously. optimus @xcite , pioneering the fusion of transformers with vaes, adopts bert @xcite as the vae encoder and gpt-2 (radford et al., 2019) as the vae decoder. subsequent works attempt to improve upon optimus @xcite @xcite ), yet they fall short in addressing its three major drawbacks. firstly, it introduces embedding concatenation and summation to connect latent variables to the decoder, with optimus performing optimally only upon their combined use. in contrast, our novel cross-attention-based connection proves more effective. secondly, optimus' model architecture is overly intricate. it relies on two distinct transformer models, necessitating two unique tokenizers and extensive pretraining. we streamline this complexity by either employing a shared transformer decoder as the backbone network or leveraging an encoder-decoder transformer model. this renders our framework compatible even with large language models (llms) such as llama @xcite or @xcite . lastly, while optimus solely handles continuous latent variables, volta expands its scope to cover discrete inputs by encoding them into discrete latent variables, enriching the model's generalizability. the vae framework offers increased generative diversity, yet its input-dependent latent variables limit exploration within the latent space, restricting the model's ability to generate a wider array of diverse content. in pursuit of an input-independent approach to vary the generation process, we pro-pose attaching latent codes to vae latent variables, inspired by infogan @xcite . our method employs the variational mutual information maximization (vmim) objective to encourage the decoder to autonomously identify distinct semantic features via latent codes. consequently, this enables more variability in generated content without any reliance on the input. to the best of our knowledge, our work represents the first utilization of latent codes within nlg. our framework, dubbed volta (variational mutual information maximizing autoencoder), derives its name from its adherence to the variational autoencoder framework and the incorporation of the variational mutual information maximization objective from infogan. to validate the effectiveness of volta, we benchmark it against state-of-the-art baseline models across six datasets from three representative nlg tasks: language modeling, question-answer generation, and dialog response generation. we also conduct comprehensive ablation studies to examine the impact of the different components of volta. the main contributions of this paper are: ## related work in recent years, a multitude of transformer-based models has emerged, such as the gpt series @xcite @xcite @xcite , t5 @xcite , opt @xcite , and llama @xcite , etc. these models are primarily trained to optimize the alignment between generated content and annotations, often prioritizing quality over diversity in the generative process. variational autoencoders (vaes) (kingma and welling, 2014) represent a powerful approach to diverse generation in nlg. they diverge from autoencoders @xcite by introducing low-dimensional latent variables. originally applied in computer vision, vaes were later adapted for natural language processing. early attempts, such as info-hcvae (lee et al., 2020), employed lstms @xcite as both encoder and decoder, achieving diversity in question-answer generation (qag). however, these lstm-based models suffered from architectural complexities, utilizing separate lstm modules for encoding and decoding context, questions, and answers. optimus @xcite addressed some of these challenges by using bert @xcite as the encoder and gpt-2 @xcite as the decoder, surpassing lstm-based models in vae language modeling. subsequent models like varmae @xcite focused on applying vaes in language understanding, while regavae @xcite attempted augmentation through retrieval methods, and adavae @xcite explored the usage of two adaptive gpt-2 models. our volta model further simplifies the architecture by leveraging a shared backbone network or utilizing an encoderdecoder transformer model. in pursuit of more variability and subsequently, higher diversity, several methods have employed unique strategies such as special prompt tokens or control phrases. these include simpletod (hosseini-asl et al., 2020), ctrl @xcite , soloist @xcite , cgrg @xcite , and megatron-cntrl @xcite . @xcite proposed the plug and play language model, which guides language generation by plugging simple attribute classifiers into existing language models. infogan @xcite originally controlled image generation using latent codes trained with the variational mutual information maximization (vmim) objective. in computer vision, attempts to merge info-gan with vae for controllable generative models have resulted in models like vae-info-cgan @xcite and infovaegan @xcite . however, infovae @xcite , infomax-vae @xcite , @xcite , and vae-mine @xcite applied vmim to vae to address the latent variable collapse problem rather than focusing on improving variability. to the best of our knowledge, our model is the first to integrate transformer models with the vae and infogan frameworks in natural language generation (nlg). although we focus on diversity in this paper, other aspects of nlg are also worth exploring in the future @xcite , such as multi-modality, bias, and fairness. ## our method our volta framework is meticulously designed to facilitate diverse generation, leveraging latent variables from the vae framework (kingma and welling, 2014) in conjunction with infogan-style latent codes @xcite . initially, volta encodes the input into latent variables. subsequently, by sampling new latent variables, slight alterations in the decoded content can be achieved, promoting greater diversity. differing from vae latent variables, infogan-style latent codes operate independently of input, providing the freedom to explore a broader latent space. this distinct attribute offers an alternative avenue to introduce increased variability within the generated sequences. figure 1 includes an overview of volta. ## conclusion we present volta, a framework merging the power of transformers with the variability inherent in vae and infogan. diverging from prior approaches, volta introduces a novel crossattention-based connection linking the latent space to the decoder, enhancing stability in optimization. this innovative architecture accommodates diverse transformer types, including decoder-only or encoder-decoder architectures, and supports varying input types, be it continuous or discrete. additionally, our framework incorporates infoganstyle latent codes, enabling input-independent variability, thereby further enriching generative diversity. comprehensive experiments across six datasets spanning three distinct nlg tasks showcase volta's significant enhancement in generative diversity while preserving quality.