M ac L a S a: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space (2023)

Abstract: Multi-aspect controllable text generation aims to generate fluent sentences that possess multiple desired attributes simultaneously. Traditional methods either require expensive iteration / searching within the discrete text space during the decoding stage, or train separate controllers for each aspect, resulting in a degradation of text quality due to the discrepancy between different aspects. To address these limitations, we introduce a novel approach for \textbf{M} ulti- \textbf{a} spect \textbf{c} ontrol, namely MacLaSa, that estimates compact \textbf{La} tent space for multiple aspects, and performs efficient \textbf{Sa} mpling with a fast sampler. To eliminate the domain discrepancies between different aspects, we first utilize a variational autoencoder (VAE) network to map text sequences from various data sources into close latent representations. The estimated latent space enables the formulation of joint energy-based models and the plugging in of arbitrary attribute discriminators to achieve multi-aspect control. Afterwards, we draw latent samples with a fast sampler based on ordinary differential equations and feed sampled examples to the VAE decoder to produce target text sequences. Experimental results demonstrate that MacLaSa outperforms strong baselines on both attribute relevance and textual quality while maintaining a high inference speed.

## introduction attribute-based controllable generation aims to generate text that exhibits desired attributes in certain aspects @xcite . early work focused on single-aspect control tasks and involved re-training or fine-tuning language models (lms) using well-labeled data, which resulted in good performance @xcite @xcite @xcite . recent studies focus on a more challenging and practical setting, multi-aspect controllable text generation foot_0 @xcite @xcite . for instance, a dialogue system may require the control of emotions, persona, politeness, etc, at the same time. however, training multi-aspect controllers directly is difficult due to the limited availability of sentences with multi-attribute annotations. thus, recent works focus on training separate single-aspect discriminators or controllers for each aspect and combining them for multi-aspect controllable text generation @xcite . as illustrated in figure 1 , recent works on multi-aspect controllable text generation task can be primarily categorized into two types. firstly, optimization-based methods either apply extra attribute classifiers to adjust the conditional probability distributions of language model at every generation step @xcite @xcite , or regard the decoding process as an optimization objective and search for optimal soft-representations that satisfy multi-objective constraints @xcite @xcite . however, from a distributional perspective, optimization-based methods often conduct complicated gradient-descent iterations or searching in the distorted text space, and the discrete nature makes it difficult to find high-quality texts, leading to poor linguistic quality and slow inference speeds. secondly, prefix-based methods are introduced to guide conditional generation using lightweight continuous task-specific vectors @xcite . they typically train single-aspect prefixes separately and suffer from text quality degeneration when combining them for multi-aspect control due to the mutual interference between multiple prefixes. as depicted in figure 1 , prefix-based methods combine multiple prefixes to obtain the interpolation or average of these distribution centers appraised by prefixes. however, there could be a mismatch between interpolation points and target intersection regions when the distribution centers of different aspects are far away, leading to the degradation of textual fluency. therefore, an ideal method for multi-aspect controllable generation should enhance controllability and textual quality, while enabling rapid inference speeds. in this paper, we introduce a new technique for multi-aspect controllable text generation, dubbed maclasa, which estimates a compact space containing latent representations of various attributes and performs effective sampling using a fast sampler based on ordinary differential equations (odes). to eliminate the domain discrepancies between different aspects, we initially employ a vae encoder network to map attribute-related sentences into latent representations and penalize the distance between each pair of aspect distribution centers. the acquired compact latent space aids in formulating joint latent-space energy-based models (ebms) and allows us to integrate arbitrary attribute discriminators to satisfy multi-aspect combinations. subsequently, we utilize an efficient ode-based sampler @xcite to draw latent samples possessing desired attributes from the distribution formed by multiple attribute classifiers. ultimately, the selected latent vectors are input into a vae decoder to generate target text sequences. in short, our approach improves controllability and textual quality by estimating a compact latent space to mitigate mutual interference among various aspects, and the fast ode-based sampler contributes to efficient sampling. we conduct experiments on the multi-aspect control task with two attributes from the sentiment aspect and four attributes from the topic aspect, with datasets imdb movie reviews @xcite and agnews @xcite , respectively. experimental results of both automatic and human evaluation demonstrate that our method achieves encouraging improvements in attribute relevance and text quality compared to previous strong baselines. our work also exhibits significant advantages in inference speed over existing baselines foot_1 . ## related work in this section, we discuss the related work on multi-aspect control. recent researches on multiaspect can be divided into two types: optimizationbased methods and prefix-based methods. ## methodology in this section, we first present the task definition of multi-aspect controllable text generation ( §3.1). next, we describe how to build the compact latent space ( §3.2), how to define the joint ebms on the latent space ( §3.3), and how to sample from the ebms to generate the final results ( §3.4). the overall structure of maclasa is illustrated in figure 2 . our approach primarily relies on the variational autoencoder architecture for manipulating latent spaces. to weaken the mutual interference among different aspects, we initially employ the vae encoder to estimate a continuous lowdimensional latent space, incorporating additional losses to ensure its compactness. subsequently, we establish joint latent-space energy-based models, which allow us to integrate multiple constraint functions for guiding sophisticated multi-aspect control. finally, we utilize a fast ode-based sampler to draw samples from the ebms and input them into the vae decoder to generate the desired multi-aspect sequences. ## experiments in this section, we demonstrate the effectiveness of our proposed maclasa in the multi-aspect control setting through both automatic and human evaluations. additionally, we provide further analysis and visualization on efficiency, and case studies. ## conclusion and future work in this study, we introduce a novel method, namely maclasa, for multi-aspect controllable text generation that estimates a compact, low-dimensional latent space and employs a fast ode-based sampler for efficient sampling. our experiments on the two-aspect control task demonstrate the effectiveness and efficiency of our approach. additionally, we carry out in-depth analytical experiments to emphasize the impact of each module and visualize the estimated latent space. in the future, we aim to expand our work by incorporating arbitrary attribute discriminators into the diffusion process using a plug-and-play approach. furthermore, we plan to explore more powerful models to enhance the linguistic quality of generated sentences.