A Distributional Lens for Multi-Aspect Controllable Text Generation (2022)

Abstract: Multi-aspect controllable text generation is a more challenging and practical task than single-aspect control. Existing methods achieve complex multi-aspect control by fusing multiple controllers learned from single-aspect, but suffer from attribute degeneration caused by the mutual interference of these controllers. To address this, we provide observations on attribute fusion from a distributional perspective and propose to directly search for the intersection areas of multiple attribute distributions as their combination for generation. Our method first estimates the attribute space with an autoencoder structure. Afterward, we iteratively approach the intersections by jointly minimizing distances to points representing different attributes. Finally, we map them to attribute-relevant sentences with a prefix-tuning-based decoder. Experiments on the three-aspect control task, including sentiment, topic, and detoxification aspects, reveal that our method outperforms several strong baselines on attribute relevance and text quality and achieves the SOTA. Further analysis also supplies some explanatory support for the effectiveness of our approach.

## introduction controllable text generation is a challenging task in natural language generation, which aims to generate fluent text with desired attributes. pilot studies attempt single-aspect control by directly finetuning a conditional model @xcite , or turn to methods with language models fixed @xcite due to the high cost of large-scale pre-trained language models @xcite . recent works focus on a more practical setting, multi-aspect 2 controllable text generation, with existing approaches mainly divided into three tech1 our dataset and code are available at: https://github .com/happygu0524/multicontrol 2 for example, positive is an attribute from sentiment aspect while sports is an attribute from topic aspect. ## related work variational autoencoders are often used for controllable text generation in early work @xcite @xcite where they spend a lot of effort into improving text fluency. the prosperity of large-scale pre-trained language models @xcite provides more exploration directions for attribute control such as fine-tuning @xcite @xcite . recent work has made gratifying progress on single-aspect control @xcite , leading studies gradually turn to a more difficult task, multi-aspect control, including the following three main approaches. weighted decoding as the scale of language models increases rapidly, weighted decoding @xcite @xcite @xcite becomes a simple and practical choice. it is a framework that decomposes the probability of sentences conditioned on attributes into a language model and a classifier with the bayesian rule directly at decoding time. when handling multiaspect control, it can be easily generalized by interpolating classifiers @xcite . ## methodology in this section, we first introduce the motivation and overall process of our method, after which we describe each module in detail. ## experiment in this section, we demonstrate the effectiveness of our method on three-aspect control, including sentiment, topic, and detoxification. ## discussion on distributional lens pilot work such as dgc @xcite estimates the language distribution with an energybased model and optimizes this distribution to satisfy constraints by approaching the constraints manifold. recent distributional approaches like cold decoding @xcite and mucola @xcite take the language and attribute distribution in the same space so as to sample attributerelated sentences with langevin dynamics. concurrent work on the image side, promptgen @xcite , simulates the complex distribution of images relevant to target attributes using a deep generative model. however, as a consensual hypothesis in manifold learning, the pre-trained language model estimates a low-dimensional manifold of language in a high-dimensional embedding space, which means most points in the embedding space are not probabilistically modeled by the language model. we believe that placing too much trust in the distributional modeling ability of language models is not a good choice. our method attempts to depict the attribute space with discrete sample points of attributed sentences and make these discrete points, along with their coverage areas, compose the support set of our estimated distribution. ## conclusion in this work, we present a distributional perspective for the multi-aspect controllable text generation with experimental results confirming the superiority of our model. further observations on the 2d projection of the estimated attribute space show that our hypothesis about the attribute space is more feasible. in the future, we can explore the correlation between different attribute combinations for more fine-grained control and capture the bias in datasets to eliminate or utilize it.