A Hierarchical VAE for Calibrating Attributes while Generating Text using Normalizing Flow (2021)

Abstract: In this digital age, online users expect personalized content. To cater to diverse group of audiences across online platforms it is necessary to generate multiple variants of same content with differing degree of characteristics (sentiment, style, formality, etc.). Though text-style transfer is a well explored related area, it focuses on flipping the style attribute polarity instead of regulating a fine-grained attribute transfer. In this paper we propose a hierarchical architecture for finer control over the at- tribute, preserving content using attribute dis- entanglement. We demonstrate the effective- ness of the generative process for two different attributes with varied complexity, namely sentiment and formality. With extensive experiments and human evaluation on five real-world datasets, we show that the framework can generate natural looking sentences with finer degree of control of intensity of a given attribute.

## introduction the ubiquity of online social networks and world wide web has brought in diverse and often conflicting groups of users consuming similar information but from different perspectives. so the onus falls on the content producer to cater customized content based on the users' profile. consider an example related to a spanish football (soccer) league. say the news is "barcelona has defeated real madrid". this news needs to be presented in different tones to a barcelona fan -"barcelona smashed real-madrid", a real-madrid fan -"real madrid lost the epic battle" and a (say) villarreal fan -"barcelona wins three points against real-madrid". automatic generation of content with fine regulation of attributes like sentiment and style is extremely beneficial in this context. there are several related works in similar space of text-style-transfer techniques @xcite @xcite which attempt to switch polarity of a text from, e.g., formal to casual, or positive to negative sentiment. however, none of the work focuses on more involved problem of fine-grained regulation of attributes to generate multiple variants of a sentence. several of the existing style-transfer methods @xcite convert a continuous entangled generative representation space obtained using variational auto-encoder @xcite into disentangled attribute and content space. it facilitates attribute polarity switch by perturbing attribute representation without interfering with context. however, a disentangled generative representation may result in a loss of information about complex inter-dependency of content and attributes otherwise captured in an unmodified entangled generative space. hence, trivial extension of the variational inference (encoding) mechanism for finer attribute control by allowing incremental perturbation of the attribute representation in the disentangled generative space often leads to generation of 'not-so-natural' sentence mostly unrelated to the original content. more specifically, there are two design challenges which need to be tackled to achieve fine grained attribute control (a) smooth regulation of attributes via disentangled attribute space perturbation and (b) natural sentence generation preserving the content. this paper builds up a layered vae to tackle these problems simultaneously. specifically, we propose the model control text vae (ctvae), that transforms a derived representation of entangled and enriched text embedding (obtained using the bert encoder) into a disentangled representation of attribute and context using a transformation module followed by a factored prior imposition to ensure independence between context and attribute dimensions. further using attribute supervision on the dimension designated for a given attribute, we establish a correlation between the continuous representation to the discrete attribute value facilitating smooth interpolation as intended in (a). it preserves both the disentangled and entangled representations in different hierarchy of inference module. designing the transformation network as reversible, it restores the original entangled sentence representation which is our generative space, from the disentangled space to achieve (b). we demonstrate the effectiveness of ctvae to generate controlled text by fine tuning two different attributes namely sentiment and formality. using five publicly available datasets, we show that ct-vae improves the performance significantly over previous controlled text generative models while performing content preserving style transfer and fine tuning of the target attribute. with human evaluation on generated sentences, for three different metrics -meaning preservation, degree of target attribute transfer and naturalness -we show that ctvae can generate attribute regulated content preserving natural sentences. foot_0 2 related work unlike style-transfer, fine grained attribute regulated text generation is less explored yet extremely necessary. state-of-the-art methods for style transfer are categorized as supervised and unsupervised techniques. if parallel examples are available for any attribute, i.e., training data consisting of original and corresponding attribute flipped sentences, then supervised techniques @xcite could be used to perform style transfer. the papers @xcite @xcite introduced parallel corpora consisting of formal and corresponding informal sentences and showed that coarse-grained formality transfer is possible and benchmarked various neural frameworks for the same. generating parallel training corpus for fine grained attribute transfer is expensive and impractical as for one sentence we need to generate multiple style transferred text bearing fine-grained attribute. some recent works focus on semi-supervised approaches incorporating attribute informations with non-parallel datasets. these techniques mainly focus on disentangling the attribute and content representation in the latent space @xcite @xcite @xcite by using different encoding modules along with feature supervision. a recent work @xcite uses adversarial setup in a multitasking setting to achieve attribute representation independent of the content. as this work disentangles context and attribute in multidimensional spaces it limits interpolation of the attribute space to desired degree. moreover, the disentangled generative space causes loss in important context. similarly, the paper @xcite uses attribute information as a structured or one-hot vector, which is not continuous restricting interpolation. they replace the attribute representation to a desired value (corresponding to opposite polarity) and generate sentences from this disentangled space. however, a naive extension for fine grained control by perturbing the attribute space by a small amount is difficult as the representation is multidimensional moreover, leads to unnatural, poorly readable sentence. from a different perspective, a recent work @xcite proposed an unsupervised framework to achieve style transfer. they propose a generative probabilistic model that assumes non-parallel corpus as partially observed parallel corpus. they do not infer posterior distribution of the observed data, hence fine grained attribute transfer is difficult. as the extensions of current style transfer methods are non-trivial, a recent work @xcite has proposed fine grained sentiment regulation keeping the content intact. it gradually updates the entangled latent representation using costly fastgradient-iterative modification until it can generate a sentence entailing target attribute from that . however, overemphasis on content preservation often results in generation of the original unmodified sentence followed by new phrases bearing target attribute. this is not ideal to extend them for more difficult attributes like casual to formal transformation. understanding the criticality of fine grained attribute transfer, we propose a new framework towards this direction, which does not only facilitate fine-grained control even for complex attributes, but is also able to mitigate the existing problems of disentangled generative space. ## ctvae for fine grained control we propose a hierarchical model using variational autoencoders @xcite to achieve fine grained control over attribute space while maintaining the quality of the generated sentences. we provide a high level overview of ct-vae along with key technical aspects of the individual components followed by training procedure. ## experiments we broadly looked into two evaluation criteria to compare the performance of different generative models (a) attribute control: efficiency in generating sentences entailing target attribute of interest (b) fine-grained transfer: efficiency of content preserving fine-grained attribute regulated text generation. in this section we discuss datasets, baselines followed by the performance across datasets. ## conclusion the major contribution of this paper is to propose ctvae which consists of a carefully designed hierarchical architecture facilitating disentangled representation to control attribute without affecting context as well as enriched entangled generative representation for meaningful sentence generation. the invertible normalizing flow as a transformation module between the two representation of ct-vae enables learning of complex interdependency between attribute and context without the loss of information. such a design choice is key to achieving accurate fine tuning of attributes (be it sentiment or formality) while keeping the content intact. this is a key achievement considering the difficulty of the problem and modest performance of state-of-the-art techniques. extensive experiments on real-world datasets emphatically establish the well-rounded performance of ctvae and its superiority over the baselines.