Polarized- VAE : Proximity Based Disentangled Representation Learning for Text Generation (2021)

Abstract: Learning disentangled representations of realworld data is a challenging open problem. Most previous methods have focused on either supervised approaches which use attribute labels or unsupervised approaches that manipulate the factorization in the latent space of models such as the variational autoencoder (VAE) by training with task-specific losses. In this work, we propose polarized-VAE, an approach that disentangles select attributes in the latent space based on proximity measures reflecting the similarity between data points with respect to these attributes. We apply our method to disentangle the semantics and syntax of sentences and carry out transfer experiments. Polarized-VAE outperforms the VAE baseline and is competitive with state-of-the-art approaches, while being more a general framework that is applicable to other attribute disentanglement tasks.

## introduction learning representations of real-world data using deep neural networks has accelerated research within a number of fields including computer vision and natural language processing @xcite . previous work has advocated for the importance of learning disentangled representations @xcite . although attempts have been made to formally define disentangled representations @xcite , there is no widely accepted definition of disentanglement. however, the general consensus is that a disentangled representation should separate the distinct factors of variation that explain the data @xcite . intuitively, a greater level of interpretability can be achieved when independent latent units are used to encode different attributes of the data @xcite . however, recovering and separating all the distinct factors of variation in the data is a challenging problem. for real-world datasets, there may not be a way to separate each factor of variation into a single dimension in the learnt fixed-size vector representation. an easier problem would be to separate complex factors of interest into distinct subspaces of the learnt representation. for instance, a representation for text could be separated into content and style subspaces, which then enables style transfer. unsupervised disentanglement of underlying factors using variational autoencoders (kingma and welling, 2014) has been explored in previous work @xcite . however, @xcite argue that completely unsupervised disentanglement of the underlying factors may be impossible without supervision or inductive biases. disentangling textual attributes in a completely unsupervised manner has been shown to be especially difficult, but attempts have been made to leverage it for controllable text generation @xcite . in this work, we propose an approach referred to as polarized-vae foot_0 to disentangle the latent space into subspaces corresponding to different factors of variation. we control the relative location of representations in a particular latent subspace based on the similarity of their respective input data points according to a defined criterion (that corresponds to an attribute in the input space, e.g., syntax). this encourages similar points to be grouped together and dissimilar points to be farther away from each other in that subspace. figuratively, we polarize the latent subspaces, and hence the name. most previous work on supervised disentanglement for text has focused on adversarial training @xcite . recently, the task of disentangling textual semantics and syntax into distinct subspaces has received attention from researchers. for instance, @xcite use a sentence vae model with several multitask losses such as paraphrase loss and word position loss for this disentanglement task. @xcite incorporate adversarial training and make use of syntax trees along with specific multitask losses to disentangle semantics and syntax. in polarized-vae, we achieve disentanglement through distance based learning. in contrast to previous approaches, our method does not require the use of several multitask losses or adversarial training, both of which can result in optimization challenges. furthermore, we do not need precise attribute labels, and we show that using proxy labels based on the concept of similarity is sufficient. in summary, the main contributions of this paper are three-fold: (1) we propose a general framework for learning disentangled representations. even though we test our method on an nlp task, the underlying concept is very general and can be applied to other domains such as computer vision; (2) we provide a method for disentanglement that does not rely on adversarial training or specialized multitask losses; (3) we demonstrate an application of our method by disentangling the latent space into subspaces corresponding to syntax and semantics. such a setting can be used to perform controlled text decoding such as generating a paraphrase with a desired sentence structure. ## proposed approach in vaes, a probabilistic encoder q φ (z|x) is used to encode a sentence x into a latent variable z, and a probabilistic decoder p θ (x|z) attempts to reconstruct the original sentence x from its latent representation z. the objective is to minimize the following loss function: the idea behind our polarized-vae approach is to impose additional proximity regularization on the latent subspaces learnt by vaes. let @xmath0, ..., c k } be the collection of criteria, based on which we wish to disentangle the latent space z of the vae into k subspaces: @xmath1, . . . , z (k) ]. here z (i) denotes the latent subspace corresponding to the criterion c i (see figure 1 ). in this paper, we focus on the case where the latent space is disentangled into semantics (c 1 ) and syntax (c 2 ), i.e., @xmath2. ## conclusion and future work we proposed a general approach for disentangling latent representations into subspaces using proximity functions. given a pair of data points, a predefined similarity criterion in the original input space determines their relative distance in the corresponding latent subspace, which is modelled via a proximity function. we apply our approach to the task of disentangling semantics and syntax in text. our model substantially outperforms the vae baseline and is competitive with the state-of-theart approach while being more general as we do not use specific multitask losses or architectures to encourage preservation of semantic or syntactic information. our methodology is orthogonal to the multitask learning approaches by @xcite and @xcite and can be naturally combined with their methods. we would further like to investigate this approach on disentanglement applications outside of nlp. another interesting research direction would be to further explore suitable proximity functions and identify their properties that could facilitate disentanglement. a woman showing her face from something to her friend. a man directing traffic on a bicycle to an emergency vehicle. a woman works on a loom while sitting outside. a family in a party preparing food and enjoying a meal.