On the Effect of Isotropy on VAE Representations of Text (2022)

Abstract: Injecting desired geometric properties into text representations has attracted a lot of attention. A property that has been argued for, due to its better utilisation of representation space, is isotropy. In parallel, VAEs have been successful in areas of NLP, but are known for their sub-optimal utilisation of the representation space. To address an aspect of this, we investigate the impact of injecting isotropy during training of VAEs. We achieve this by using an isotropic Gaussian posterior (IGP) instead of the ellipsoidal Gaussian posterior. We illustrate that IGP effectively encourages isotropy in the representations, inducing a more discriminative latent space. Compared to vanilla VAE, this translates into a much better classification performance, robustness to input perturbation, and generative behavior. Additionally, we offer insights about the representational properties encouraged by IGP.

## introduction in recent years, with the success facilitated by pretrained representations across various nlp tasks, more attention has been placed on studying and utilising the geometric properties of learned representations. a phenomena that has been studied more recently in this direction is anisotropy (ethayarajh, 2019), indicating a sub-optimal property where the learned embeddings only utilise a small subset of the representation space. various methods have been proposed to rectify this and encourage the representations to be more discriminative or to exploit the representation dimensions more effectively @xcite @xcite @xcite . in parallel, variational autoencoders (vaes) @xcite have been widely used in various areas of nlp, from representation learning for downstream tasks @xcite , to generation @xcite , representational sparsity and disentanglement @xcite , and semi-supervised learning @xcite @xcite . in recent years, most of the developments around vaes have focused on avoiding the posterior collapse @xcite which leads to learning sub-optimal representations @xcite @xcite @xcite @xcite . despite the success of these techniques, a non-collapsed vae still utilises the representation space sub-optimally @xcite @xcite , as very commonly the learned representations do not fully utilise the latent space to encode information. in this paper we bridge between the two lines of research by injection isotropy in the latent space of vaes. such property could be encouraged by using an isotropic gaussian posterior (igp) which involves a simple modification of vaes. an isotropic gaussian distribution, n (µ, σ foot_1 i), is similar to vanilla vae's posterior with the exception that all dimensions share the same unified variance. tying the variances would encourage encoder of vaes towards the extreme where all dimensions are either active or inactive. 2 our experimental findings indicate that, compared to vanilla vae, the use of igp is effective in both increasing dimension activation and injecting isotropy in the learned representation space. we observe that isotropy results in a more discriminative representation space which is much more suited for classification tasks and robust to input perturbation. our generative experiment for sentence completion suggests that the vae trained with igp is more capable of maintaining semantic cohesiveness. ## isotropic gaussian posterior (igp) variational autoencoder (vae). let x denote datapoints in data space and z denote latent variables in the latent space, and assume the datapoints are generated by the combination of two random processes: the first random process is to sample a point z (i) from the latent space in vaes with prior distribution of z, denoted by p(z). the second random process is to generate a point x (i) from the data space, denoted by p(x|z (i) ). vae uses a combination of a probabilistic encoder q φ (z|x) and decoder p θ (x|z), parameterised by φ and θ, to learn this statistical relationship between x and z. vae is trained by maximizing the lower bound of the logarithmic data distribution log p(x), called evidence lower bound (elbo), l(φ, θ; x): the first term of objective function is the expectation of the logarithm of data likelihood under the posterior distribution of z. the second term is kldivergence, measuring the distance between the recognition distribution q φ (z|x) and the prior distribution p(z) and can be seen as a regularisation. in the presence of auto-regressive and powerful decoders, a common optimisation challenge of training vaes in text modelling is called posterior collapse, where the learned posterior distribution q φ (z|x), collapses to the prior p(z). several strategies have been proposed to alleviate this problem @xcite @xcite . in this work, we follow @xcite , l(φ, θ; x): ## experiments we trained our models on yahoo question and dbpedia @xcite which have (100k/10k/10k, 12k, 10) and (140k/14k/14k, 12k, 14) for (sentences in training/dev/test, vocabulary size, classes), respectively. we use one unidirectional lstm layer for encoder and decoder, and fully-connected layers to produce mean and variance of posteriors. we concatenate the latent figure 1 : results are calculated on the test set (average of 3 runs reported) of (top) dbpedia corpus and (bottom) yahoo question @xcite . au is bounded by the dimensionality of z (32). we compare the choice of isotropic gaussian posterior (igp) with vanilla diagonal gaussian posterior (dgp) on various grounds, from reconstruction loss and unit activation ( §3.1) to downstream classification task, sample efficiency, robustness, and generation ( §3.2), posterior sharpness ( §3.3), and distributional properties of the induced representations ( §3.4). ## conclusion we proposed isotropic gaussian posteriors (igp) as a means of encouraging isotropy in the latent space induced by vaes. the injection of isotropy addressed a sub-optimal behaviour of vaes by activating more dimensions of the representation and encouraging a more discriminative latent space. our experiments illustrated a significant improvement of classification performance and robustness to input perturbations with igp. we also observed, in the sentence completion task, that vae trained with igp is more capable at maintaining semantic cohesiveness. our ongoing work suggests the representation utilisation achieved by igp has the potential to be exploited towards representational properties such as disentanglement.