Towards Robust and Semantically Organised Latent Representations for Unsupervised Text Style Transfer (2022)

Abstract: Recent studies show that auto-encoder based approaches successfully perform language generation, smooth sentence interpolation, and style transfer over unseen attributes using unlabelled datasets in a zero-shot manner. The latent space geometry of such models is organised well enough to perform on datasets where the style is “coarse-grained” i.e. a small fraction of words alone in a sentence are enough to determine the overall style label. A recent study uses a discrete token-based perturbation approach to map “similar” sentences (“similar” defined by low Levenshtein distance/ high word overlap) close by in latent space. This definition of “similarity” does not look into the underlying nuances of the constituent words while mapping latent space neighbourhoods and therefore fails to recognise sentences with different style-based semantics while mapping latent neighbourhoods. We introduce EPAAEs (Embedding Perturbed Adversarial AutoEncoders) which completes this perturbation model, by adding a finely adjustable noise component on the continuous embeddings space. We empirically show that this (a) produces a better organised latent space that clusters stylistically similar sentences together, (b) performs best on a diverse set of text style transfer tasks than its counterparts, and (c) is capable of fine-grained control of Style Transfer strength. We also extend the text style transfer tasks to NLI datasets and show that these more complex definitions of style are learned best by EPAAE. To the best of our knowledge, extending style transfer to NLI tasks has not been explored before.

## introduction the text style transfer (tst) task is a form of controlled language generation. the goal is to produce fluent style-altered sentences from a given base sentence, while also preserving its style-independent content. the definition of "style" depends on the class labels of the end task. our work focuses on the unsupervised scenario i.e. performing the training on completely unlabelled corpora. by inducing the latent space organization through input perturbation, tst can be performed using a simple vector arithmetic method (discussed in section 6). background. several well-known architectures use auxiliary objectives that serve as regularizers to ensure that the latent space geometry is smoothly interpolatable and learns high-level semantic features (refer appendix d). inspired by successes in denoising approaches in vision @xcite , we look at input perturbation based approaches for unsupervised style transfer. unlike text, in vision, freedom exists to finely control the degree of gaussian "blur" on the continuous input image space. we conjecture that this notion of "controlling blur" may be useful in the text domain as well, serving as our motivation for our chosen model for input perturbation. @xcite map "similar" (defined by low levenshtein distance/ high word overlap) sentences together in the latent space by introducing a simple denoising objective over an underlying adversarial autoencoder (aae) @xcite . this noise model includes simple token manipulation (token dropout and substitution) with some probability p, to reconstruct the original sentences from the perturbed inputs. we can reason intuitively that discrete word dropout allows sentences with high word overlap (or low levenshtein distance) to have a higher chance of being perturbed into one another, thereby mapping them close in the latent space during the training time. however, this method allows for stylistically dissimilar sentences, albeit with high word overlap, to be mapped together in the latent space. idea. we argue that this negatively impacts the quality of final latent geometry and the results of the subsequent style transfer task. as an alternative, we explore embedding perturbation, where a noise vector is sampled, appropriately scaled and added to the embeddings of each input sentence, such that the resultant embeddings are constrained to live inside an e-dimensional hypersphere. the radius of this hypersphere is controllable using a tune-able hyper-parameter ζ. this hypersphere constraint is partially inspired by concepts in adversarial robustness and ensures that each resultant noised word embedding is not altered to the extent of causing the underlying semantics of the sentence to change. we argue that this allows sentences with stylistically similar constituent embeddings to be mapped together, also encouraging the formation of stylepreserving latent neighbourhoods (more on this in section 4). the resulting latent representation from "embedding-perturbed" autoencoders consequently are better semantically organized and stylistically robust, enabling us to perform tst using an inductive method i.e. simple vector arithmetic on latent vectors. contributions. we show that this extended model of input perturbation with both discrete and continuous components, allows for overall better quality text style transfer, particularly in its ability to preserve style-independent "content" information. to expand the traditional definitions of styles such as polarity, formality, which are based on simple "intra-sentence" attributes, we introduce the "discourse style transfer task" by salvaging nli datasets in which the flow of logic between sentences are captured using "entailment", "contradiction", and "neutral" labels. this enables interesting applications such as discourse manipulation in which a pair of sentences agreeing with each other can be made to contradict, and vice versa. we also test our model on fine-grained styles present in the style-ptb dataset. we empirically show that our model performs the best on a diverse set of datasets with styles ranging from coarse-grained styles (like sentiment) to fine-grained styles (like tenses) and complex inter-sentence styles (like discourse or flow of logic). ## method the underlying language model is a generative auto-encoder which models an input distribution p(x) assumed to be from an underlying latent distribution p(z). a deterministic encoder e representing the distribution q(z|x), in the form of an rnn, whose output is reparameterized by another distribution z i ∼ n (µ i (x), σ i (x)) to give the aggregated posterior distribution q(z). various auxiliary loss functions are used to enforce the learned prior q(z) to match p(z). the generator g representing p(x|z), also in the form of an rnn, decodes back the sample from the learnt prior q(z) into its corresponding input from p(x). gradient descent is applied on the reconstruction loss of the autoencoder given by: we use the aae @xcite as our choice for the underlying generative autoencoder over which the input perturbation techniques were applied. aae uses a discriminator d to enforce q(z) to match p(z), a standard gaussian, by learning to distinguish between samples from the two different distributions. this adversarial loss serves as a regularizer for the latent space, giving it the ability the organize itself better for smooth sentence interpolation. the final min-max objective is a sum of the reconstruction loss (given below) and λ weighted adversarial loss: we found empirically that aaes performed well and were stable during training. on the other hand, β-vaes required careful tuning of the β hyperparameter to prevent posterior collapse and did not perform as well. to further organize the latent geometry of the underlying aae to encode style-based semantic similarity among sentences, we propose a perturbationbased approach on the continuous embedding space. word embeddings of dimensionality e, of each input token x i are denoted as e i . consider an input sentence of length l containing tokens x 0 , • • • , x l-1 having embeddings e 0 , • • • , e l-1 respectively. our objective is to blur every embedding vector e i by adding some appropriately scaled noise vector n i to produce a resultant noised embedding vector e ′ i , such that e ′ i does not lie too far away from e i to not change the underlying semantics of the word completely. we do this by ensuring that each new e ′ i lives inside an e-dimensional hypersphere. the center of the hypersphere is the original embedding e i and its radius is defined as |e i | * s i , where s @xmath0. p (s) is probability distribution of the form y(µ = 0, σ) where y is some arbitrarily chosen distribution and σ as function of hyperparameter ζ ∈ r. this distribution p (s) parameterized by y, ζ) models the probability density cloud inside the embedding hypersphere, controlling its radius and interior densities. figure 1 neatly summarizes the aforementioned embedding perturbation mechanism for a simple example with only two individual word embeddings. in practice, we use a vectorized representation of the above mechanism to blur the embeddings of a minibatch of sentences of size l in constant o(1) time. the embedding perturbation method is summarized below in a vectorized form: ## semantic similarity in latent space neighbourhoods we contrast and compare the resultant latent space neighbourhoods of daaes and epaaes. ## conclusion we introduce the "embedding perturbed aae" or epaae and show that it best captures underlying style-based semantic features in the latent space in an unsupervised manner compared to its baselines. by inducing robust latent space organization through embedding perturbation in an unsupervised manner, we also demonstrate the possibility of finegrained tst, where we can control the strength of the target style. using a diverse set of datasets with varying formulations and complexities of style, we empirically that epaae performs overall best in the text style transfer task, particularly in its ability to preserve style-independent content across all datasets.