Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders (2021)

Abstract: The ability of learning disentangled representations represents a major step for interpretable NLP systems as it allows latent linguistic features to be controlled. Most approaches to disentanglement rely on continuous variables, both for images and text. We argue that despite being suitable for image datasets, continuous variables may not be ideal to model features of textual data, due to the fact that most generative factors in text are discrete. We propose a Variational Autoencoder based method which models language features as discrete variables and encourages independence between variables for learning disentangled representations. The proposed model outperforms continuous and discrete baselines on several qualitative and quantitative benchmarks for disentanglement as well as on a text style transfer downstream application.

## introduction a fundamental challenge in natural language processing (nlp) is being able to control generative factors for text, such as tense, gender, negation, which are characterised by an entangled representation in traditional neural networks, making it difficult to control them. disentangled representation learning aims to provide an interpretable representation of latent features, and a framework for controlling the change of specific features, by separating distinct generative factors in the data @xcite . various disentanglement approaches for neural networks have been proposed. @xcite achieve disentanglement with a generative adversarial network (gan), by maximizing the mutual information between latent variables and generated samples, while @xcite @xcite fine-tune the parameter which controls the kl divergence in a variational autoencoder (vae) @xcite . in the nlp domain, @xcite use adversarial losses to separate the style and content embeddings, while @xcite disentangle style and content embeddings by minimizing their mutual information. after a thorough review of the literature, we find that there are currently two main issues in the area of disentanglement in nlp. first, the mentioned approaches operate using gaussian distributions in a continuous space. although a continuous representation may be suitable to encode images, for text data one should rather specifically consider discrete distributions of the feature set. in fact, generative features of a sentence mostly belong to a discrete domain, for example, one would encode the gender feature either as male or female, while the tense can be modelled as having three values, such as present, past, future. secondly, while in the realm of representation learning, numerous quantitative and qualitative evaluation methods for disentanglement have been designed, there is still a gap in the adaptation and adoption of these methods into the language domain. as a result, despite being a critical step for enabling the interpretability of nlp models, text disentanglement is still not fully explored. we address the highlighted problems as follows: 1) we design the first vae-based architecture (shown in figure 1 ), where linguistic features are encoded as discrete latent variables via the gumbel-softmax trick @xcite and disentanglement is jointly enforced in the objective function. we derive a decomposition of the vae evidence lower bound (elbo), where independence between variables is encouraged by fine-tuning the total correlation term, with the goal of achieving disentanglement. at the same time, an information channel mechanism is introduced, and increased incrementally during training, to avoid the issue of the posterior collapse @xcite . the vae considers a multivariate gaussian prior distribution p(z) and generate a sentence x with the decoder p θ (x|z). the joint distribution for the decoder is defined as p(z)p θ (x|z), which, for a sequence of tokens x of length t result as p θ (x|z) = t @xmath0, z). the framework's objective is to maximize the expectation over the full dataset of the log-likelihood, in other words, e p(x) log p θ (x). however, this term is intractable, and thus the variational distribution q θ is introduced to approximate p θ (z|x). as a result, an evidence lower bound l vae (elbo) where e p(x) [log p θ (x)] ≥ l vae , is derived as follows: 1 ), i represents the reconstruction error, which encourages the model to encode the data x into the latent variables z, while ii represents the kl divergence regularization term, which forces the variational distribution q φ (z|x) to be similar to the prior p(z). ## proposed approach model overview. differently from previous approaches to disentanglement @xcite @xcite , we focus our efforts into leveraging the discrete generative factors present in natural language, and design a framework, which we name discrete controlled total correlation (dctc), where language factors are encoded as discrete latent variables, while the representation is enforced to be disentangled. we first reformulate the continuous vae for discrete variables by providing a suitable discrete reparameterization process. we then decompose the elbo, and provide a modification of the total correlation term for controlling disentanglement. ## experiments in this section, we evaluate the disentanglement of the proposed model with qualitative and quantitative methods, against several baselines. furthermore, the benefits of our model's encodings are demonstrated in the downstream task of text style transfer. ## conclusion in this work, we propose the first approach where a discrete encoding of the linguistic features in sentences is integrated with an objective function that encourages disentanglement. we provide a vae-based architecture where latent variables are back-propagated with a discrete reparameterization mechanism. we then design a decomposition of the elbo, where 1) the independence between latent variables is encouraged, to aid disentanglement, and 2) the amount of encoded information is controlled, to avoid the posterior collapse. we provide a novel evaluation procedure where representations learned from text data are probed in terms of their disentanglement, using metrics from representation learning. with this evaluation tool, we demonstrate that the presented model consistently outperforms continuous and discrete baselines for disentanglement, on qualitative evaluation, quantitative metrics, and text style transfer. we conclude that the modeling of discrete variables, which is currently under-explored in disentanglement research, may represent a fundamental encoding tool for enhancing interpretability and control in nlp models.