Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets (2021)

Abstract: To highlight the challenges of achieving representation disentanglement for text domain in an unsupervised setting, in this paper we select a representative set of successfully applied models from the image domain. We evaluate these models on 6 disentanglement metrics, as well as on downstream classification tasks and homotopy. To facilitate the evaluation, we propose two synthetic datasets with known generative factors. Our experiments highlight the existing gap in the text domain and illustrate that certain elements such as representation sparsity (as an inductive bias), or representation coupling with the decoder could impact disentanglement. To the best of our knowledge, our work is the first attempt on the intersection of unsupervised representation disentanglement and text, and provides the experimental framework and datasets for examining future developments in this direction.

## introduction learning task-agnostic unsupervised representations of data has been the center of attention across various areas of machine learning and more specifically nlp. however, little is known about the way these continuous representations organise information about data. in recent years, the nlp community has focused on the question of design and selection of suitable linguistic tasks to probe the presence of syntactic or semantic phenomena in representations as a whole @xcite @xcite @xcite @xcite . nonetheless, a finegrain understanding of information organisation in coordinates of a continuous representation is yet to be achieved. arguably, a necessity to move in this direction is agreeing on the cognitive process behind language generation (fusing semantic, syntactic, and lexical components), which can then be reflected in the design of representation learning frameworks. however, this still remains generally as an area of debate and perhaps less pertinent in the era of self-supervised masked language models and the resulting surge of new state-of-the-art results. even in the presence of such an agreement, learning to disentangle the surface realization of the underlying factors of data (e.g., semantics, syntactic, lexical) in the representation space is a nontrivial task. additionally, there is no established study for evaluating such models in nlp. a handful of recent works have looked into disentanglement for text by splitting the representation space into predefined disentangled subspaces such as style and content @xcite , or syntax and semantics @xcite @xcite , and rely on supervision during training. however, a generalizable and realistic approach needs to be unsupervised and capable of identifying the underlying factors solely via the regularities presented in data. in areas such as image processing, the same question has been receiving a lot of attention and inspired a wave of methods for learning and evaluating unsupervised representation disentanglement @xcite @xcite @xcite and creation of large scale datasets @xcite . it has been argued that disentanglement is the means towards representation interpretability @xcite , generalization @xcite , and robustness @xcite . however, these benefits are yet to be realized and evaluated in text domain. in this work we take a representative set of unsupervised disentanglement learning frameworks widely used in image domain ( §2.1) and apply them to two artificially created corpora with known underlying generative factors ( §3). having known generative factors (while being ignored during the training phase) allows us to evaluate the performance of these models on imposing representation disentanglement via 6 disentanglement metrics ( §2.2; §4.1). additionally, taking the highest scoring models and corresponding representations, we investigate the impact of representation disentanglement on two downstream text classification tasks ( §4.3), and dimension-wise homotopy ( §4.4). we show that existing disentanglement models, when evaluated on a wide range of metrics, are inconsistent and highly sensitive to model initialisation. however, where disentanglement is achieved, it shows its positive impact on improving downstream task performance. our work highlights the potential and existing challenges of disentanglement on text. we hope our proposed datasets, accessible description of disentanglement metrics and models, and experimental framework will set the path for developments of models specific to for text. ## disentanglement models and metrics let x denote data points and z denote latent variables in the latent representation space, and assume data points are generated by the combination of two random process: the first random process samples a point z (i) from the latent space with prior distribution of z, denoted by p(z). the second random process generates a point x (i) from the data space, denoted by p(x|z (i) ). we consider z as a disentangled representation for x, if the changes in single latent dimensions of z are sensitive to changes in single generative factors of x while being relatively invariant to changes in other factors @xcite . several probabilistic models are designed to reveal this process, here we look at some of the most widely used ones. ## generative synthetic datasets the use of synthetic datasets is the common practice for evaluating disentanglement in image domain @xcite @xcite . generative simplistic datasets in image domain define independent generative factors (e.g. shape, color) behind the data generation. however, a comparable resource is missing in text domain. we develop two synthetic generative datasets with varying degrees of difficulty to analyse and measure disentanglement: the ynoc dataset ( §3.1) which has only three structures and generative factors appearing in every sentence, and the pos dataset ( §3.2) which has more structures while some generative factors are not guaranteed simple sentence structures # of sentences n. v. n. end-punc. 200 n. v. adj. n. end-punc. ## experiments and analysis in this section, we examine the introduced disentanglement models on text. we measure the disentanglement scores of each model on our two synthetic datasets and quantify how well-correlated these metrics are with reconstruction loss, active units, and kl ( §4.1). we then look at various strategies for coupling the latent code during decoding and highlight their impacts on training and disentanglement behaviors ( §4.2). we continue our analysis by showing how the representation learned by the highest scoring model (on disentanglement metrics) performs compared to vanilla vae in two text classification tasks ( §4.3), and finish our analysis by looking at these models' generative behaviors ( §4.4). ## conclusion and future directions we evaluated a set of recent unsupervised disentanglement learning frameworks widely used in image domain on two artificially created corpora with known underlying generative factors. our experiments highlight the existing gaps in text domain, the daunting tasks state-of-the-art models from image domain face on text, and the confounding elements that pose further challenges towards representation disentanglement in text domain. motivated by our findings, in future, we will explore the role of inductive biases such as representation sparsity in achieving representation disentanglement. additionally, we will look into alternative forms of decoding and training which may compromise reconstruction quality but increase the reliance of decoding on the representation, hence allowing for a more controlled analysis and evaluation. our synthetic datasets and experimental framework provide a set of quantitative and qualitative measures to facilitate and future research in developing new models, datasets, and evaluation metrics specific for text. space r ij , who has a bijection mapping with s ij . hence, when sampling representations which have the same value on one generative factor, we only need to sample in one r ij . under these notations, we write the pseudo code of metrics in algorithm 1-6. for algorithm 5 and 6, although we only use one criterion in the main paper, we still provide the details for other criteria. we set @xmath0, and @xmath1, 4, 5, and 6. algorithm 1 metric of @xcite 1: @xmath2: for f @xmath3: tr and test set te with proportion (80%, 20%) 11: train 10 mlps with only input and output layer on tr 12: calculate the accuracy on te for 10 models 13: calculate the mean and variance of accuracy algorithm 2 metric of kim and mnih (2018) 1: @xmath4: for @xmath5, 2, . . . , dim z do 3: calculate the standard deviation σ d of dimension d 4: for f @xmath6: tr and test set te with proportion (80%, 20%) 12: train 10 majority vote classifiers on tr 13: calculate the accuracy on te for 10 models 14: calculate the mean and variance of accuracy algorithm 3 metric of kumar et al. (2018) 1: for f @xmath7: for @xmath8, 2, . . . , n j do 7: algorithm 6 metric of @xcite 1: for f @xmath9: