A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations (2021)

Abstract: Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others. The existent dominant approaches in the context of text data either rely on training an adversary (discriminator) that aims at making attribute values difficult to be inferred from the latent code or rely on minimising variational bounds of the mutual information between latent code and the value attribute. However, the available methods suffer of the impossibility to provide a fine-grained control of the degree (or force) of disentanglement. In contrast to adversarial methods, which are remarkably simple, although the adversary seems to be performing perfectly well during the training phase, after it is completed a fair amount of information about the undesired attribute still remains. This paper introduces a novel variational upper bound to the mutual information between an attribute and the latent code of an encoder. Our bound aims at controlling the approximation error via the Renyi’s divergence, leading to both better disentangled representations and in particular, a precise control of the desirable degree of disentanglement than state-of-the-art methods proposed for textual data. Furthermore, it does not suffer from the degeneracy of other losses in multi-class scenarios. We show the superiority of this method on fair classification and on textual style transfer tasks. Additionally, we provide new insights illustrating various trade-offs in style transfer when attempting to learn disentangled representations and quality of the generated sentence.

## introduction learning disentangled representations hold a central place to build rich embeddings of highdimensional data. for a representation to be disentangled implies that it factorizes some latent cause or causes of variation as formulated by @xcite . for example, if there are two causes for the transformations in the data that do not generally happen together and are statistically distinguishable (e.g., factors occur independently), a maximally disentangled representation is expected to present a sparse structure that separates those causes. disentangled representations have been shown to be useful for a large variety of data, such as video @xcite , image @xcite ), text @xcite , audio @xcite , among others, and applied to many different tasks, e.g., robust and fair classification @xcite , visual reasoning @xcite , style transfer @xcite , conditional generation @xcite , few shot learning @xcite , among others. in this work, we focus our attention on learning disentangled representations for text, as it remains overlooked by @xcite . perhaps, one of the most popular applications of disentanglement in textual data is fair classification @xcite and sentence generation tasks such as style transfer @xcite or conditional sentence generation @xcite . for fair classification, perfectly disentangled latent representations can be used to ensure fairness as the decisions are taken based on representations which are statistically independent from-or at least carrying limited information about-the protected attributes. however, there exists a trade-offs between full disentangled representations and performances on the target task, as shown by @xcite , among others. for sequence generation and in particular, for style transfer, learning disentangled representations aim at allowing an easier transfer of the desired style. to the best of our knowledge, a depth study of the relationship between disentangled representa-tions based either on adversarial losses solely or on vclu b -s and quality of the generated sentences remains overlooked. most of the previous studies have been focusing on either trade-offs between metrics computed on the generated sentences @xcite or performance evaluation of the disentanglement as part of (or convoluted with) more complex modules. this enhances the need to provide a fair evaluation of disentanglement methods by isolating their individual contributions @xcite . methods to enforce disentangled representations can be grouped into two different categories. the first category relies on an adversarial term in the training objective that aims at ensuring that sensitive attribute values (e.g. race, sex, style) as statistically independent as possible from the encoded latent representation. interestingly enough, several works @xcite @xcite @xcite @xcite , @xcite have recently shown that even though the adversary teacher seems to be performing remarkably well during training, after the training phase, a fair amount of information about the sensitive attributes still remains, and can be extracted from the encoded representation. the second category aim at minimising mutual information (mi) between encoded latent representation and the sensitive attribute values, i.e., without resorting to an adversarial discriminator. mi acts as an universal measure of dependence since it captures non-linear and statistical dependencies of high orders between the involved quantities (kinney and atwal, 2014). however, estimating mi has been a long-standing challenge, in particular when dealing with high-dimensional data @xcite . recent methods rely on variational upper bounds. for instance, @xcite ) study vclub-s @xcite for sentence generation tasks. although this approach improves on previous state-of-the-art methods, it does not allow to fine-tuning of the desired degree of disentanglement, i.e., it enforces light or strong levels of disentanglement where only few features relevant to the input sentence remain (see @xcite for further discussion). ## main definitions and related works we introduce notations, tasks, and closely related work. consider a training set @xmath0, y i )} n @xmath1, . . . , |y|} which indicates a discrete attribute to be disentangled from the resulting representations. we study the following scenarios: disentangled representations. learning disentangled representations consists in learning a model m : @xmath2. in this framework, content is defined as any relevant information present in x that does not depend on y . applications to binary fair classification. the task of fair classification through disentangled representations aims at building representations that are independent of selective discrete (sensitive) attributes (e.g., gender or race). this task consists in learning a model m : @xmath3, 1} that maps any input x to a label @xmath4, 1}. the goal of the learner is to build a predictor that assigns each x to either 0 or 1 "oblivious" of the protected attribute y. recently, much progress has been made on devising appropriate means of fairness, e.g., @xcite @xcite . in particular, @xcite @xcite approach the problem based on adversarial losses. more precisely, these approaches consist in learning an encoder that maps x into a representation vector h x , a critic c θc which attempts to predict y, and an output classifier f θ d used to predict l based on the observed h x . the classifier is said to be fair if there is no statistical information about y that is present in h x @xcite . ## model and training objective this section describes the proposed approach to learn disentangled representations. we first review mi along with the model overview and then, we derive the variational bound we will use, and discuss connections with adversarial losses. ## numerical results in this section, we present our results on the fair classification and binary sequence generation tasks, see ssec. 5.1 and ssec. 5.2, respectively. we additionally show that our variational surrogates to the mi-contrarily to adversarial losses-do not suffer in multi-class scenarios (see ssec. 5.3). ## summary and concluding remarks we devised a new alternative method to adversarial losses capable of learning disentangled textual representation. our method does not require adversarial training and hence, it does not suffer in presence of multi-class setups. a key feature of this method is to account for the approximation error incurred when bounding the mutual information. experiments show better trade-offs than both adversarial training and vclub-s on two fair classification tasks and demonstrate the efficiency to learn disentangled representations for sequence generation. as a matter of fact, there is no free-lunch for sen-tence generation tasks: although transferring style is easier with disentangled representations, it also removes important information about the content. the proposed method can replace the adversary in any kind of algorithms @xcite with no modifications. future work includes testing with other type of labels such as dialog act @xcite , emotions @xcite , opinion (garcia et al., 2019) or speaker's stance and confidence @xcite . since it allows more finegrained control over the amount of disentanglement, we expect it to be easier to tune when combined with more complex models.