Learning Disentangled Textual Representations via Statistical Measures of Similarity (2022)

Abstract: When working with textual data, a natural application of disentangled representations is the fair classification where the goal is to make predictions without being biased (or influenced) by sensible attributes that may be present in the data (e.g., age, gender or race). Dominant approaches to disentangle a sensitive attribute from textual representations rely on learning simultaneously a penalization term that involves either an adversary loss (e.g., a discriminator) or an information measure (e.g., mutual information). However, these methods require the training of a deep neural network with several parameter updates for each update of the representation model. As a matter of fact, the resulting nested optimization loop is both times consuming, adding complexity to the optimization dynamic, and requires a fine hyperparameter selection (e.g., learning rates, architecture). In this work, we introduce a family of regularizers for learning disentangled representations that do not require training. These regularizers are based on statistical measures of similarity between the conditional probability distributions with respect to the sensible attributes. Our novel regularizers do not require additional training, are faster and do not involve additional tuning while achieving better results both when combined with pretrained and randomly initialized text encoders.

## introduction as natural language processing (nlp) systems are taken up in an ever wider array of sectors (e.g., legal system @xcite , insurance @xcite , education @xcite , healthcare @xcite ), there are growing concerns about the harmful potential of bias in such systems @xcite . recently, a large body of research aims at analyzing, understanding and addressing bias in various applications of nlp including language modelling @xcite , machine translation @xcite , toxicity detection @xcite and classification @xcite . in nlp, current systems often rely on learning continuous embedding of the input text. thus, it is crucial to ensure that the learnt continuous representations do not exhibit bias that could cause representational harms @xcite , i.e., representations less favourable to specific social groups. one way to prevent the aforementioned phenomenon is to enforce disentangled representations, i.e., representations that are independent of a sensitive attribute (see fig. 1 for a visualization of different degrees of disentangled representations). learning disentangled representations has received a growing interest as it has been shown to be useful for a wide variety of tasks (e.g., style transfer @xcite , few shot learning @xcite , fair classification @xcite . for text, the dominant approaches to learn such representations can be divided into two classes. the first one, relies on an adversary that is trained to recover the discrete sensitive attribute from the latent representation of the input @xcite . however, as pointed out by @xcite , even though the adversary seems to do a perfect job during training, a fair amount of the sensitive information can be recovered from the latent representation when training a new adversary from scratch. the second line of research involves a regularizer that is a train-able surrogate of the mutual information (mi) (e.g., club @xcite , mireny @xcite , knife @xcite , mine @xcite ) and achieves higher degrees of disentanglement. however, as highlighted by recent works @xcite , these estimators are hard to use in practice and the optimization procedure (see app. d.4) involves several updates of the regularizer parameters at each update of the representation model. as a consequence, these procedures are both time consuming and involve extra hyperparameters (e.g., optimizer learning rates, architecture, number of updates of the nested loop) that need to be carefully selected which is often not such an easy task. contributions. in this work, we focus our attention on learning to disentangle textual representations from a discrete attribute. our method relies on a novel family of regularizers based on discrepancy measures. we evaluate both the disentanglement and representation quality on fair text classification. formally, our contribution is two-fold: ## related work considering a tuple (x, s) where x is a random variable (r.v.) defined on the space of text x and s is a binary r.v. which corresponds to a sensitive attribute. learning disentangled representations aims at learning the parameter θ of the encoder f θ : @xmath0, where @xmath1. the goal is that z retains as much useful information from x while being oblivious of s. among the numerous possible applications for disentangled representations, we choose to focus on fair classification as it is a natural task to define the aforementioned useful information. in the fair classification task, we assume access to y , a binary r.v., which corresponds to the main label/attribute. in order to learn disentangled representations for fair classification, we follow previous works @xcite ) and we will be minimizing the loss l(ϕ, ψ, θ), which is defined as follows: ## proposed method this section describes our approach to learn disentangled representations. we first introduce the main idea and provide an algorithm to implement the general loss. we next describe the four similarity measures proposed in this approach. ## experimental setting in this section, we describe the datasets, metrics, encoder and baseline choices. additional experimental details can be found in app. d. for fair comparison, all models were re-implemented. ## numerical results in this section, we gather experimental results for fair classification task. we study our framework when working either with rnn or bert encoders. the parameter λ (see ( 3 )) controls the trade-off between success on the main task and disentanglement for all models. ## further analysis here, we aim at understanding of this phenomena. ## summary and concluding remarks we have introduced a new framework for learning disentangled representations which is faster to train, easier to tune and achieves better results than adversarial or mi-based methods. our experiments on the fair classification task show that for rnn encoders, our methods relying on the closed-form of similarity measures under a multivariate gaussian assumption can achieve perfectly disentangled representations with little cost on the main tasks (e.g. using wasserstein). on bert representations, our experiments show that the sinkhorn divergence should be preferred. it can achieves almost perfect disentanglement at little cost but allows for fewer control over the degree of disentanglement.