Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projection (2023)

Abstract: Natural language processing models tend to learn and encode social biases present in the data. One popular approach for addressing such biases is to eliminate encoded information from the model’s representations. However, current methods are restricted to removing only linearly encoded information. In this work, we propose Iterative Gradient-Based Projection (IGBP), a novel method for removing non-linear encoded concepts from neural representations. Our method consists of iteratively training neural classifiers to predict a particular attribute we seek to eliminate, followed by a projection of the representation on a hypersurface, such that the classifiers become oblivious to the target attribute. We evaluate the effectiveness of our method on the task of removing gender and race information as sensitive attributes. Our results demonstrate that IGBP is effective in mitigating bias through intrinsic and extrinsic evaluations, with minimal impact on downstream task accuracy.

## introduction the increasing reliance on natural language processing models in decision-making systems has led to a renewed focus on the potential biases that these models may encode. recent studies have demonstrated that word embeddings exhibit gender bias in their associations of professions @xcite and that learned representations of language models capture demographic data about the writer of the text, such as race or age @xcite . model decisions can be affected by these encoded biases and irrelevant attributes, leading to a wide range of inequities toward certain demographics. for example, a model designed to review job resumes should not factor in the applicants' gender or race. consequently, it is desirable to be able to manipulate the type of data encoded within text representations and to exclude any sensitive information in order to create more fair and equitable models. removing the presence of sensitive attributes from the representations learned by deep neural networks is non-trivial, as these representations are often learned using complex and hard-to-interpret non-linear models. re-training the language model can be a costly solution, therefore post-hoc removal methods that work at the representation layer have been proposed, such as linear projection of the embeddings on a hyperplane that distinguishes between the sensitive attribute @xcite . however, neural networks do not necessarily represent concepts in a linear manner. to address this issue, @xcite proposed kernelization of a linear minimax game for concept erasure, but this approach is restricted to the selection of kernel and the attribute protection does not transfer to different types of non-linear probes. accordingly, @xcite considered non-linear concept erasure to be an open problem. in this paper, we propose a non-linear concept erasure method, igbp, to eliminate information about the protected attribute from neural representations. we use a trained probe classifier that attempts to predict the protected attribute and a novel loss function suited for the task of concept removal. then, we leverage the gradients of this loss to guide for projection of the representations to a hypersurface that does not contain information used by the classifier regarding the sensitive-attribute. this is done by projecting the representations to the separating boundary of the classifier. figure 1 illustrates a 2-dimensional example. our approach supports the use of non-linear neural classifiers. when used with a linear classifier, it is equivalent to iterative null space projection (inlp), a popular linear concept removal method @xcite . we perform an empirical evaluation of the proposed method using: (1) intrinsic evaluation of word embeddings measuring word-level gender bias removal and (2) extrinsic fair classification evaluation over tasks that uses contextualized word representations. the empirical results show that the proposed method is successful in sensitiveattribute removal and mitigating bias, outperforming competing algorithms with minimal impact on the downstream task accuracy. ## related work many studies (e.g., @xcite investigated social biases in word embeddings and text representations. recent work have showed how applications that use pre-trained representations reflect and amplify these kinds of social biases @xcite . the approaches tackling this problem can be categorized into three lines of work: pre-processing methods which manipulate the input distribution before training (e.g., @xcite , in-processing methods which focus on learning fair models during training (e.g., @xcite @xcite and post-hoc methods (e.g., @xcite @xcite , which assume a fixed, pre-trained set of representations from any encoder and aim to learn a new set of unbiased representations. since re-training a model can be costly, a lot of focus was given to post-hoc methods, which is the main focus of this work. the most common post-hoc approach to remove sensitive information from word embeddings is to use a linear projection. @xcite identified a gender subspace, which is a subspace spanned by the directions of embeddings that capture the bias, such as the direction "he" -"she". they suggested projecting all the gender-neutral word embeddings on the gender subspace's first principle component to make neutral words equally distant from male and female-gendered words. however, @xcite showed that this method only covers up bias and not fully removes it from the representation. another critical drawback of the method is that it requires user selection of a few gender directions. @xcite tried to overcome this drawback of manually defining gender direction, and presented the iterative null-space projection (inlp) method. it is based on training linear classifiers that predict the attribute they wish to remove, then projecting the representations on the classifiers' null-space. @xcite aims to linearly remove information from neural representations by using a linear minimax gamebased approach, and derive a closed-form solution for certain objectives. one of the limitations of linear removal methods is their inability to remove non-linear information about the protected attribute, which is often encoded in text representations through complex neural networks. in contrast, our method is capable of removing both linear and non-linear information, resulting in a more effective reduction of extrinsic bias (section 4.4). @xcite proposed a nonlinear extension of the concept-removal objective of @xcite . they identify the subspace to be neutralized in kernel space by running a kernelized version of a minimax game as in @xcite . @xcite also use kernels to try and remove non-linear information while this approach aims to remove non-linear information, it can only choose data mapping from a pre-defined set of kernels, and as shown in @xcite , the attribute protection does not transfer to other non-linear kernels . our approach uses a deep neural network as the bias signal modeling, thus has the potential to express any non-linear function. our empirical results (section 4) show that our method significantly outperforms these methods on a variety of tasks. ## experiments in this section we compare competing methods for bias removal with the igbp algorithm in both intrinsic (section 4.3) and extrinsic evaluations (section 4.4), which are common in the literature on bias removal. ## analysis we conduct a series of analyses of our proposed method: an examination of probe's complexity impact on debiasing and an analysis of the effect of number of iterations on performance. ## conclusion we presented a gradient-based method for the erasure of non-linearly encoded concepts in text representations. its ability to remove non-linear information makes it particularly useful for addressing the complex biases that may be present in text representations learned through complex models. we empirically show the effectiveness of our approach to mitigate social biases in representations, thereby improving fairness in models' decision-making. beyond mitigating bias, the iterative gradient-based projection method has the potential to be applied in a wide range of other contexts, such as increasing model interpretability by applying causal interventions, adapting models to new domains by removing domain-specific information and ensuring privacy by removing sensitive information. in future work, we plan to explore these and other potential applications of the proposed method.