Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020)

Abstract: The ability to control for the kinds of information encoded in neural representation has a variety of use cases, especially in light of the challenge of interpreting these models. We present Iterative Null-space Projection (INLP), a novel method for removing information from neural representations. Our method is based on repeated training of linear classifiers that predict a certain property we aim to remove, followed by projection of the representations on their null-space. By doing so, the classifiers become oblivious to that target property, making it hard to linearly separate the data according to it. While applicable for multiple uses, we evaluate our method on bias and fairness use-cases, and show that our method is able to mitigate bias in word embeddings, as well as to increase fairness in a setting of multi-class classification.

## introduction what is encoded in vector representations of textual data, and can we control it? word embeddings, pre-trained language models, and more generally deep learning methods emerge as very effective techniques for text classification. accordingly, they are increasingly being used for predictions in real-world situations. a large part of the success is due to the models' ability to perform representation learning, coming up with effective feature representations for the prediction task at hand. however, these learned representations, while effective, are also notoriously opaque: we do not know what is encoded in them. indeed, there is an emerging line of work on probing deep-learning derived representations for syntactic @xcite @xcite , semantic @xcite and factual knowledge @xcite . there is also evidence that they capture a lot of information regarding the demographics of the author of the text @xcite . what can we do in situations where we do not want our representations to encode certain kinds of information? for example, we may want a word representation that does not take tense into account, or that does not encode part-of-speech distinctions. we may want a classifier that judges the formality of the text, but which is also oblivious to the topic the text was taken from. finally, and also our empirical focus in this work, this situation often arises when considering fairness and bias of languagebased classification. we may not want our wordembeddings to encode gender stereotypes, and we do not want sensitive decisions on hiring or loan approvals to condition on the race, gender or age of the applicant. we present a novel method for selectively removing specific kinds of information from a representation. previous methods are either based on projection on a pre-specified, user-provided direction @xcite , or on adding an adversarial objective to an end-to-end training process @xcite . both of these have benefits and limitations, as we discuss in the related work section ( §2). our proposed method, iterative nullspace projection (inlp), presented in section 4, can be seen as a combination of these approaches, capitalizing on the benefits of both. like the projection methods, it is also based on the mathematical notion of linear projection, a commonly used de-terministic operator. like the adversarial methods, it is data-driven in the directions it removes: we do not presuppose specific directions in the latent space that correspond to the protected attribute, but rather learn those directions, and remove them. empirically, we find it to work well. we evaluate the method on the challenging task of removing gender signals from word embeddings @xcite . recently, @xcite showed several limitations of current methods for this task. we show that our method is effective in reducing many, but not all, of these ( §4). we also consider the context of fair classification, where we want to ensure that a classifier's decision is oblivious to a protected attribute such as race, gender or age. there, we need to integrate the projection-based method within a pre-trained classifier. we propose a method to do so in section §5, and demonstrate its effectiveness in a controlled setup ( §6.2) as well as in a real-world one ( §6.3). finally, while we propose a general purpose information-removal method, our main evaluation is in the realm of bias and fairness applications. we stress that this calls for some stricter scrutiny, as the effects of blindly trusting strong claims can have severe real-world consequences on individuals. we discuss the limitations of our model in the context of such applications in section §7. ## related work the objective of controlled removal of specific types of information from neural representation is tightly related to the task of disentanglement of the representations @xcite , that is, controlling and separating the different kinds of information encoded in them. in the context of transfer learning, previous methods have pursued representations which are invariant to some properties of the input, such as genre or topic, in order to ease domain transfer @xcite . those methods mostly rely on adding an adversarial component @xcite @xcite to the main task objective: the representation is regularized by an adversary network, that competes against the encoder, trying to extract the protected information from its representation. while adverserial methods showed impressive performance in various machine learning tasks, and were applied for the goal of removal of sensitive information @xcite @xcite , they are notoriously hard to train. @xcite have evaluated adverserial methods for the removal of demographic information from representations. they showed that the complete removal of the protected information is nontrivial: even when the attribute seems protected, different classifiers of the same architecture can often still succeed in extracting it. another drawback of these methods is their reliance on a main-task loss in addition to the adverserial loss, making them less suitable for tasks such as debiasing pretrained word embeddings. @xcite utilized a "nullspace cleaning" operator for increasing privacy in classifiers. they remove from the input a subspace that contains (but is not limited to) the nullspace of a pre-trained classifier, in order to clean information that is not used for the main task (and might be protected), while minimally impairing classification accuracy. while similar in spirit to our method, several key differences exist. as the complementary settingremoving the nullsapce of the main-task classifier vs. projection onto the nullspace of protected attribute classifiers -aims to achieve a distinct goal (privacy preserving), there is no notion of exhaustive cleaning. furthermore, they do not remove protected attributes that are used by the classifier (e.g. when it conditions on gender). a recent line of work focused on projecting the representation to a subspace which does not encode the protected attributes. under this method, one identifies directions in the latent space that correspond to the protected attribute, and removes them. in a seminal work, @xcite aimed to identify a "gender subspace" in word-embedding space by calculating the main directions in a subspace spanned by the differences between gendered word pairs, such as # » he -# » she. they suggested to zero out the components of neutral words in the direction of the "gender subspace" first principle components, and actively pushed neutral words to be equally distant from male and female-gendered words. however, @xcite have shown that these methods only cover up the bias and that in fact, the information is deeply ingrained in the representations. a key drawback of this approach is that it relies on an intuitive selection of a few (or a single) gender directions, while, as we reveal in our experiments, the gender subspace is actually spanned by dozens to hundreds of orthogonal directions in the latent space, which are not necessarily as interpretable as the # » he -# » she direction. this observation aligns with the analysis of @xcite who demonstrated that debiasing by projection is theoretically effective provided that one removes all relevant directions in the latent space. ## objective and definitions our main goal is to "guard" sensitive information, so that it will not be encoded in a representation. given a set of vectors x @xmath0, and corresponding discrete attributes z, z @xmath1, ..., k} (e.g. race or gender), we aim to learn a transformation g : r @xmath2, such that z i cannot be predicted from g(x i ). in this work we are concerned with "linear guarding": we seek a guard g such that no linear classifier w(•) can predict z i from g(x i ) with an accuracy greater than that of a decision rule that considers only the proportion of labels in z. we also wish for g(x i ) to stay informative: when the vectors x are used for some end task, we want g(x) to have as minimal influence as possible on the end task performance, provided that z remains guarded. we use the following definitions: guarded w.r.t. a hypothesis class let @xmath3, ..., x @xmath4, with corresponding discrete attributes z, z @xmath5, ..., k}. we say the set x is guarded for z with respect to hypothesis class h (conversely z is guarded in x) if there is no classifier @xmath6. guarding function a function g : r @xmath7.r. ## iterative nullspace projection given a set of vectors x @xmath8, we seek a linear guarding function g that remove the linear dependence between z and x . we begin with a high-level description of our approach. let c be a trained linear classifier, parameterized by a matrix @xmath9, that predicts a property z with some accuracy. we can construct a projection matrix p such that w (p x) = 0 for all x, rendering w useless on dataset x . we then iteratively train additional classifiers w and perform the same procedure, until no more linear information regarding z remains in x. constructing p is achieved via nullspace projection, as described below. this method is the core of the inlp algorithm (algorithm 1). ## application to fair classification the previous section described the inlp method for producing a linearly guarding function g for a set of vectors. we now turn to describe its usage in the context of providing fair classification by a (possibly deep) neural network classifier. in this setup, we are given, in addition to x and z also labels y , and wish to construct a classifier f : @xmath10, while being fair with respect to z. fairness in classification can be defined in many ways @xcite @xcite . we focus on a notion of fairness by which the predictor f is oblivious to z when making predictions about y . to use linear guardedness in the context of a deep network, recall that a classification network f (x) can be decomposed into an encoder enc followed by a linear layer w : f (x) = w • enc(x), where w is the last layer of the network and enc is the rest of the network. if we can make sure that z is linearly guarded in the inputs to w , then w will have no knowledge of z when making its prediction about y , making the decision process oblivious to z. adversarial training methods attempt to achieve such obliviousness by adding an adversarial objective to make enc(x) itself guarding. we take a different approach and add a guarding function on top of an already trained enc. we propose the following procedure. given a training set x,y and protected attribute z, we first train a neural network @xmath11. this results in an encoder that extracts effective features from x for predicting y . we then consider the vectors enc(x), and use the inlp method to produce a linear guarding function g that guards z in enc(x). at this point, we can use the classifier w • g(enc(x)) to produce oblivious decisions, however by introducing g (which is lower rank than enc(x)) we may have harmed w s performance. we therefore freeze the network and fine-tune only w to predict y from g(enc(x)), producing the final fair classifier f (x) = w • g(enc(x)). notice that w only sees vectors which are linearly guarded for z during its training, and therefore cannot take z into ac-count when making its predictions, ensuring fair classification. we note that our notion of fairness by obliviousness does not, in the general case, correspond to other fairness metrics, such as equality of odds or of opportunity. it does, however, correlate with fairness metrics, as we demonstrate empirically. further refinement. guardedness is a property that holds in expectation over an entire dataset. for example, when considering a dataset of individuals from certain professions (as we do in §6.3), it is possible that the entire dataset is guarded for gender, yet if we consider only a subset of individuals (say, only nurses), we may still be able to recover gender with above majority accuracy, in that sub-population. as fairness metrics are often concerned with classification behavior also within groups, we propose the following refinement to the algorithm, which we use in the experiments in §6.2 and §6.3: in each iteration, we train a classifier to predict the protected attribute not on the entire training set, but only on the training examples belonging to a single (randomly chosen) main-task class (e.g. profession). by doing so, we push the protected attribute to be linearly guarded in the examples belonging to each of the main-task labels. in the first set of experiments, we evaluate the inlp method in its ability to debias word embeddings @xcite . after "debiasing" the embeddings, we repeat the set of diagnostic experiments of @xcite . ## limitations a limitation of our method when used in the context of fairness is that, like other learning approaches, it depends on the data x,z that is fed to it, and works under the assumption that the training data is sufficiently large and is sampled i.i.d from the same distribution as the test data. this condition is hard to achieve in practice, and failure to provide sufficiently representative training data may lead to biased classifications even after its application. like other methods, there are no magic guarantees, and the burden of verification remains on the user. it is also important to remember that the method is designed to achieve a very specific sense of protection: removal of linear information regarding a protected attribute. while it may correlate with fairness measures such as demographic parity, it is not designed to ensure them. finally, it is designed to be fed to a linear decoder, and the attributes are not protected under non-linear classifiers. ## conclusion we present a novel method for removing linearlyrepresented information from neural representations. we focus on bias and fairness as case studies, and demonstrate that across increasingly complex settings, our method is capable of attenuating societal biases that are expressed in representations learned from data. while we focused on bias, iterative nullspace projection has broader possible use-cases, and can be utilized to remove specific components from a representation, in a controlled manner. this method can be applicable for other end goals, such as style-transfer, disentanglement of neural representations and increasing their interpretability. we aim to explore those directions in a future work.