Adversarial Concept Erasure in Kernel Space (2022)

Abstract: The representation space of neural models for textual data emerges in an unsupervised manner during training. Understanding how human-interpretable concepts, such as gender, are encoded in these representations would improve the ability of users to control the content of these representations and analyze the working of the models that rely on them. One prominent approach to the control problem is the identification and removal of linear concept subspaces – subspaces in the representation space that correspond to a given concept. While those are tractable and interpretable, neural network do not necessarily represent concepts in linear subspaces. We propose a kernelization of the recently-proposed linear concept-removal objective, and show that it is effective in guarding against the ability of certain nonlinear adversaries to recover the concept. Interestingly, our findings suggest that the division between linear and nonlinear models is overly simplistic: when considering the concept of binary gender and its neutralization, we do not find a single kernel space that exclusively contains all the concept-related information. It is therefore challenging to protect against all nonlinear adversaries at once.

## introduction large neural networks in nlp produce real-valued representations that encode the bit of human language that they were trained on, e.g., words, sentences, or text grounded in images. for instance, glove @xcite produces realvalued representations of isolated words, bert @xcite produces real-valued representations of sentences, and vilbert produces real-valued representations of visually grounded language @xcite . these real-valued representations naturally encode various properties of the objects they represent. for instance, a good representation of the first author's laptop computer ought to encode its manufacturer, size, and color somewhere among its real values. we now describe the premise of our paper in more detail. we adopt the notion of a concept due to @xcite . for gärdenfors, objects can be thought of as having a geometric representation. different dimensions in the representation space that objects inhabit might correspond to their color, size, and shape. gärdenfors then goes further and defines a concept as a convex region of the representation space. 1 building on gärdenfors' notion of a concept, this paper studies a task that we refer to as concept erasure. we now motivate the task more formally by extending the example. imagine we have n different laptops, whose real-valued representations are denoted as x 1 , . . . , x n . now, consider concept labels y 1 , . . . , y n that encode each laptop's color and are taken from the set {grey, silver, black, white}. in the concept erasure paradigm, we seek an erasure function r(•) such that r(x 1 ), . . . , r(x n ) are no longer predictive of the colors of the laptops, y 1 , . . . , y n , but retain all the other information encoded in the original representations, i.e., they remain predictive with respect to other laptop-related concepts. our hope is that the geometry of the erasure function r(•) then tells us the structure of the laptop color concept. concept erasure is tightly related to concept identification. once we have successfully removed a given concept, e.g., color, from a representation x n , it is reasonable to argue that the erasure function r has meaningfully identified the concept within the representation space. in the rest of the paper, we will say that r(•) neutralizes the concept in the representation space. for instance, we say that the r in our example neutralizes the concept of a laptop's color. it follows that concept identification is related to bias mitigation @xcite @xcite , e.g., one may want to identify and remove the gender encoded in learned word representations produced by an embedding method such as word2vec @xcite or glove @xcite . indeed, the empirical portion of this paper will focus on removing gender from word representations in order to mitigate bias. previous work on concept erasure @xcite focuses on the linear case, i.e., where r is a linear function. while linear concept erasure methods have certainly found success @xcite , there is no a-priori reason to suspect that neural networks encode concepts in a linear manner. in this work, we take the first step toward the goal of identifying a non-linear function r(•) and a corresponding non-linearly encoded concept subspace. we directly build on @xcite , who cast linear concept erasure as a minimax game. under their formulation, the function r(•) learns to remove the concept, while an adversary tries to predict the concept. we extend their work by deriving a class of general minimax games based on kernelization that largely maintains the tractability of the linear approach. our kernelized method performs concept erasure in a reproducing kernel hilbert space, which may have a much higher dimensionality @xcite and correspond to a non-linear subspace of the original representation space. empirically, we experiment with gender erasure from glove and bert representations. we show that a kernelized adversary can classify the gender of the representations with over 99% accuracy if r(•) is taken to be a linear function. this gives us concrete evidence that gender is indeed encoded non-linearly in the representations. we further find that solving our kernelized minimax game yields an erasure function r(•) that protects against an adversary that shares the same kernel. however, we also find that it is difficult to protect against all kernelized adversaries at once: information removed by one kernel type can be recovered by adversaries using other kernel types. that is, the gender concept is not exclusively encoded in a space that corresponds to any one kernel. this suggests that non-linear concept erasure is very much an open problem. ## linear concept erasure we provide an overview of the linear minimax formulation before we introduce its kernelization. ## experimental setup in §3, we established an algorithm that allows us to attempt kernelized concept erasure of non-linearly encoded concepts. to summarize, this method requires first solving the game in eq. ( 10 ) for a chosen neutralizing kernel, then training a pre-image network according to eq. ( 11 ) to obtain neutralized representations in the input space. we hypothesize that a non-linearly encoded concept can be exhaustively removed after mapping into the right rkhs. in order for this to hold, the neutralized representations must satisfy two conditions: adversaries using non-linear classifiers should not be able to predict the erased concept from these representations, and these representations should preserve all other information encoded in them prior to erasure. with binary gender as our non-linearly encoded concept, we conduct several experiments testing both conditions. before presenting our results, we lay out our experimental setup (see app. a.3 for more details). data. we run our main experiments on the identification and erasure of binary gender in static glove representations @xcite . we focus on @xcite dataset, where word representations are coupled with binary labels indicating whether they are male-biased or femalebiased. as a preprocessing step, we normalize the glove representations to have unit norm. for an extrinsic evaluation of our method on a main task (profession prediction), we use the bias-in-bios dataset of @xcite , which consists of a large set of short biographies annotated for both gender and race. following @xcite , we embed each biography using the [cls] representation of pre-trained bert. kernels. we consider the following kernels: we experiment with different values for the hyperparameters γ > 0, α > 0 and @xmath0. a.3 for details). we use @xmath1. reported metrics. each result is reported as the @xmath2, computed across four runs of the experiment with random restarts. solving the minimax game. we solve the relaxed adversarial game given in eq. ( 10 ) by alternate gradient-based optimization @xcite . concretely, we alternate between updating the predictor's parameters θ and the projection matrix p . updates to θ are performed with gradient descent, and updates to p are performed with gradient ascent, including a projection onto the fantope to ensure that the constraint is met. for the fantope projection step, we use vu et al.'s (2013) algorithm, the details of which are restated in @xcite . ## effect on concept encoding in this section, we pose the exhaustive rkhs hypothesis: the hypothesis that binary gender can be exhaustively removed when the representations are mapped into the right rkhs. that is, there exists a unique kernel such that, for any choice of non-linear predictor, the adversary cannot recover gender information from the pre-image representations obtained via our special kernel. as a baseline, we note that gender prediction accuracy on the original representations, prior to any intervention, is above 99% with every kernel, including the linear kernel. this means that the gender concept is linearly separable in the original input space. in this context, we conduct the following experiments on gender neutralization. same adversary. we start by calculating the neutralized pre-images for each kernel type, and then apply the same kernel adversary to recover gender information. this experiment tests whether we can protect against the same kernel adversary. ## negative impact on the representations our method has shown a satisfactory ability to prevent the same kernel from recovering the concept. however, does erasure remove too much information? 11 as previously stated, our intervention should erase a concept without altering any of the other information encoded in the original representations. in this section, we evaluate whether the non-gender related semantic content of the original representations is preserved in our neutralized pre-images. we do so via the following tests: i) an intrinsic evaluation of the semantic content of the neutralized pre-image word representation space, and ii) an extrinsic evaluation of our method when applied to contextualized word representations for a profession prediction task, measuring the extent to which we hinder a model's ability to perform the main task. ## discussion we have demonstrated that in the case where the neutralizing kernel and the adversarial kernel are the same, we are able to neutralize a non-linearly encoded concept reasonably well. we have also shown that our method neutralizes gender in a comprehensive manner, without damaging the representation. however, this neutralization does not transfer to different non-linear adversaries, which are still able to recover gender. while the lack of transfer to other non-linear predictors may seem surprising, one should keep in mind that changing the kernel type, or changing kernel hyperparameters, results in a different implicit feature mapping. even the features defined by a linear kernel are not a proper subset of the fea-tures defined by a polynomial kernel of degree 2. 12 as such, removing the features which make the concept of interest linearly separable in one rkhs does not necessarily prevent a classifier parameterized by another kernel or an mlp from predicting the concept. in the context of gender erasure, these results suggest that protection against a diverse set of non-linear adversaries remains an open problem. ## conclusion we propose a novel method for the identification and erasure of non-linearly encoded concepts in neural representations. we first map the representations to an rkhs, before identifying and neutralizing the concept in that space. we use our method to empirically assess the exhaustive rkhs hypothesis: we hypothesize that there exists a unique kernel that exhaustively identifies the concept of interest. we find that while we are able to protect against a kernel adversary of the same type, this protection does not transfer to different nonlinear classifiers, thereby contradicting to the rkhs hypothesis. exhaustive concept erasure and protection against a diverse set of non-linear adversaries remains an open problem.