Log-linear Guardedness and its Implications (2023)

Abstract: Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful. However, the impact of this removal on the behavior of downstream classifiers trained on the modified representations is not fully understood. In this work, we formally define the notion of linear guardedness as the inability of an adversary to predict the concept directly from the representation, and study its implications. We show that, in the binary case, under certain assumptions, a downstream log-linear model cannot recover the erased concept. However, we constructively demonstrate that a multiclass log-linear model can be constructed that indirectly recovers the concept in some cases, pointing to the inherent limitations of linear guardedness as a downstream bias mitigation technique.These findings shed light on the theoretical limitations of linear erasure methods and highlight the need for further research on the connections between intrinsic and extrinsic bias in neural models.

## introduction neural models of text have been shown to represent human-interpretable concepts, e.g., those related to the linguistic notion of morphology @xcite , syntax @xcite , semantics @xcite , as well as extra-linguistic notions, e.g., gender distinctions @xcite . identifying and erasing such concepts from neural representations is known as concept erasure. linear concept erasure in particular has gained popularity due to its potential for obtaining formal guarantees and its empirical effectiveness @xcite @xcite @xcite @xcite . a common instantiation of concept erasure is removing a concept (e.g., gender) from a representation (e.g., the last hidden representation of a transformer-based language model) such that it cannot be predicted by a log-linear model. then, one fits a secondary log-linear model for a downstream task over the erased representations. for example, one may fit a log-linear sentiment analyzer to predict sentiment from gender-erased representations. the hope behind such a pipeline is that, because the concept of gender was erased from the representations, the predictions made by the log-linear sentiment analyzer are oblivious to gender. previous work @xcite @xcite has implicitly or explicitly relied on this assumption that erasing concepts from representations would also result in a downstream classifier that was oblivious to the target concept. in this paper, we formally analyze the effect concept erasure has on a downstream classifier. we start by formalizing concept erasure using @xcite v-information. 1 we then spell out the related notion of guardedness as the inability to predict a given concept from concept-erased representations using a specific family of classifiers. formally, if v is the family of distributions realizable by a log-linear model, then we say that the representations are guarded against gender with respect to v. the theoretical treatment in our paper specifically focuses on log-linear guardedness, which we take to mean the inability of a log-linear model to recover the erased concept from the representations. we are able to prove that when the downstream classifier is binary valued, such as a binary sentiment classifier, its prediction indeed cannot leak information about the erased concept ( § 3.2) under certain assumptions. on the contrary, in the case of multiclass classification with a log-linear model, we show that predictions can potentially leak a substantial amount of information about the erased concept, thereby recovering the guarded information completely. the theoretical analysis is supported by experiments on commonly used linear erasure techniques ( § 5). while previous authors @xcite , inter alia) have empirically studied concept erasure's effect on downstream classifiers, to the best of our knowledge, we are the first to study it theoretically. taken together, these findings suggest that log-linear guardedness may have limitations when it comes to preventing information leakage about concepts and should be assessed with extreme care, even when the downstream classifier is merely a log-linear model. ## information-theoretic guardedness in this section, we present an information-theoretic approach to guardedness, which we couch in terms of v-information @xcite . ## theoretical analysis in the following sections, we study the implications of guardedness on subsequent linear classifiers. specifically, if we construct a third random variable @xmath0: r @xmath1, what is the degree to which y can reveal information about z? as a practical instance of this problem, suppose we impose ε-guardedness on the last hidden representations of a transformer model, i.e., x in our formulation, and then fit a linear classifier t over the guarded representations h(x) to predict sentiment. can the predictions of the sentiment classifier indirectly leak information on gender? for expressive v, the data-processing inequality @xcite applied to the markov chain @xmath2. the reason is that, in this case, v-information is equivalent to mutual information and the data processing inequality tells us such leakage is not possible. however, the data processing inequality does not generally apply to v-information @xcite . thus, it is possible to find such a predictor t for less expressive v. surprisingly, when |y| = 2, we are able to prove that constructing such a t that leaks information is impossible under a certain restriction on the family of log-linear models. ## experimental evaluation in the empirical portion of our paper, we evaluate the extent to which our theory holds in practice. ## related work techniques for information removal are generally divided into adversarial methods and post-hoc linear methods. adversarial methods @xcite @xcite @xcite use a gradient-reversal layer during training to induce representations that do not encode the protected attribute. however, @xcite have shown that these methods fail to exhaustively remove all the information associated with the protected attribute. linear methods have been proposed as a tractable alternative, where one identifies a linear subspace that captures the concept of interest, and neutralizes it using algebraic techniques. different methods have been proposed for the identification of the subspace, e.g., pca and variants thereof @xcite , orthogonal rotation @xcite , classification-based @xcite , spectral @xcite and adversarial approaches @xcite . different definitions have been proposed for fairness @xcite , but they are mostly extrinsic-they concern themselves only with the predictions of the model, and not with its representation space. intrinsic bias measures, which focus on the representation space of the model, have been also extensively studied. these measures quantify, for instance, the extent to which the word representation space encodes gender distinctions @xcite @xcite . the relation between extrinsic and intrinsic bias measures is understudied, but recent works have demonstrated empirically either a relatively weak or inconsistent correlation between the two @xcite @xcite @xcite @xcite . ## conclusion we have formulated the notion of guardedness as the inability to directly predict a concept from the representation. we show that log-linear guardedness with respect to a binary protected attribute does not prevent a subsequent multiclass linear classifier trained over the guarded representations from leaking information on the protected attribute. in contrast, when the main task is binary, we can bound that leakage. altogether, our analysis suggests that the deployment of linear erasure methods should carefully take into account the manner in which the modified representations are being used later on, e.g., in classification tasks.