Probing Classifiers are Unreliable for Concept Removal and Detection
Abstract
Neural network models trained on text data have been found to encode undesirable linguistic or sensitive concepts in their representation.
Removing such concepts is non-trivial because of a complex relationship between the concept, text input, and the learnt representation.
Recent work has proposed post-hoc and adversarial methods to remove such unwanted concepts from a model’s representation.
Through an extensive theoretical and empirical analysis, we show that these methods can be counter-productive: they are unable to remove the concepts entirely, and in the worst case may end up destroying all task-relevant features.
The reason is the methods’ reliance on a probing classifier as a proxy for the concept.
Even under the most favorable conditions for learning a probing classifier when a concept’s relevant features in representation space alone can provide 100% accuracy, we prove that a probing classifier is likely to use non-concept features and thus post-hoc or adversarial methods will fail to remove the concept correctly.
These theoretical implications are confirmed by experiments on models trained on synthetic, Multi-NLI, and Twitter datasets.
For sensitive applications of concept removal such as fairness, we recommend caution against using these methods and propose a spuriousness metric to gauge the quality of the final classifier.
Introduction
Neural models in text classification have been shown to learn spuriously correlated features [17, 29] or embed sensitive attributes like gender or race [9, 8, 7] in their representation layer.
Classifiers that use such sensitive or spurious concepts (henceforth concepts ) raise concerns of model unfairness and out-of-distribution generalization failure [41, 3, 16].
Removing the influence of these concepts is non-trivial because the classifiers are based on hard-to-interpret deep neural networks.
Moreover, since many concepts cannot be modified at the input tokens level, removal methods that work at the representation layer have been proposed: 1) post-hoc removal [8, 50, 15] on a pre-trained model (e.g., null space projection [34]), and 2) adversarial removal [16, 49, 13] by jointly training the main task classifier with an (adversarial) classifier for the concept.
In this paper, we theoretically show that both these classes of methods can be counter-productive in real-world settings where the main task label is often correlated with the concept.
Examples include natural language inference ( main task) where the presence of negation (spurious concept ) may be correlated with the “contradicts” label; or tweet sentiment classification ( main task) where the author’s gender (sensitive concept ) may be correlated with the sentiment label.
Our key result is based on the observation that both these methods internally use an auxiliary (or probing) classifier [1, 43] that aims to predict the spurious concept based on the representation learnt by the main classifier.
We show that an auxiliary classifier cannot be a reliable signal on whether the representation includes features that are causally derived from the concept.
As previous work has argued [5, 45, 42, 3], if the representation features causally derived from the concept are not predictive enough, the probing classifier for the concept can be expected to rely on correlated features to obtain a higher accuracy.
However, we show a stronger result: this behavior holds even when there is no potential accuracy gain and the concept’s features are easily learnable.
Specifically, even when the concept’s causally-related features alone can provide 100% accuracy and are linearly separable with respect to a binary probing task label, the probing classifier may still learn non-zero weights for the correlated main-task relevant features.
Based on this result, under some simplifying assumptions, we prove that both post-hoc and adversarial training methods can fail to remove the undesired concept, remove useful task-relevant features in addition to the undesired concept, or do both.
As an extreme case, we show that post-hoc removal methods can lead to a random-guess main-task classifier by removing all task-relevant information from the representation.
Empirical results on four datasets—natural language inference, sentiment analysis, tweet-mention detection, and a synthetic task—confirm our claims.
Across all datasets, as the correlation between the main task and the concept increases, post-hoc removal using null space projection removes a higher amount of the main-task features, eventually leading to a random-guess classifier.
In particular, for a pre-trained classifier that does not use the concept at all, the method modifies the representation to yield a lower main-task accuracy, irrespective of the correlation between the main task and the concept.
Similarly, for the adversarial removal method, we find that it does not remove all conceptrelated features.
For most datasets, the concept-related features left within an adversarial classifier’s representation are comparable to that for a standard main-task classifier.
Our theoretical analysis complements past empirical critique of adversarial methods for concept removal [13].
More generally, we extend the literature on probing classifiers and their unreliability [5].
Adding to known limitations of standard methods [20, 37] based on accuracy of a probing classifier, our results show that recent causally-inspired methods like amnesic probing [14] are also flawed because they depend on access to a good quality concept classifier.
Our contributions include:
• Theoretical analysis of null space and adversarial removal methods showing that they fail to remove an undesirable concept from a model’s representation, even under favorable conditions.
• Empirical results on four datasets showing that the methods are unable to remove a spurious concept’s features fully and end up unnecessarily removing task-relevant features.
• A practical spuriousness score motivated by theory for evaluating the output of removal methods.
Concept removal: Background and problem statement
For a classification task, let ( x i , y im ) ni
=1 be set of examples in the dataset D m , where x i ∈
X are the input features and y
im ∈ Y m the label.
We call this the main task and label y
im the main task label.
The main task classifier can be written as c m ( h ( x )) where h : X → Z is an encoder mapping the input x to a latent representation z :
= h ( x ) and c m : Z → Y m is the classifier on top of the representation Z .
Additionally, we are given labels for a spurious or sensitive concept,
y p ∈ Y p , and our goal is to ensure that the representation h ( x ) learnt by the main classifier does not include features causally derived from the concept.
Below we define what it means to be “causally derived”: the representation should not change under an intervention on concept.
Definition 2.1.
( Concept-causal feature )
A feature Z j ∈ Z (jth dimension of h ( x ) ) at the representation layer is defined to be causally derived from a concept (concept-causal for short) if upon changing the value of the concept label, the corresponding change in the input’s value x will lead to a change in the feature
’s
( Z j ) value.
For simplicity, we assume that the non-concept-causal features are the main task features.
Often, the main task and the concept label are correlated; hence the learnt representation h ( x ) for the main task may include concept-causal features too.
A concept removal algorithm is said to be successful if it produces a clean representation h ′ ( x ) to be used by the main-classifier that has no concept-causal features and it does not corrupt or remove the main-task features.
If the representation does not contain such features, the main classifier cannot use them [13].
In practice, it is okay if the concept-causal features are not completed removed, but our key criterion is that the removal process should not remove the correlated main task features.
Existing concept removal methods.
When the text input can be changed based on changing the value of concept label, methods like data augmentation [24] have been proposed for concept removal.
However, for most sensitive or spurious concepts, it is not possible to know the correct change to apply at the input level corresponding to a change in the concept’s value.
Instead, methods based on the representation layer have been proposed.
To determine whether features in a representation are causally derived by the concept, these methods train an auxiliary, probing classifier c p : Z → Y p where y p ∈
Y p is the label of the concept we want to remove from the latent space z ∈ Z .
The accuracy of the classifier indicates the predictive information about the concept embedded in the representation.
This probing classifier is then used to remove the sensitive concept from the latent representation which will ensure that the main-task classifier cannot use them.
Two kinds of feature removal methods have been proposed: 1) post-hoc methods such as null space removal [34, 14, 26, 21], where removal is done after the main-task classifier is trained; 2) adversarial methods that jointly train the main task with the probing classifier as the adversary (e.g., [16, 49, 35, 36]).
For adversarial removal, recent empirical results cast doubt on the method’s capability to fully remove the sensitive concept from the model’s representation [13].
We extend those results with a rigorous theoretical analysis and provide experiments for both adversarial and post-hoc removal methods.
Attribute removal using probing classifier can be counter-productive
As mentioned above, both removal methods internally use a probing classifier as a proxy for the concept’s features.
In §3.1, we start off by showing that for any classification task be it probing or main-task classification, it is difficult to learn a clean classifier which doesn’t use any spuriously correlated feature (Lemma 3.1 and Lemma B.5).
Hence the key assumption driving the use of predictive classifiers within both removal methods is incorrect.
Next in §3.2 and 3.3, we will show how these individual components’ failure leads to the failure of both removal methods.
Finally, in §3.4, we propose a practical spuriousness score to assess the output classifier from any of the removal methods.
Throughout this section, we assume that both the main task label y m and probing task label y p are binary ( ∈ {− 1 , 1 } ) and there is a basic, fixed encoder h converting the text input to features in the representation space (e.g., it can be a pre-trained model like BERT [11]).
3.1 Fundamental limits to learning a clean classifier: Probing and Main Classifier
Given z = h ( x ) and the concept label y p , the goal of the probing task is to learn a classifier c p ( z ) such that it only uses the concept-causal features and the accuracy for y p is maximized.
We assume that the main task and concept labels are correlated, so it can be beneficial to use main-task features to maximize accuracy for y p .
As argued in the probing literature [20, 5], if there are features in z outside concept-causal that help improve the accuracy of the classifier, a classifier trained on standard losses such as cross-entropy or max-margin is expected to use those features too.
Below we show a stronger result: even when there is no accuracy benefit of using non concept-causal features, we find that a probing classifier may still use those features.
Creating a favorable setup for the probing classifier.
Specifically, we create a setting that is the most favorable for a probing classifier to use only concept-causal features: 1) no accuracy gain on using features outside of concept-causal because concept labels are linearly separable using concept-causal features; and 2) disentangled representation so that no further representation learning is required.
Yet we find that a trained probing classifier would use non-concept-causal features.
Assumption 3.1 (Disentangled Latent Representation) .
The latent representation z is disentangled and is of form [ z m , z p ] , where z p ∈ R d p are the concept-causal features and z m ∈
R
d m are the features causally derived from the main task label along with rest of the features in latent representation.
Here d m and d p are the dimensions of z m and z p respectively.
Assumption 3.2 (Concept-causal Feature Linear Separability) .
The concept-causal features ( z p ) of the latent representation ( z ) are linearly separable/fully predictive for the concept labels y p , i.e., y ip · (ˆ ϵ p · z ip + b p ) > 0 for all datapoints ( x i , y ip ) for some ˆ ϵ p ∈ R d p .
The effect of spurious correlation between concept and label.
Now we are ready to state the key lemma which will show that if there is a spurious correlation between the main task and concept labels such that the main-task features z m are predictive of the concept label for only a few special points, then the probing classifier c p ( z ) will use those features.
We operationalize spurious correlation as,
Assumption 3.3 (Spurious Correlation) .
For a subset of training points S ⊂ D p in the training dataset for a probing classifier, z m is linearly-separable with respect to concept label y p , i.e., y ip · (ˆ ϵ m · z
im + b m ) >
0 ∀ i ∈ S , where ˆ ϵ m ∈ R d m and b m ∈ R .
[Figure]
[Figure]
Note that even for points in S , there is no accuracy benefit of using z m features since z p already provides 100% accuracy for all points in D p .
Without loss of generality, the lemma below assumes a zero-centered latent space ( b m and b p are zero).
For simplicity, we assume that the encoder h ( · ) which maps the input X to latent representation Z is frozen or non-trainable.
Following [31], we assume max-margin as training loss; under some mild conditions on separable data, a classifier trained using logistic/exponential loss converges to max-margin classifier given infinite training time [44, 22].
Lemma 3.1.
Let the latent representation be frozen and disentangled
such that z
=
[ z m , z p ] (Assm 3.1), and concept-causal features z p are fully predictive for the concept label y p (Assm 3.2).
Let c ∗ p ( z )
= w p · z p be the desired clean linear classifier trained using max-margin objective (§B.1) that only uses z p for its prediction.
Let z m be the main task features, spuriously correlated s.t.
z m are linearly-separable w.r.t.
probing task label y p for the margin points of c ∗ p ( z ) (Assm 3.3).
Then, assuming a zero-centered latent space, a concept-probing classifier c p trained using max-margin objective will use the spurious feature, i.e., c p ( z ) =
w p · z p + w m · z m where w m ̸ = 0 .
Proof Sketch.
We can perturb c ∗ p ( z ) to have w m ̸ = 0 and show that there always exist a perturbed classifier which have bigger margin than c ∗ p ( z ) .
For detailed proof see §B.2.
Our result shows that not just accuracy, even geometric skews in the dataset can yield an incorrect probing classifier.
In §B.3 we prove that the assumptions for Lemma 3.1 are both sufficient and necessary for a classifier to use non-concept-features z m when z p is 1-dimensional.
Lemma 3. generalizes a result from [31] by using fewer assumptions (we do not restrict z m to be binary, do not assume that z m and z p are conditionally independent given y , and do not assume monotonicity of classifier norm with dataset size).
We present a similar result for the main task classifier: under spurious correlation of concept and main task labels, the main task classifier would use concept-causal features even when 100% accuracy can be achieved using only main task features (Lemma B.5, §B.4).
3.2 Failure mode of post-hoc removal methods: Null-space removal (INLP)
The null space method [34, 14], henceforth referred as INLP , removes a concept from latent space by projecting the latent space to a subspace that is not discriminative of that concept.
First, it estimates the subspace in the latent space discriminative of the concept we want to remove by training a probing classifier c p : Z → Y p , where Y p is the concept label.
Then the projection is done on to the null-space of this probing classifier which is expected to be non-discriminative of the concept.
For instance, [34] use a linear probing classifier c p ( z ) to ensure that the any linear classifier cannot recover the removed concept from modified latent representation z ′ and hence the main task classifier ( c m ( z ′ ) ) becomes invariant to removed concept.
Also, they recommend running this removal step for multiple iterations to ensure the unwanted concept is removed completely (details are in §C.1).
Below we state the failure of the null-space method using z i ( k ) to denote the representation z
i after k steps of INLP.
Theorem 3.2.
Let c m ( z ) be a pre-trained main-task classifier where the latent representation z
= [ z m , z p ] satisfies Assm 3.1 and 3.2.
Let c p ( z ) be the probing classifier used by INLP to remove the unwanted features z p from the latent representation.
Under Assm 3.3, Lemma 3.1 is satisfied for the probing classifier c p ( z ) such that c p ( z )
=
w p · z p + w m · z m and w m ̸ = 0 .
Then, 1.
Damage in the first step of INLP.
The first step of INLP will corrupt the main-task features and this corruption is non-invertible with subsequent projection steps of INLP.
(a) Mixing:
If w p ̸ = 0 , the main task z m and concept-causal features z p will get mixed such that z
i (1) =
[ g ( z im , z ip ) , f ( z ip , z im )]
̸ =
[ g ( z im ) , f ( z ip )] .
Thus, the latent representation is no longer disentangled and removal of concept-causal features will also lead to removal of main task features.
(b) Removal: If w p = 0 , then the first projection step of linear-INLP will do opposite of what is intended, i.e., damage the main task features z m (in case z m ∈ R , it will completely remove z m ) but have no effect on the concept-causal features z p .
Removal in the long term: The L2-norm of the latent representation z decreases with every projection step as long as the parameters of probing classifier at a step ( w k ) does not lie completely in the space spanned by parameters of previous probing classifiers, i e., span( w 1 , . . .
, w k − , z i (0) and z i (0) in direction of w k is not trivially zero.
Thus, after sufficiently many steps, INLP can destroy all information in the representation, z i ( ∞ ) =
Proof Sketch.
From Lemma 3.1, probing classifier will have non-zero weight on the spurious feature z m .
Hence, after projection operation they get mixed up and we show projection operator is noninvertible.
Then we show projection operation is lossy, i.e., removes some norm of z when null-space of c p ( z ) is not orthogonal to z .
For detailed proof see §C.2.
Failure Mode: Fig.
1a-1c demonstrate the mixing problem stated in Theorem 3.2, where a fair classifier becomes unfair after the first step of projection.
Note that after first step the main task classifier’s accuracy will drop because of this mixing of features, affecting INLP-based probing methods like Amnesic Probing [14] that interpret a drop in the main classifier’s accuracy after INLP projection as evidence that the main classifier was using the sensitive concept.
3.3 Failure mode of adversarial removal methods
To remove the unwanted features z p from the latent representation, adversarial removal methods jointly train the main classifier c m : Z → Y m and the probing classifier c p : Z → Y p by specifying c p ’s loss as an adversarial loss.
For details refer to §D.1.
As in Lemma 3.1, we assume that the encoder h : X → Z mapping the input to the latent representation Z is frozen.
To allow training for the main task classifier, we introduce additional representation layers after it.
For simplicity in the proof, we assume a linear transformation to the latent representation h 2 : Z → ζ .
This layer is followed by the linear main-task classifier
c m : ζ → Y m , as before.
The probing classifier c p : ζ → Y p is trained adversarially to remove z p from the latent representation ζ .
Thus, the goal of the adversarial method can be stated as removing the information of z p from ζ .
Let the main-task classifier satisfy assumptions of the generalized version of Lemma 3.1 (Lemma B.5, §B.4).
We also need two assumptions on the hard-to-classify margin points to ensure that main-task labels and concept labels are correlated on the margin points of a clean main-task classifier.
Assumption 3.4 (Label Correlation on Margin Points) .
For the margin points of a clean classifier for the main task, the adversarial-probing labels y p and the main task labels y m are correlated, i.e., w.l.o.g.,
y im
= y ip for all margins points of the clean main task classifier.
Theorem 3.3.
Let the latent representation z satisfy Assm 3.1 and be frozen, h 2 ( z ) be a linear transformation over Z s.t.
h 2 : Z → ζ , the main-task classifier be c m ( ζ ) =
w c m · ζ , and the adversarial classifier be c p ( ζ )
=
w c p · ζ .
Let all the assumptions of Lemma B.5 be satisfied for main-classifier c m ( · ) when using Z directly as input and Assm 3.2 be satisfied on Z w.r.t.
the adversarial task.
Let h ∗ 2 ( z ) be the desired encoder which is successful in removing z p from ζ .
Then
[Figure]
there exists an undesired/incorrect encoder h 2 ( z ) s.t.
h 2 ( z ) is dependent on z p and the main-task classifier
c m ( h 2 ( z )) has bigger margin than c m ( h ∗ 2 ( z )) and has, 1.
Accuracy ( c p ( h 2 ( z )) , y p ) =
Accuracy ( c p ( h ∗ 2 ( z )) , y p ) when adversarial probing classifier c p ( · ) is trained using any learning objective like max-margin or cross-entropy loss.
Thus, the undesired encoder h 2 ( z ) is indistinguishable from desired encoder h ∗ 2 ( z ) in terms of adversarial task prediction accuracy but better for main-task in terms of max-margin objective.
L h 2 (cid:0) c m ( h 2 ( z )) , c p ( h 2 ( z )) (cid:1) <
L h 2 (cid:0) c m ( h ∗ 2 ( z )) , c p ( h ∗ 2 ( z )) (cid:1) additionally under Assm 3.4, D. and given concept-causal feature z
Mp of any margin point z M of c m ( h ∗ 2 ( z )) is more predictive of main-task than z Pp of any margin-point z P of c p ( h ∗ 2 ( z )) for probing task (see Assm D.1).
Here
L h 2 ( · )
=
L ( c m ( · ) , y m ) − L ( c p ( · ) , y p ) is the combined adversarial loss w.r.t.
to h 2 and L ( · ) is the max-margin loss for a classifier (see §D.1).
Thus undesired encoder h 2 ( z ) is preferable over desired encoder h ∗ 2 ( z ) for both main and combined adversarial objective.
Proof Sketch.
First we show that w.l.o.g. ζ ∈ R .
Then the main classifier is same as h 2 ( ζ ) and adversarial classifier is α · h 2 ( ζ ) for some α ∈ R .
Then we can show that given a clean h ∗ 2 such that ζ dont contain sensitive z p , we can always perturb it to get h 2 =
w m · z m + w p · z p st. w p ̸ = 0 .
Then we show that this perturbed encoder has bigger margin for main classifier and equivalent accuracy or larger loss for adversarial classifier.
For detailed proof see §D.2.
3.4 Implications for real-world data: A metric for quantifying degree of spuriousness
Our theoretical analysis shows that probing-based removal methods fail to make the main task classifier invariant to unwanted concepts.
However, to verify whether the final classifier is using the concept or not, the theorem statements require knowledge of the concept’s features z p .
For practical usage, we propose a metric that quantifies the degree of failure or spuriousness for both the main and probing classifier.
For simplicity, we define it assuming that both main task and concept are binary.
Let D m,p be the dataset where for every input x i we have both the main task label y m and the concept label y p .
We define 2 × 2 groups, one for each combination of ( y m , y p ) .
Without loss of generality, assume that the main-task label y m = 1 is spuriously correlated with concept label y p = 1 and similarly y m = 0 is correlated with y p = 0 .
Thus, ( y m = 1 , y p = 1) and ( y m = 0 , y p = 0) are the majority group S maj while groups ( y m = 1 , y p = 0) and ( y m = 0 , y p
= 1) make up the minority group ( S min ).
We expect the main classifier to exploit this correlation and hence perform badly on S min where the correlation breaks.
Following [41], we posit that minority group accuracy i.e
Acc ( S min ) can be a good metric to evaluate the degree of spuriousness .
We bound the metric by comparing it with the accuracy on S min of a “clean” classifier that does not use the concept features.
Definition 3.1 (Spuriousness Score ( ψ )) .
Given a dataset, D m,p = S min ∪ S maj containing binary task label and binary concept, let Acc f ( S min ) be the accuracy of the given main-task classifier ( f ) on the minority group S min and Acc ∗ ( S min ) be the accuracy on the minority group of a clean classifier that does not use the unwanted spurious concept.
We define spuriousness score of “ f ” as:
[Equation]
To estimate Acc ∗ ( S min ) , we subsample the dataset such that y p takes a single value in the sample and train the main classifier on it, as in [37].
Here the probing label y p no longer is correlated with the main task label y m .
There can be other ways to estimate Acc ∗ ( S min ) , e.g., by reweighting the data or using the accuracy on S maj .
However, we found that the former had high variance and the latter requires an equal-noise assumption, Acc ( S maj ) =
Acc ( S min ) , on a clean main task classifier.
Experimental Results
Theorems 3.2 and 3.3 show the failure of concept removal methods under a simplified setup and max-margin loss.
But current deep-learning models are not trained using max-margin objective and might not satisfy the required assumptions (Assm 3.1,3.2,3.3,3.4).
Thus, we now verify the failure modes on three real-world datasets and one synthetic dataset, without making any restrictive assumptions.
We use RoBERTa [27] as default encoder and fine-tune it over each real-world dataset.
For Synthetic-Text dataset we use the sum of pre-trained GloVe embeddings [32] of words in sentence as the default encoder.
For details on the experimental setup, refer §E.
4.1 Datasets: Main task and spurious/sensitive concept
Real-world data.
We use three datasets: MultiNLI [47], Twitter-PAN
[33] and Twitter-AAE [7].
In MultiNLI, given two sentences—premise and hypothesis—the main task is to predict whether hypothesis entails , contradicts or is neutral with respect to premise.
We simplify to a binary task of predicting whether a hypothesis contradicts the premise or not.
Since negation words like nobody,no,never and nothing have been reported to be spuriously correlated with the contradiction label [18], we create a ‘negation’ concept denoting the presence of these words.
The goal is to remove the negation concept from a NLI model’s representation space.
In Twitter-PAN16, the main task is to detect whether a tweet mentions another user or not, as in [13].
The dataset contains gender label for each tweet, which we consider as the sensitive concept to be removed from the main model’s representation.
In Twitter-AAE, again following [13], the main-task is to predict binary sentiment labels from a tweet’s text.
The tweets are associated with race of the author, the sensitive concept to be removed from the main model’s representation.
Synthetic-Text.
To understand the reasons for failure, we introduce a Synthetic-Text dataset where it is possible to change the input text based on a change in concept (thus implementing Def. 2.1).
Here we can directly evaluate whether the concept is being used by the main-task classifier by intervening on the concept (adding or removing) and observing the change in model’s prediction.
The main-task is to predict whether a sentence contains a numbered word (e.g., one, fifteen, etc. ).
We introduce a spurious concept (length) by increasing the length of sentences that contain numbered words.
Predictive correlation.
To assess robustness of removal methods, we create multiple datasets with different predictive correlation between the two labels y m and y p .
The predictive correlation ( κ ) measures how informative one variable is for predicting the other, κ =
Pr ( y m · y p > 0) =
(cid:80)
Ni =1 1 [ y m · y p > 0] N , where N is the size of dataset and 1 [ · ] is the indicator function that is 1 if the argument is true otherwise 0 .
Predictive-correlation lies in κ ∈ [0 . 5 , 1] where κ = 0 .
5 indicates no correlation and κ = 1 indicates that the attributes are fully correlated.
For more details on the datasets and how we vary the predictive-correlation, refer to §E; and for additional results see §F.
4.2 Results: Null space removal
In general, for any model given as input to INLP, it may be difficult to verify whether INLP removed the correct features.
Hence, we construct a benchmark where the input classifier is clean , i.e., it does not use the concept at all.
We do so by training on a subset of data with one particular value of spurious concept label, as in [37].
Since the input classifier does not use the concept-causal features, we expect that INLP should not have any effect on the main task classifier.
[Figure]
i n
[Figure]
T
[Figure]
k
[Figure]
cc u r a c y (
[Figure]
Eventually all task-relevant features are destroyed.
We start with the Synthetic-Text dataset by training a clean classifier on the main-task and input it to INLP for removing the spurious concept.
To keep the conditions favorable for INLP, both the main task and concept-probing task can achieve 100% accuracy using their causally derived features respectively.
In Fig.
3a, colored lines show datasets with different levels of predictive correlation κ that are provided to INLP and iterations 21show individual steps of null-space removal.
Since, the given pre-trained classifier was clean , i.e., not using the concept features, null-space removal shouldn’t have any effect on it.
We observe that for all value of κ , the main-task classifier’s accuracy eventually goes to 50% random guess accuracy implying that the main-task related attribute has been removed by INLP, as predicted by Theorem 3.2.
Higher the value of correlation κ , faster the removal of main-task attribute happens.
We obtain a similar pattern over the real-world datasets.
Fig.
3d,3e and 3f show a decrease in the main-task accuracy even when the input classifier for each dataset is ensured to be clean : except for κ = 0 .(no correlation), all values of κ yield a random-guess classifier after applying INLP on MultiNLI while they yield classifiers with less than 60% accuracy for Twitter-PAN16 and Twitter-AAE.
Early stopping increases the reliance on spurious features.
To avoid the full collapse of the main-task features, a stopping criteria in INLP is to stop when the main-task classifier’s performance drops [34].
In Fig.
3b we measure sensitivity of the Synthetic-Text main task classifier w.r.t.
to the spurious concept by changing the feature in input sentences corresponding to spurious concept and measuring the change in main-task classifier prediction probability ( ∆ Prob, see §E.8).
At lower iterations of INLP, the change in main-task output due to change in spurious concept’s value, ∆ Prob, is higher than that of the input classifier.
For example, for κ = 0 . 8 , the main-task classifier’s performance drops for the first time at iteration 27, but it has a high ∆ Prob ≈ 25% as shown in
Fig.
3b.
Hence it is possible that stopping prematurely will lead us to a classifier which is more unfair or reliant to spurious concept than it was before, consistent with the first statement in Theorem 3.
stating that INLP will lead to mixing of features in latent space.
Failure of causally-inspired probing.
Amnesic Probing [14] declares that a sensitive concept is being used by the model if, after removal of the concept from from the latent representation using
INLP, there is a drop in the main-task performance.
But Fig. 3a, 3d, 3e and 3f show that even when the input classifier for its corresponding main task is clean, i.e., does not use the sensitive concept, INLP leads to drop in performance of the main-classifier.
Hence, removal-based methods like Amnesic probing will falsely conclude that the sensitive concept is being used.
[Figure]
i n
[Figure]
u r i o u s n e ss S c
o
r e
[Figure] [Figure]
4.3 Results: Adversarial removal
We now demonstrate failure of the adversarial-removal method (AR) in removing the spurious concept from the main classifier.
We train a separate main-task classifier without any adversarial objective with standard cross-entropy loss objective (referred as ERM ).
Then we compare standard ERM training of the main classifier with the AR method over the same number of epochs (20).
We follow the training procedure of [13] and conduct a hyper-parameter sweep on the adversarial training strength to select the value which is most effective in removing the concept.
For details, refer to §E.
Cannot remove the spurious concept fully.
For MultiNLI, Fig.
4b shows the spuriousness score (Def 3.1) of ERM and AR classifiers as we vary the predictive correlation ( κ ) between the main-task label and sensitive concept label in the training dataset.
While the spuriousness score for classifier trained using AR (blue curve) is lower than that of ERM for all values of κ , it is substantially away from zero.
Thus, the AR method fails to completely remove the spurious concept completely from the latent representation.
By inspecting the concept probing classifier accuracy for ERM and AR in Fig.
4c, we obtain a possible explanation.
The probe accuracy after adversarial training doesn’t decrease to 50% but stops at accuracy proportional to the predictive correlation κ .
This is expected since even if the AR would have been successful in removing the concept-causal features, the maintask features would still be predictive of the concept label by κ due to the spurious correlation between them.
However, the converse is not true: an accuracy of κ does not imply that the concept is fully removed.
The results substantiate the first statement of Theorem 3.3: given two representations where one ( desired ) does not have concept features while the other ( undesired ) contains the concept features, the undesired one may be better for the main task accuracy even as both may have the same probing accuracy.
Fig.
4d, 4e and 4f show the spuriousness score of AR in comparison to classifier trained with ERM on Twitter-PAN16, Twitter-AAE and Synthetic-Text datasets respectively.
The failure of
AR is worse here: there is no significant reduction in spuriousness score for AR in comparison to
ERM.
For the Synthetic-Text dataset, ERM has zero spuriousness score but AR has non-zero score.
We expand more on this observation and include additional results on adversarial removal in §F.3.
Comparison with previous work.
If post-removal the latent representation used by the main-task classifier is still predictive of the removed concept, [13] claimed it as a failure of the removal method.
However, this claim may not be correct since a feature could be present in the latent space and yet not used by model [37].Therefore, we propose the spuriousness score for the main classifier as a metric to verify if a feature is being used.
Ablations.
In Appendix, we report results on using BERT instead of RoBERTa as the input encoder (§F.2, F.3), the effect of using different modeling choices like loss-function, regularization, e.t.c.
(§F.4), and the behavior of probing classifiers when concept is not present in latent space (§F.1).
Related work
Concept removal methods.
When the removal of a concept can be simulated in input space (e.g., in tabular data or simpler concepts), removing a concept directly using data augmentation [24] or gradient regularization [40, 23] can work.
However, concept removal is non-trivial when change in a concept cannot be propagated via change in input tokens.
Combining the ideas of null space and adversarial removal, methods like [35, 36] restrict the adversarial function to be a projection operation and derive a closed form solution for concept removal.
Other approaches use explanations of the classifier’s prediction for concept removal [25].
Our work highlights the difficulty of building an estimator for the features causally derived from a concept, as a general limitation for concept removal.
Limitations of a probing classifier for model interpretability.
We also contribute to the growing literature on the limitations of a probing classifier’s accuracy in capturing whether the main classifier is using a concept [5].
It is known that probing classifiers capture not just the concept but any other features that may be correlated with it [41, 3, 24, 46].
As a result, many improvements have been proposed to better estimate whether a concept is being used, including the use of control labels or datasets [20, 37].
Parallely, new causality-inspired probing methods [14] compare the main task accuracy on a representation without the concept that is constructed using the null space removal method.
The hope is that such improvements can make probing more robust.
Our results question this direction.
To demonstrate the fundamental unreliability of a probing classifier, we construct a setup that is most favorable for learning only the concept’s features and still find that learned probing classifier includes a non-zero weight for other features, thus limiting the effectivenevss of any interpretation method based on it.
Conclusion
Our theoretical and experimental evaluations show that it is difficult to create a probing-based removal method due to the fundamental limitation of learning a “clean” probing classifier.
We recommend two tests for validating removal methods.
First, we provide a sanity-check: any reasonable removal method should not modify a “clean” classifier that does not use any spurious features to produce a final classifier that uses those features.
Second, we propose a spuriousness score that can be used to evaluate the dependence of any classifier on spurious features.
As a future direction, we encourage the community to develop more such sanity-check tests to evaluate proposed removal methods.
Alternatively, we point attention to other approaches that may provide better guarantees for concept removal.
An example is extending data augmentation techniques like counterfactual data augmentation ([24, 10]) to non-trivial concepts.
For a given training point that may include a spurious correlation, a new data point is generated that breaks the correlation but keeps the semantics of the rest of the text identical (hence the name, “counterfactual”).
This can be done by human labeling or handcrafted rules for modifying text (e.g., Checklists [39]).
Then the main classifier is regularized to have the same representation for such pairs of inputs ([4, 28]).
By construction, with good quality pairs, such a method will not remove task-relevant features and will satisfy the sanity checks listed above.
That said, a limitation is that the removal quality will depend on the diversity of the counterfactual examples generated and whether they capture all aspects of the spurious concept.
Another direction is to take inspiration from the algorithmic fairness literature [19, 30] and focus on the predictions of the classifier rather than the representation.
Compared to removal in latent space, enforcing certain fairness properties on model predictions is a more well-formed task, more interpretable, and definitely more relevant if the final goal is fair decision making.
Limitations.
A limitation of our theoretical work is assuming frozen or non-trainable latent representation which makes the analysis of task-classifier trained on top of them relatively easier.
We address this limitation in our empirical work where we do not make such assumptions.
Also, our work addresses failure modes of two popular methods, null space removal and adversarial removal.
We conjecture that any other method based on probing classifiers will lead to similar failure modes.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See §3 and(b)
Did you describe the limitations of your work?
[Yes] See § (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See §A (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] Our work show failure modes of existing concept removal methods.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See § (b) Did you include complete proofs of all theoretical results?
[Yes] See §B,C and D. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See §E. (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See §E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See §4, §E and §F. (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See §E.9.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See §4 and §E. (b) Did you mention the license of the assets?
[Yes] See §E.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Broader Impact and Ethical Consideration
Removal of spurious or sensitive concepts is an important problem to ensure that machine learning classifiers generalize better to new data and are fair towards all groups.
We found multiple limitations with current removal methods and recommend caution against the use of these methods in practice.
B Probing and Main Classifier Failure Proofs B.1 Notation and Setup: Max-margin Classifier
We assume that encoder h : X → Z , mapping the input to latent representation is frozen/nontrainable.
Thus for every input x i in the dataset D , we have a corresponding latent representation z i which is fixed.
Also, the latent representation Z is disentangled i.e z
=
[ z m , z p ]
where z m are the main task features, i.e., causally derived from the main task label and z p are the concept-causal features, causally derived from the concept label.
Let c p ( z )
=
w p · z p + w m · z m be the linear probing classifier which we train using max-margin objective.
The hyperplane c p ( z ) = 0 is the decision boundary of this linear classifier.
The points which fall on one side of the decision boundary ( c p ( z ) > 0 ) are assigned one label (say positive label 1) and the rest are assigned another label (say negative label -1).
The margin M c p of this probing classifier ( c p ( z )) is the distance of the closest latent representation ( z ) from the decision boundary.
The points which are closest to the decision boundary are called the margin points .
The distance of a given latent representation z
i having class label
y i , where y i ∈ {− 1 , 1 } , from the decision boundary is given by
[Equation]
where ∥ w ∥ is the L2 norm of parameters w = [ w p , w m ] of the probing classifier c p ( z ) .
Max-Margin (MM):
Then the max-margin classifier is trained by optimizing the following objective:
[Equation]
For ease of exposition we convert this objective into multiple equivalent forms.
To do this we observe that scaling the parameters of c p ( z ) by a positive scalar γ i.e w → γ w and b → γb does not change the distance of the point ( M c p ( z i ) ) from the decision boundary.
MM-Denominator Version: We can use this freedom of scaling the parameters to set m c p ( z i )
=for the closest point of any given probing classifier c p ( z ) , thus all the data points will satisfy the constraint,
[Equation]
giving us the final max-margin objective:
[Equation]
under the constraint m c p ( z i ) ≥ 1 corresponding to all the points in the dataset.
MM-Numerator Version: Alternatively, one can choose γ such that ∥ w ∥
= c where c ∈ R is some constant value.
The the modified objective becomes:
[Equation]
under constraint ∥ w ∥ = c which is usually set to 1 .
We will use one of these formulations in our proofs based on the ease of exposition and give a clear indication when we do so.
One can refer to Chapter 7 , Section 7 .
1 of [6] for furthur details about max-margin classifiers and different formulations of the max-margin objective.
B.2 Proof of Sufficient Condition: Lemma 3.1 and Lemma B.
Lemma 3.1.
Let the latent representation be frozen and disentangled
such that z
=
[ z m , z p ] (Assm 3.1), and concept-causal features z p are fully predictive for the concept label y p (Assm 3.2).
Let c ∗ p ( z )
= w p · z p be the desired clean linear classifier trained using max-margin objective (§B.1) that only uses z p for its prediction.
Let z m be the main task features, spuriously correlated s.t.
z m are linearly-separable w.r.t.
probing task label y p for the margin points of c ∗ p ( z ) (Assm 3.3).
Then, assuming a zero-centered latent space, a concept-probing classifier c p trained using max-margin objective will use the spurious feature, i.e., c p ( z ) =
w p · z p + w m · z m where w m ̸ = 0 .
In this section we prove that, given the assumption in Lemma 3.1 is satisfied, they are sufficient for a probing classifier c p ( z ) to use the spuriously correlated main-task feature z m .
See §B.1 for detailed setup and max-margin training objective.
Also, we could use the same line of reasoning to prove a similar result for the main-task classifier i.e. when conditions in Lemma B.5 is satisfied, the main-task classifier will use the spuriously correlated concept-causal feature z p .
To keep the proof general for both the lemmas, we prove the result for a general classifier c ( z ) trained to predict a task label y .
Here the latent representation z be of form z =
[ z inv , z sp ] where z inv are the features which are causally-derived from the task-label y (“invariant” features) and z sp be the features spuriously correlated to the task label y .
With respect to probing classifier c p ( z ) in Lemma 3.1 z inv :=
z p and z sp :=
z m .
Similarly, for the main-task classifier in Lemma B.5, z inv :=
z m and z sp :=
z p .
For ease of exposition, we define two categories of classifiers based on which features they use:
Definition B.1 (Purely-Invariant Classifier) .
A linear classifier of form c ( z ) =
w inv · z inv + w sp · z sp + b is called "purely-invariant" if it does not use the spurious features z sp i.e., w sp = 0 .
Definition B.2 (Spurious-Using Classifier) .
A linear classifier of form c ( z ) =
w inv · z inv + w sp · z sp + b is called "spurious-using" if it uses the spurious features z sp i.e., both w sp ̸ = 0 and w inv ̸ = 0 .
Proof of Lemma 3.1 and B.5.
Let c inv ( z ) =
w inv · z inv be the clean /purely-invariant classifier trained using the max-margin objective using the MM-Denominator formulation given in Eq.
4 such that w inv ̸
= 0 .
The classifier c inv ( z ) is 100% predictive of the task labels y
(from Assm 3.2 for the probing task or Assm B.1 for the main-task).
Here the bias term b = 0 since we assume the latent representation z is zero-centered.
The norm of this classifier ( c inv ( z ) ) is ∥ w inv ∥ and the distance of each input latent representation ( z i ) with class label
y
i
( y i ∈ {− 1 , 1 } ) from the decision boundary ( c inv ( z ) = 0 ) is given by Eq.
1 i.e.:
[Equation]
Since we have used the MM-Denominator version of max-margin to train c inv ( z ) , from Eq.
3 we have m inv ( z i )
=
1 for the margin-points and greater than 1 for rest of the points.
Next we will construct a new classifier parameterized by α ∈ [0 , 1] by perturbing the clean/purely-invariant classifier c inv ( z ) such that:
[Equation]
where ˆ ϵ sp ∈ R d sp is a unit vector in spurious subspace of features, d sp is the dimension of the spurious feature subspace ( z sp ).
We observe that the norm of this perturbed classifier c
α
( z ) is also ∥ w inv ∥ , which is same as the clean/purely-invariant classifier c inv ( z ) .
Thus from Eq.
1, the distance of any input z
i with class label y i from the decision boundary of this perturbed classifier c
α
( z ) is given by:
[Equation]
The perturbed classifier will be spurious-using i.e use the spurious feature z sp when α ∈ [0 , 1) since w sp
= ( ∥ w inv ∥ √ 1 − α 2 ) ̸
= 0 for these setting of α .
Thus to show that there exist a spurioususing classifier which has a margin greater than the margin of the purely-invariant classifier, we need to prove that there exist an α ∈ [0 , 1) such that c α ( z ) has bigger margin than c inv ( z ) i.e. min z M α ( z ) >
min z M inv ( z ) .
Since norm of parameters of both the classifier is same, substituting the expression of M α and M inv from Eq.
6 and 8, we need to show m α ( z i ) > 1 for all z i .
We have:
[Equation]
Let S my denote the set of margin-points of purely-invariant classifier c inv ( z ) with class label y
having m inv ( z ) = 1 and S ry contain rest of points (non-margin points) having m inv ( z ) > 1 with the class label y .
Here “m” stands for margin-point in superscript of S and “r” stands for rest of point with label y .
In rest of the proof, first we will show that for margin-points z m ∈ ( S my =1 ∪ S my = − 1 ) , we need the assumption that spurious feature ( z msp ) be linearly separable with respect to class label y
(Assm 3.3 for probing task or B.2 for main-task)
for having m α ( z i ) > 1 .
But for all non-margin points z r ∈ ( S ry =1 ∪ S ry = − 1 ) , we can always choose α ∈ [0 , 1) such that m α
( z i ) > 1 .
Below we handle margin and non-margin of points separately.
Case 1 : Margin Points ( S my =1 ∪ S my = − 1 ) :
For the margin-points in latent space, z m ∈ S my we have m inv ( z m )
=
and we need to show that there exists α ∈ [0 , 1) such that m α ( z m ) > 1 for all z m ∈ S
my .
From Eq.
10 we have:
[Equation] [Equation]
From Assm 3.3 for probing task or B.2 for the main-task, we know that spurious-feature z msp of marginpoints z m are linearly-separable w.r.t to task label y .
Thus there exist an unit vector ˆ ϵ sp ∈ R d sp such that y (cid:0) ˆ ϵ sp · z msp (cid:1) > 0 for all z m ∈ S my .
Also since α ∈ [0 , 1] and ∥ w inv ∥ > 0 , we have ( ∥ w inv ∥√ 1 − α 2 ) > 0 .
Hence the left hand side of Eq.
12 is > 0 for such ˆ ϵ sp .
If Assm 3.3 or
B.2 (corresponding to the task) wouldn’t have been satisfied then the above equation might have been inconsistent since right hand side is always > 0 ; since we need to find a solution to Eq.when α ∈ [0 , 1) thus (1 − α ) > 0 , but left hand side wouldn’t have been always greater than 0.
This shows the motivation why we need Assm 3.3 or B.2 for proving this lemma.
Continuing, let β :=
( y (cid:0) ˆ ϵ sp · z msp (cid:1) ) , then squaring both sides and cancelling (1 − α ) since we need to find a solution to Eq.
12 when α ∈ [0 , 1) =
⇒
[Equation]
1 − α ) > 0 , we get:
[Equation] [Equation]
After substituting back the value of β and rearranging we get:
[Equation] [Equation]
Lets define α lb 1 y :=
max z m ∈S m y
( α lb 1 y ( z m )) .
Since ∥ w inv ∥ 2 ·
[Equation]
> 0 , the right hand side of above equation α lb 1 y ( z m ) < 1 for all z m ∈ S my = ⇒
α lb 1 y < 1 , which sets a new lower bound on allowed value of α for which Eq.
12 is satisfied.
Thus when α ∈ ( α lb 1 y , 1) , m p ( z m ) >for all z m ∈ S
my .
That is, the perturbed probing
classifier c
α
( z ) has larger margin than purelyinvariant/clean classifier c inv ( z ) for the margin points z m ∈ S my .
Case 2: Non-Margin Points ( S ry =1 ∪ S ry = − 1 ) :
For the non-margin points z r ∈ S ry in the latent space we have m inv ( z r ) > 1 .
Let γ :=
min
z r ∈S ry (cid:0) m inv ( z r ) (cid:1) thus we also have γ > 1 .
Let α ̸
=
and we choose α such that:
[Equation]
Substituting the value of γ we get:
[Equation]
Since γ > 1 , thus right hand side in above equation α lb 2 y < 1 , which sets a new lower bound on allowed values of α .
Since m inv ( z r ) ≥ γ > 1 α for all z r ∈ S ry for α ∈ ( α lb 2 y , 1) (Eq. 20), we can write m inv ( z r )
= 1 α + η ( z r ) where η ( z r ) :
= ( m inv ( z r ) − 1 α ) > 0 for all z r ∈ S ry .
Now we need to show that there exist an α ∈ ( α lb 2 y , 1) such that m α ( z r ) > 1 for all z r ∈ S ry .
Thus from Eq.we need:
[Equation]
[Equation]
Since α ∈ ( α 2 lb , 1) , we have ( α · η ( z r )) > 0 and ∥ w inv ∥√ 1 − α 2 > 0 .
Let’s define δ ( z r ) :
=
y (cid:0) ˆ ϵ sp · z rsp (cid:1) .
Thus for the latent-points z r ∈ S ry which have δ ( z r ) ≥ 0 , Eq.
23 is always satisfied since left side of inequality is greater than or equal to zero and right side is always less than zero.
For the points for which δ ( z r ) < 0 we have:
[Equation]
[Equation]
Now different z r will have different η ( z r ) which will give different lower bound of α .
Since the m α ( z r ) > 1 has to be satisfied for every point in z r ∈ S ry , it has to be satisfied for the point with maximum value of α lb 3 y ( z r ) which will give tightest lower bound on value of α .
Lets define α lb 3 y :=
max z r ∈S ry ( z r ) , then for m α ( z r ) > 1 , we need α > α lb 3 y .
Also, since for all z r ∈ S ry , η ( z r ) > 0 we have α lb 3 y ( z r ) <
1 = ⇒
α lb 3 y < 1 .
Finally, combining all the lower bound of α from Eq.
17, Eq.
and Eq.
28 let the overall lower bound of α be α lb given by:
[Equation]
This provides a way to construct a spurious-using classifier: given any purely-invariant classifier, we can always choose α ∈ ( α lb , 1) and construct a perturbed spurious-using classifier from Eq.
7 which has a bigger margin than purely-invariant .
Thus, given all the assumptions, there always exists a spurious-using classifier which has greater margin than the purely-invariant classifier.
B.3 Proof of necessary condition
In this section we will show that Assm 3.3 is also a necessary condition for the probing classifier to use the spurious features ( z m ) when | z p | = 1 .
That is, the probing classifier will use the spurious features if and only if spurious features satisfy Assm 3.3 for the margin points of the clean/purely-invariant (Def B.1) probing classifier when the concept-feature is 1-dimensional.
Also, same line of reasoning will hold for the main-task classifier where we will show that main-task classifier will use the spurious feature ( z p )
iff spurious feature satisfies Assm B.2 for the margin point of clean main-task classifier.
Formally:
Lemma B.1 (Necessary Condition for concept-Probing Classifier) .
Let the latent representation be frozen and disentangled (Assm 3.1)
such that z = [ z m , z p ] where z p is the concept-causal feature which is 1-dimensional scalar and fully predictive (Assm 3.2) and z m ∈ R d m .
Let c ∗ p ( z )
= w p · z p be the desired clean/purely-invariant probing classifier trained using max-margin objective which only uses z p for prediction.
Then the probing classifier trained using max-margin objective will be spurious-using i.e. c p ( z ) =
w p · z p + w m · z m where w m ̸ =
iff the spurious feature z m is linearly separable w.r.t to probing task label y p for the margin point of c ∗ p ( z ) (Assm 3.3).
Lemma B.2 (Necessary Condition for Main-task Classifier) .
Let the latent representation be frozen and disentangled (Def 3.1) such that z
=
[ z m , z p ]
where z m is the main-task feature which is 1-dimensional scalar and fully predictive (Assm B.1) and z p ∈ R d p .
Let c ∗ m ( z ) =
w m · z m be the desired clean/purely-invariant main-task classifier trained using max-margin objective which only uses z m for prediction.
Then the main-task classifier trained using max-margin objective will be spurious-using i.e. c m ( z )
=
w m · z m + w p · z p where w p ̸ =
iff the spurious feature z p is linearly separable w.r.t to main task label y m for the margin point of c ∗ m ( z ) (Assm B.2) .
Since proof of both Lemma B.1 and B.2 follows same line of reasoning, hence for brevity, following §B.2, we will prove the lemma for a general classifier c ( z ) trained using max-margin objective to predict the task-label y .
Let the latent representation be of form z = [ z inv , z sp ] where z inv ∈ R is the feature causally derived from the task-label y and z sp ∈ R dsp is the feature spuriously correlated to task label y .
With respect to probing classifier c p ( z ) in Lemma B.1 z inv :=
z p and z sp :=
z m .
Similarly, for the main-task classifier in Lemma B.2, z inv :=
z m and z sp :=
z p .
Proof of Lemma B.1 and B.2.
Our goal is to show that Assm 3.3 for probing classifier or Assm B. for the main-task classifier is necessary for obtaining a spurious-using classifier for the case when z inv is one-dimensional.
We show this by assuming that optimal classifier is spurious-using even when Assm 3.3 or B.2 breaks and then show that this will lead to contradiction.
Contradiction Assumption: Formally, let’s assume that Assm 3.3 or B.2 is not satisfied for probing or main task respectively, and the optimal classifier for the given classification task is spurious-using c ∗ ( z ) , where:
[Equation]
(30) where ∥ w ∗ sp ∥ ̸ = 0 and ˆ w ∗ sp ∈ R d sp is a unit vector in spurious-feature subspace with dimension d sp .
Let c inv ( z ) =
w inv · z inv be the optimal purely-invariant classifier.
Let both c ∗ ( z ) and c inv ( z ) be trained using the max-margin objective using MM-Denominator formulation in Eq. 4.
Thus from the constraints of this formulation (Eq. 3) we have:
[Equation]
From Assm 3.2 or B.1, the invariant feature z inv is 100% predictive and linearly separable w.r.t task label y .
Then without loss of generality let’s assume that:
[Equation]
when
[Equation] [Equation]
[Equation]
From Eq.
33 and 34 we have y · z inv > 0 thus from Eq.
32 we get:
[Equation] [Equation]
Also, from our contradiction-assumption the max-margin trained classifier is spurious-using , thus the norm of parameters of c ∗ ( z ) is less or equal to c inv ( z )
(Eq. 4).
Thus we have:
[Equation]
Form our contradiction-assumption , Assm 3.3 for concept-probing task or Assm B.2 for the main-task breaks by one of the following two ways:
Opposite Side Failure: This occurs when the spurious part of margin points (of c inv ( z ) ) on opposite sides of decision-boundary of the optimal task classifier ( c ∗ ( z ) = 0 ) are not linearly-separable with respect to task label y .
Formally, there exist two datapoints, P m + :=
[ z m + inv , z m + sp ] and P m − :=
[ z m − inv , z m − sp ] such that they are margin points of purelyinvariant classifier c inv ( z ) where P m + has class label y
= +
and P
m − has class label y
= − 1 and ∀ ˆ ϵ sp ∈ R d sp , the spurious feature z sp of both the points lies on same side of ˆ ϵ sp i.e:
[Equation]
Same Side Failure: This occurs when the spurious part of margin points (of c inv ( z ) ) on same side of decision-boundary ( c ∗ ( z ) = 0 ) are linearly-separable.
Formally, there exist two datapoints, P m 1 y :=
[ z m 1 inv , z m 1 sp ] and P m 2 y
[Equation]
inv , z m 2 sp ] such that they are margin points of purely-invariant classifier c inv ( z ) and both points have same class label y and ∀ ˆ ϵ sp ∈ R d sp , w.l.o.g we have:
[Equation]
We will use the following two lemma to proceed with our proof:
Lemma B.3.
If Assm 3.3 or B.2 breaks by opposite-side failure mode, it leads to contradiction.
Lemma B.4.
If Assm 3.3 or B.2 breaks by same-side failure mode, it leads to contradiction.
This implies that our contradiction-assumption which said that the max-margin trained optimal classifier is spurious-using even when Assm 3.3 or B.2 breaks, is wrong.
Thus Assm 3.3 for conceptprobing task or Assm B.2 for main-task is necessary for the optimal max-margin classifier to be spurious-using .
This completes our proof.
Proof of Lemma B.3.
We have two points, P m + :=
[ z m + inv , z m + sp ] and P m − :=
[ z m − inv , z m − sp ] , which break the Assm 3.3 or B.2.
From Eq.
33, z inv > 0 for all the points with label y
= 1 , thus we have z m + inv > 0 and using Eq.
39 ( w ∗ inv < w inv ) we get:
[Equation] [Equation]
[Equation]
(44) where the right hand side w inv · z m + inv = 1 since P m + is the margin-point of c inv ( z ) (Eq. 32).
Similarly from Eq.
34, z inv < 0 for all the points with label y = − 1 , thus we have z m − inv < 0 and using Eq.
39 ( w ∗ inv < w inv ) we get:
[Equation]
where the right hand side ( − 1) · ( w pinv · z m − inv )
= 1 since P m − is the margin-point of c inv ( z ) (Eq. 32).
Next from Eq.
31 we have m ∗ ( z ) ≥ 1 for all
z hence it is also true for P m + with y = 1 and P m − with y = − 1 .
Then:
[Equation]
where β m + = (cid:0) ˆ w ∗ sp · z m + sp (cid:1)
Also we have:
[Equation]
where β m − = (cid:0) ˆ w ∗ sp · z m − sp (cid:1) .
From Eq.
40 we have (cid:0) ( ˆ ϵ sp · z m + sp ) · ( ˆ ϵ sp · z m − sp ) (cid:1) ≥ 0 for all ˆ ϵ sp ∈ R d sp which states the opposite-side failure of Assm 3.3 or B.2.
Thus:
[Equation]
Now we will show that Eq. 44, 47, 50 and 53 cannot be satisfied simultaneously for any allowed value of β m + and β m − (given by Eq. 54) which are:
β m + > 0 and β m − > 0 :
From Eq.
we have − w ∗ inv z m − inv > 1 since ∥ w ∗ sp ∥ ̸ = 0 and β m − > 0 .
But from Eq.
47 we have − w ∗ inv z m − inv < 1 which is a contradiction.
β m + < 0 and β m − < 0 :
From Eq.
50 we have w ∗ inv z m + inv > 1 since ∥ w ∗ sp ∥ ̸ = 0 and β m + < 0 .
But from Eq.
44 we have w ∗ inv z m + inv < 1 which is a contradiction.
β m + = 0 and β m − ∈ R : From Eq.
50 we have w ∗ inv z m + inv ≥ 1 but from Eq.
44 we have w ∗ inv z m + inv < 1 which is a contradiction.
β m + ∈ R
and
β m − =
From Eq.
53 we have − w ∗ inv
z m − inv ≥ 1 but from Eq.
47 we have − w ∗ inv z m − inv < 1 which is a contradiction.
Thus we have a contradiction for all the possible values β m +
and β m − could take, completing the proof of this lemma.
Proof of Lemma B.4.
We have two margin-points, P m 1 y :=
[ z m 1 inv , z m 1 sp ] and P m 2 y :=
[ z m 2 inv , z m 2 sp ] , which break Assm 3.3 or B.2.
From Eq.
33 and Eq.
34 we have y · z m 1 inv > 0 and y · z m 2 inv > 0 .
Using
Eq.
( w ∗ inv < w inv ) we get:
[Equation] [Equation]
[Equation] [Equation]
where j ∈ { 1 , 2 } and right hand side w inv · ( y · z mjinv )
= 1 since P mjy is the margin point of purely-invariant classifier c inv ( z ) (Eq. 32).
From Eq.
31 we have m ∗ ( z ) ≥ 1 for all z thus also true for P m 1 y and P m 2 y .
Then:
[Equation]
where β m 1 = (cid:0) ˆ w ∗ sp · z m 1 sp (cid:1) .
Also we have:
[Equation]
where β m 2 = (cid:0) ˆ w ∗ sp · z m 2 sp (cid:1) .
Now from Eq.
41 we have (cid:0) ( ˆ ϵ sp · z m 1 sp ) · ( ˆ ϵ sp · z m 2 sp ) (cid:1) ≤ 0 for all unit vectors
ˆ ϵ sp ∈ R d sp which states the same-side failure mode of Assm 3.3 or B.2.
Thus we have: β m
[Equation] [Equation]
Now we will show that for all allowed values of β m 1 and β m 2 , Eq. 57, 60 and 63 will lead to a contradiction.
Following are the cases for different allowed values of β m 1 and β m 2 : 1.
β m 1 = 0 and β m 2 ∈ R :
Substituting β m 1 = 0 in Eq.
60 we get y · ( w ∗ inv z m 1 inv ) ≥ 1 , but from Eq.
57 we have y · ( w ∗ inv z m 1 inv ) < 1 .
Thus we have a contradiction.
β m 1 ∈ R and β m 2 = 0 :
Substituting β m 2 = 0 in Eq.
63 we get y · ( w ∗ inv z m 2 inv ) ≥ 1 , but from Eq.
57 we have y · ( w ∗ inv z m 2 inv ) < 1 .
Thus we have a contradiction.
The only case which is left now is when both β m 1 and
β m 2 is non-zero but of opposite sign.
Without loss of generality, let β m 1 > 0 , β m 2 <
and y
=
(+1) :
Substituting β m 2 < 0 and y =
(+1) in Eq.
63 we get y · ( w ∗ inv z m 2 inv ) ≥ 1 , but from Eq.
57 we have y · ( w ∗ inv z m 2 inv ) < 1 .
Thus we have a contradiction.
β m 1 > 0 , β m 2 < 0 and y =
( − 1) :
Substituting β m 1 > 0 and y
=
( − 1) in Eq.
60 we get y · ( w ∗ inv z m 1 inv ) ≥ 1 , but from Eq.
57 we have y · ( w ∗ inv z m 1 inv ) < 1 .
Thus we have a contradiction.
Thus we have a contradiction for all the possible values β m 1 , β m 2 and y could take, completing the proof of this lemma.
B.4 Problem with learning a clean main-task classifier
In this section we will restate the assumptions and results of Lemma 3.1 for the main-task classifier (instead of the probing classifier) and show that the same results will hold.
Assm 3.1 remains the same since it is made on the latent-representation being disentangled and frozen/non-trainable.
Next, parallel to Assm 3.2 show that even when main-task feature is 100% predictive of main-task and a linearly separable, the trained main-task classifier will also use the concept-causal features.
Formally,
Assumption B.1 (main-task feature Linear Separability) .
The main-task features ( z m ) of the latent representation ( z ) for every point are linearly separable/fully predictive for the main-task labels y m , i.e y
im · (ˆ ϵ m · z im + b m ) > 0 for all datapoints ( x i , y im ) for some ˆ ϵ m ∈ R d p .
For the case of zero-centered latent space, we have b m = 0 .
Next similar to Assm 3.3, we define the spurious correlation between main-task and concept label: a function using only z p may also be able to classify correctly on some non-empty subset of points w.r.t.
main-task label ( y m ).
Assumption B.2 (Main-Task Spurious Correlation) .
For a subset of training points S ⊂ D m , maintask label y m is linearly separable using z p
i.e y
i m · ( ˆ ϵ p · z i p + b p ) > 0 for some ˆ ϵ p ∈ R d p and b p ∈ R .
For the case of zero-centered latent space we have b p = 0 .
Next we rephrase Lemma 3.1 which shows that for only a few special points if the concept-features z p are linearly-separable w.r.t.
to main task classifier y m (Assm B.2), then the main-task classifier c m ( z ) will use those features.
Lemma B.5 (Sufficient Condition for Main-task Classifier) .
Let the latent representation be frozen and disentangled
such that z = [ z m , z p ] (Assm 3.1), where main-task-features z m be fully predictive (Assm B.1).
Let c ∗ m ( z ) =
w m · z m be the desired/clean linear main-task classifier trained using max-margin objective (§B.1) which only uses z m for its prediction.
Let z p be the spurious feature s.t.
for the margin points of c ∗ m ( z )
, z p be linearly-separable w.r.t.
task label y m (Assm B.2).
Then, assuming the latent space is centered around 0 , the main-task classifier trained using max-margin objective will be of form
c m ( z ) =
w m · z m + w p · z p where w p ̸ = 0 .
The proof of Lemma B.5 is identical to Lemma 3.1 and is provided in §B.2.
C Null-Space Removal Failure: Setup and Proof of Theorem 3. C.1 Null-Space Setup
As described in §3, the given main-task classifier has an encoder h :
X → Z mapping the input X to latent representation Z .
Post that, the main-task classifier c m : Z → Y m is used to predict the main-task label
y
im from latent representation
z i for every input x i .
Given this (pre) trained main-task classifier the goal of a post-hoc removal method is to remove any undesired/sensitive/spurious concept from the latent representation Z without retraining the encoder h or main-task classifier
c m ( z ) .
The null space method [34, 14], henceforth referred as INLP , is one such post-hoc removal method which removes a concept from latent space by projecting the latent space to a subspace that is not discriminative of that attribute.
First, it estimates the subspace in the latent space discriminative of the concept we want to remove by training a probing classifier c p ( z ) → y p , where y p is the concept label.
[34] used a linear probing classifier ( c p ( z )) to ensure that the any linear classifier cannot recover the removed concept from modified latent representation z ′ and hence the main task classifier ( c m ( z ′ ) ), which is also a linear layer, become invariant to removed attribute.
Let linear probing classifier c p ( z ) be parametrized by a matrix W , and null-space of matrix W is defined as space N ( W )
= { z | W z = 0 } .
Give the basis vectors for the N ( W ) we can construct the projection matrix P N ( W ) such that W ( P N ( W ) z ) = 0 for all z .
This projection matrix is defined as the guarding operator g :=
P N ( W ) (estimated by c p ( z ) ), when applied on the z will remove the features which are discriminative of undesired concept from z .
For the setting when Y p is binary we have:
[Equation]
where I is the identity matrix and ˆ w is the unit vector in the direction of parameters of classifier c p ( z ) ([34]).
Also, the authors recommend running this removal step for multiple iterations to ensure the unwanted concept is removed completely.
Thus after S steps of removal, the final guarding function is:
[Equation] [Equation]
) where P i N ( W ) is the projection matrix at i th removal step.
Amnesic Probing ([14]) builds upon this idea for testing whether concept is being used by a given pre-trained classifier or not.
The core idea is to remove the concept we want to test from the latent representation.
If the prediction of the given classifier is influenced by this removal then the concept was being used by the given classifier otherwise not.
C.2 Null-Space Removal Failure : Proof of Theorem 3.
Theorem 3.2.
Let c m ( z ) be a pre-trained main-task classifier where the latent representation z
= [ z m , z p ] satisfies Assm 3.1 and 3.2.
Let c p ( z ) be the probing classifier used by INLP to remove the unwanted features z p from the latent representation.
Under Assm 3.3, Lemma 3.1 is satisfied for the probing classifier c p ( z ) such that c p ( z )
=
w p · z p + w m · z m and w m ̸ = 0 .
Then,
Damage in the first step of INLP.
The first step of INLP will corrupt the main-task features and this corruption is non-invertible with subsequent projection steps of INLP.
(a) Mixing:
If w p ̸ = 0 , the main task z m and concept-causal features z p will get mixed such that z
i (1) =
[ g ( z im , z ip ) , f ( z ip , z im )]
̸ =
[ g ( z im ) , f ( z ip )] .
Thus, the latent representation is no
longer disentangled and removal of concept-causal features will also lead to removal of main task features.
(b) Removal: If w p = 0 , then the first projection step of linear-INLP will do opposite of what is intended, i.e., damage the main task features z m (in case z m ∈ R , it will completely remove z m ) but have no effect on the concept-causal features z p .
Removal in the long term: The L2-norm of the latent representation z decreases with every projection step as long as the parameters of probing classifier at a step ( w k ) does not lie completely in the space spanned by parameters of previous probing classifiers, i.e., span( w 1 , . . .
, w k − 1 ), z i (0) and z i (0) in direction of w k is not trivially zero.
Thus, after sufficiently many steps, INLP can destroy all information in the representation, z i ( ∞ ) =
The proof of Theorem 3.2 proceeds in following steps:
First using Lemma 3.1 we show that even under very favourable conditions probing classifier will not be clean i.e will also use other features in addition to concept-causal feature for it’s prediction.
Then, for the more likely case when probing classifier uses both main-task and concept-causal feature, we show that after first step of null-space projection (INLP), both the main-task features and concept-causal features get mixed .
Next, for the extreme case when probing classifier uses only main-task feature, the first step of INLP will do opposite of what is intended.
It will damage the main-task feature but will have no effect on the concept-causal feature which we wanted to remove from latent space representation.
In addition, we also show that the damage or mixing of latent space after first step of INLP projection cannot be corrected in subsequent step since the projection operation is non-invertible.
Next, we show that the projection operation is lossy, i.e removes the norm of latent representation under some conditions.
Hence after sufficient steps, INLP could destroy all the information in latent representation.
Proof of Theorem 3.2.
First Claim (1a).
Let c p ( z )
=
w p z p + w m z m be the linear probing classifier trained to predict the concept label y p from the latent representation z .
Since all the assumptions of
Lemma 3.1 are satisfied for the probing classifier c p ( z ) , it is spurious using , i.e., w m ̸ = 0 and for the claim 1(a)
we have w p ̸ = 0 .
Since the concept label y p is binary, the projection matrix for the first step of INLP removal is defined as P 1 N ( W ) =
I − ˆ w ˆ w T where ˆ w T
=
[ ˆ w m , ˆ w p ] , ˆ w m and ˆ w p are the unit norm parameters of c p ( z ) i.e w m and w p respectively.
On applying this projection on the latent space representation z i we get new projected representation
z i (1) s.t.:
[Equation]
ˆ
[Equation]
Next we will show that main task features and probing features get mixed after projection.
To do so, we first show that g ( z im , z ip ) ̸
= g
( z im ) for some function g i.e projected main task features z
i (1) m =
g ( z i m , z i p ) are not independent of probing features
z i p .
From Eq.
69, we have:
[Equation]
Since w p ̸ = 0 and w m ̸ = 0 , we have ( ˆ w p · z ip ) ˆ w m ̸ = 0 .
Thus g ( z im , z ip ) is not independent of z ip .
Next, we will show that f ( z im , z ip ) ̸
= f ( z ip ) for some function f i.e projected probing feature
z i (1) p = f
( z im , z ip ) is not independent of the main task feature z
im .
From Eq.
69, we have: z i
[Equation]
Since w p ̸ = 0 and w m ̸ = 0 , we have ( ˆ w m · z im ) ˆ w p ̸ = 0 .
Thus f ( z im , z ip ) is not independent of z im .
Hence both concept-feature z p and the main-task feature z m got mixed after the first step of projection.
Next, we will show that this mixing of main task and concept-causal feature cannot be corrected in subsequent steps of null-space projection.
Formally, the following Lemma C.1 proves that the above projection matrix ( P 1 N ( W ) ) which resulted in mixing of features is non-invertible.
The subsequent steps of INLP applies projection transformation which can be combined into one single matrix
P > 1 N ( W ) =
(cid:81) j> 1 P jN ( W ) .
In order for mixing to be reversed, we need P > 1 N ( W ) × P 1 N ( W )
=
I , thus we need P > 1 N ( W ) =
( P 1 N ( W ) ) − 1 which is not possible from Lemma C.1.
Hence the mixing of maintask feature and the concept-causal feature which happened after first step of projection cannot be corrected in the subsequent steps of INLP thus completing the first claim of our proof.
Lemma C.1.
The projection matrix P jN ( W ) at any projection step of INLP is non invertible.
Proof of Lemma C.1.
The projection matrix for binary target case is defined as P :
= P jN ( W ) =
I − A where A = ˆ w ˆ w T be a n × n matrix and w is the parameter vector of the probing classifier c p ( z ) trained at j th -step of INLP.
We can see that A is a symmetric matrix.
Every symmetric matrix is diagonalizable (Equation W.9 in [12]), hence we can write A = Q Λ Q T , where Q is a some orthonormal matrix such that QQ T
=
I and Λ = diag ( λ 1 , . .
. , λ n ) be a n × n diagonal matrix where the diagonal entries ( λ 1 . . .
λ n ) are the eigen-values of A .
Since QQ T
=
I we can write
P =
[Equation]
Q T = Q
[Equation]
I − Λ) Q T .
Next, for the projection matrix P to be invertible
P − 1 should exist.
We have:
[Equation]
So projection matrix is only invertible when ( I − Λ) is invertible.
We will show next that ( I − Λ) is not invertible thus completing our proof.
We have I − Λ = diag (1 − λ 1 , . . .
, 1 − λ n ) , hence:
[Equation]
Now, if one of the eigen values of A is 1 , then the diagonal matrix ( I − Λ) is not invertible.
If one of the eigen values of A is 1 , then there exist an eigen-vector x such that A x = ˆ w ˆ w T × x = 1 × x .
The vector x = ˆ w is the eigen vector of A with eigen-value 1 :
A ˆ w = ˆ w ˆ w T × ˆ w = 1 × ˆ w since ˆ w T × ˆ w = 1 as it is a unit vector.
Hence the projection matrix is not invertible.
First Claim (1b).
For a probing classifier of form c (1) p ( z )
=
w p · z p + w m · z m for the first step of
INLP projection —denoted by superscript (1)— trained to predict concept label y p and Assm 3.1,3.
and 3.3 of Lemma 3.1 is satisfied then we have w m ̸ =
i.e main task feature z m will be used by probing classifier along with the concept feature z p .
For this second case, we are given that w p =
i.e probing classifier will not use concept feature at all.
This is only possible when the main-task feature is fully predictive of the concept label i.e Assm 3.3 is satisfied for all the points in the dataset, otherwise optimal probing classifier will use the concept-causal feature to achieve better margin and accuracy.
Moreover, even though Assm 3.3 is satisfied, in order to have w p = 0 , margin achieved by probing classifier, using only main task feature, should be bigger than any other classifier i.e using both main-task feature and probing feature or probing feature alone.
Thus, it is very unlikely that optimal probing classifier will have w p = 0 .
Having said this, even in the case when we have w p = 0 , we show that first projection step of
INLP will do something unintended, i.e., damage the main-task features while having no effect on concept-causal features which we intended to remove.
First we will show that main-task features will get damaged.
From Eq.
73 we have:
[Equation]
Since w m ̸ = 0 , we see that projected main-task feature z i (1) m ̸
=
z im .
In case z m ∈ R , we have ˆ w m =
ˆ z
im , thus ( ˆ w m · z im ) ˆ w m =
z im .
Consequently, z i (1) m =
z im −
z
im = 0 .
Thus, first projection step of INLP leads to complete removal of main-task feature z m when z m ∈ R .
Also, from
Lemma C.1, this projection step is non-invertible and hence the main-task feature cannot be recovered back in subsequent projection step.
Next, we will show the the first projection step has no effect on the concept-causal features which we wanted to remove in first place.
From Eq.
76, we have:
[Equation] [Equation]
[Equation]
Thus the first step of projection have no effect on the concept-causal feature we wanted to remove.
In the next step of projection, if we again have w p = 0 , then this same case will repeat.
Otherwise if
Assm 3.3 still holds for main-task feature for the margin points of optimal probing classifier c ∗ (2) p ( z ) for this second step of projection, then we will have both w m ̸ = 0 and w p ̸ = 0 and the first case of this theorem will apply.
Second Claim.
Now for proving the second statement, we will make use of the following lemma.
The proof of the lemma is given below the proof of this theorem.
Lemma C.2.
After every projection step of INLP, the norm of every latent representation z i decreases, i.e., ∥ z i ( k ) ∥ <
∥ z i ( k − 1) ∥ for step k and k − 1 , if (1) z i ( k − 1) ̸ = 0 , (2) z i (0)ˆ w k ̸ = 0 and (3) the parameters of probing classifier in step “ k ” i.e ˆ w k don’t lie in the space spanned by parameters of previous probing classifier, span( ˆ w 1 , . . .
, ˆ w k − 1 ).
Next we will show that, starting from first step, at every k th -step of projection either we will have z i ( k ) = 0 or the norm will decrease after projection.
Once we reach a step when z i ( k ) = 0 , then after every subsequent projection we will have z
i
k +1)
=
= ⇒ ∥
z i ( k +1) ∥
= 0 since:
[Equation]
(85) where P N ( w k ) is the projection matrix at step "k".
Since ∥·∥ ≥ 0 and ∥ z i ( k ) ∥ is decreasing with every stey, thus with large number of z i ( ∞ ) → 0 .
We will start with first step of projection.
In the second statement of this Theorem 3.2, we are given that z i (0) is not trivially zero in direction of w
i.e z i (0) w 0 ̸ = 0 (satisfying Assm(2) of above
Lemma C.2).
We are also given that z
i
0) ̸
= 0 (satisfying Assm(1) of above lemma) and
since this is the first step of projection Assm(3) of above Lemma C.2) is also satisfied.
Thus from Lemma C.2, we have ∥ z i (1) ∥ <
∥ z i (0) ∥ .
Now, either z i (1) = 0 , which will imply that ∥
z i (1) ∥ = 0 and will remain 0 for all subsequent step (from Eq. 85).
Otherwise if z i (1) ̸ = 0 , it satisfies the Assm(1) of
Lemma C.2, for next step of projection.
Since Assm (2) and (3) are already satisfied, again we will have ∥ z i (2) ∥ <
∥ z i (1) ∥ and the same idea will repeat eventually making z i ( k ) =
at some step-k, thus completing our proof.
Proof of Lemma C.2.
After ( k − 1) -steps of INLP let the latent space representation
z i be denoted as z i ( k − 1) .
Let ˆ w k be the parameters of classifier c p ( z k − 1 ) trained to predict the concept label y p which we want to remove at step k .
Then prior to the projection step in the k th iteration of the INLP, we can write z i ( k − 1) as:
[Equation]
where B = { ˆ w k , N ( ˆ w k ) } is the basis set and N ( ˆ w k ) is the null-space of ˆ w k .
The parameter ˆ w k in this new basis is:
[Equation]
[Equation]
where I ˆ w k is identity matrix with dimension d ˆ w k ×
d
ˆ w k .
Now, in the new basis when we project the z k − 1 to the null space of ˆ w i ( k ) we have:
[Equation]
Thus the norm of ∥ z i ( k ) ∥
= (cid:113) ∥
z i ( k − 1) N ( ˆ w k ) ∥ + 0 is less that ∥
z k − 1 ∥ = (cid:113)
[Equation]
∥
[Equation]
if z i ( k − 1) ˆ w k ̸ = 0 .
Next we will show that z i ( k − 1) ˆ w k cannot be zero.
From assumption (2) in C.2 z i (0) w k ̸ =
i.e z i (0) w k is not trivially zero in the given latent representation
z i (0) before any projection from INLP, thus z i ( k − 1)
ˆ w k is not trivially zero from beginning.
Also, from Eq. 92, we observe that at any step “ k ”
INLP removes the part of the representation from z i ( k − 1) which is in the direction of ˆ w k i.e. z i ( k − 1) ˆ w k .
Consequently, a sequence of removal steps with parameters ˆ w 1 , . . .
, ˆ w k − 1 will remove the part of z which lies in the span( ˆ w 1 , . . .
, ˆ w k − 1 ).
Thus z i ( k − 1)
ˆ w k
=
if ˆ w k lies in the span of parameters of previous classifier i.e span( ˆ w 1 , . .
, ˆ w k − 1 ) which violates the assumption (3) in Lemma C.2.
Thus z i ( k − 1)
ˆ w k is neither trivially zero from the beginning nor it could have been removed in the previous steps of projection as long as the assumption in Lemma C.2 is satisfied, which completes our proof of the lemma.
Remark.
The following lemma from [34] tells us some of the sufficient conditions when the parameters of the probing classifier at current iteration will not be same as previous step.
Lemma C.3 (Lemma A.1 from [34]) .
If the concept-probing classifier is being trained using SGD (stochastic gradient descent) and the loss function is convex, then parameters of the probing classifier at step k , w k , are orthogonal to parameters at step k − 1 , w k − 1 .
We conjecture that Lemma C.3 will be true for any loss since after k − 1 steps of projection, the component of z in the direction of span( w 1 , . . .
, w k − 1 ) will be removed.
Hence the concept-probing classifier at step k should be orthogonal to span( w 1 , . . .
, w k − 1 ) in order to have non-random guess accuracy on probing task.
D Adversarial Removal: Setup and Proof D.1 Adversarial Setup
As described in §3.3, let h :
X → z be an encoder mapping the input x
i to latent representation z i .
The main task classifier c m : Z → Y m is applied on top of z i to predict the main task label y
im for every input x i .
As described in §3.3, the goal of adversarial-removal method, henceforth referred as AR, is to remove any undesired/sensitive/spurious concept from the latent representation
Z .
Once the concept is removed from latent representation any (main-task) classifier using the latent representation Z will not be able to use them [16, 49, 13].
These methods jointly train the main-task classifier
c m ( z ) and the probing classifier c p : Z → Y p .
The probing classifier is adversarially trained to predict the concept label y ip from latent representation
z i .
Hence, AR methods optimize the following two objectives simultaneously:
[Equation]
Here L ( · ) is the loss function which estimates the error given the ground truth y m /y
p and corresponding prediction c m ( z ) /c
p ( z ) .
The desired solution and the same time a valid equilibrium point of the above min-max objective is an encoder h such that it removes all the features from latent space useful for prediction of y p while keeping intact other features causally-derived from the main-task prediction.
In practice, the optimization to above combined objective is performed using gradient-reversal (GRL) layer ([16]).
It introduces an additional layer g λ between the latent representation h ( z ) and the adversarial classifier c p ( z ) .
The g λ layer acts as an identity layer (i.e., has no effect) during the forward pass but scales the gradient by ( − λ ) when back-propagating it during the backward pass.
Thus resulting combined objective is:
[Equation]
Setup for theoretical result: As stated in Theorem 3.3, for our theoretical result showing the failure mode of adversarial removal, we assume that the encoder is divided into two sub-parts.
The first encoder i.e h 1 : X → Z is frozen (non-trainable) and maps the input x i to intermediate latent representation z i which is frozen and disentangled (Assm 3.1).
The second encoder h 2 : Z → ζ is linear transformation mapping the intermediate latent representation
z
i to final latent representation ζ i and is trainable.
On top of this final latent representation ζ
i , we train the main task classifier
c m ( ζ i ) and probing classifier c p ( ζ i ) .
Thus the training objective from Eq.
93 and 94 becomes:
[Equation]
D.2 Adversarial Proof
For detailed discussion of adversarial training objective and specific setup for our theoretical result refer §D.1.
Assumption D.1 (Strength of Correlation) .
Let ˆ w p ∈
R
d p be the unit vector s.t.
z p is linearly separable for concept-label y p (see Assm 3.2).
Then, concept-causal feature z
Mp of any margin point z M of c m ( h ∗ 2 ( z )) is more predictive of main-task than concept-causal feature z P p of any margin-point z P of c p ( h ∗ 2 ( z )) for probing task by a factor of | β | ∈ R where | β | is norm of parameter probing
classifier c p
i.e y m ( ˆ w p · z Mp ) >
| β | y p ( ˆ w p · z Pp ) .
Theorem 3.3.
Let the latent representation z satisfy Assm 3.1 and be frozen, h 2 ( z ) be a linear transformation over Z s.t.
h 2 : Z → ζ , the main-task classifier be c m ( ζ ) =
w c m · ζ , and the adversarial classifier be c p ( ζ )
=
w c p · ζ .
Let all the assumptions of Lemma B.5 be satisfied for main-classifier c m ( · ) when using Z directly as input and Assm 3.2 be satisfied on Z w.r.t.
the adversarial task.
Let h ∗ 2 ( z ) be the desired encoder which is successful in removing z p from ζ .
Then there exists an undesired/incorrect encoder h 2 ( z ) s.t.
h 2 ( z ) is dependent on z p and the main-task classifier
c m ( h 2 ( z )) has bigger margin than c m ( h ∗ 2 ( z )) and has,
Accuracy ( c p ( h 2 ( z )) , y p ) =
Accuracy ( c p ( h ∗ 2 ( z )) , y p ) when adversarial probing classifier c p ( · ) is trained using any learning objective like max-margin or cross-entropy loss.
Thus, the undesired encoder h 2 ( z ) is indistinguishable from desired encoder h ∗ 2 ( z ) in terms of adversarial task prediction accuracy but better for main-task in terms of max-margin objective.
L h 2 (cid:0) c m ( h 2 ( z )) , c p ( h 2 ( z )) (cid:1) <
L h 2 (cid:0) c m ( h ∗ 2 ( z )) , c p ( h ∗ 2 ( z )) (cid:1) additionally under Assm 3.4, D.
and given concept-causal feature z
Mp of any margin point z M of c m ( h ∗ 2 ( z )) is more predictive of main-task than z Pp of any margin-point z P of c p ( h ∗ 2 ( z )) for probing task (see Assm D.1).
Here
L h 2 ( · )
=
L ( c m ( · ) , y m ) − L ( c p ( · ) , y p ) is the combined adversarial loss w.r.t.
to h 2 and L ( · ) is the max-margin loss for a classifier (see §D.1).
Thus undesired encoder h 2 ( z ) is preferable over desired encoder h ∗ 2 ( z ) for both main and combined adversarial objective.
Proof of Theorem 3.3.
Let the main classifier be of form
c m ( ζ ) =
w c m · ζ where w c m and ζ are d ζ dimensional vectors.
Since both parameters w c m and ζ are learnable, for ease of exposition we constrain w c m to be [1 , 0 , . . .
This constraint on w c m is w.l.o.g.
since w c m makes the prediction for main-task by projecting ζ into one specific direction to get a scalar ( w c m · ζ ) .
We constrain that direction to be the first dimension of ζ .
Since the encoder h 2 ( z ) :
z → ζ is trainable it could learn to encode the scalar ( w c m · ζ ) in the first dimension of ζ .
Thus, effectively a single dimension of the representation ζ encodes the main-task information.
Thus the main classifier is effectively of form c m ( ζ ) =
w (0) c m × ζ (0) =
1 × ζ (0) where w (0)
c m and ζ (0) are the first element of w c m and ζ respectively and w (0)
c m = 1 .
We can now write the goal of the adversarial method as removing the information of z p from ζ (0) , because the other dimensions are not used by the main classifier.
Thus the adversarial classifier can be written effectively as c p ( ζ ) =
β × ζ (0) where β ∈ R is a trainable parameter.
Since both the main and adversarial classifier are using only ζ (0) , the second encoder, with a slight abuse of notation, can be simplified as ζ := ζ (0) :
=
h 2 ( z ) =
w m · z m + w p · z p , where ζ ∈ R and w m and w p are the weights that determine the first dimension of ζ .
Also, let the desired (correct) second encoder which is successful in removing the concept-causal feature z p from ζ be ζ ∗ :
= ζ ∗ (0) :
= h ∗ 2 ( z ) =
w ∗ m · z m .
Thus using Eq.
96 and 97, our overall objective for adversarial removal method becomes:
[Equation]
First claim.
The ideal main-classifier with desired encoder can be written as, c m ( ζ ∗ )
=
× h ∗ 2 ( z ) =
w ∗ m · z m .
Therefore, it can be trained using same MM-Denominator formulation of max-margin objective and would satisfy the constraint in Eq. 3:
[Equation]
for all the points x i with latent representation z i and m ( · ) is the numerator of the distance of point from the decision boundary of classifier (Eq. 1).
However, the main-task classifier which does not use the desired encoder is of the form, c m ( ζ ) =
× h 2 ( z ) =
w m · z m + w p · z p .
Since this main-task classifier is trained using max-margin objective by MM-Denominator formulation, it would satisfy the constraint in Eq. 3:
[Equation]
Since in our case, main-task classifier is same as the encoder i.e c m ( ζ ) =
× h 2 ( z ) =
w m · z m + w p · z p , and the latent representation z satisfies the Assm 3.1, B.1 and B.2, from Lemma B.5 the main-task classifier is spurious-using i.e z p ̸ = 0 .
Hence there exists an undesired/incorrect encoder h 2 ( z ) such that the main classifier c m ( ζ )
=
h 2 ( z ) has bigger margin than c m ( ζ ∗ )
=
h ∗ 2 ( z ) .
Next we show that the accuracy of the adversarial classifier remains the same irrespective of whether the desired ( h ∗ 2 ( z ) ) or undesired encoder h 2 ( z ) is used.
The accuracy of the adversarial classifier c p ( ζ ) =
β × ζ , using the desired/correct encoder ζ = h ∗ 2 ( z ) is given by:
[Equation]
N
[Equation]
where 1 ( · ) is an indicator function which takes the value 1 if the argument is true otherwise 0 , and sign ( γ )
=
+1 if γ ≥ 0 and − 1 otherwise.
Combining Eq.
100 and Eq.
101, since y im ∈ {− 1 , 1 } , we see that whenever h 2 ( z i ) >
1 we also have h ∗ 2 ( z i ) > 1 and similarly whenever h 2 ( z i ) <
− 1 , we have h ∗ 2 ( z i ) < − 1 .
Thus,
[Equation] [Equation]
From Eq. 103, h ∗ 2 ( z i ) and h 2 ( z i ) has the same sign for every input z
i =
⇒ sign ( β · h ∗ 2 ( z i ))
= sign ( β · h 2 ( z i )) .
Thus we can replace h ∗ 2 ( z i ) with h 2 ( z i ) in the above equation and we have: [Equation] p
[Equation]
, y
[Equation]
thus completing the first part our proof.
Second claim.
Since we are training both main-task and probing classifier with max-margin objective (see MM-Numerator version at Eq. 5), we can effectively write the adversarial objective (from 98 and 99) as:
[Equation]
where m c m ( h 2 ( z )) and m c p
h 2 ( z )) are the numerator of margin of a point (Eq. 1).
Next our goal is to show that the desired encoder h ∗ 2 is not an equilibrium point of the above adversarial objective.
To do so, we will create an undesired/incorrect encoder h 2 ( z ) by perturbing h ∗ 2 by small amount and showing that the combined encoder objective E ( h 2 ) >
E ( h ∗ 2 ) (Eq. 105) irrespective of choice of β chosen by the probing objective P ( β ) (Eq. 104).
Construction of the undesired/incorrect encoder.
We have h ∗ 2 ( z ) =
∥
w m ∥ ( ˆ w ∗ m · z m ) .
We will perturb this desired encoder by parameterizing with α ∈ [0 , 1) s.t.:
[Equation]
where ˆ ϵ p ∈ R d p is a unit vector.
The clean main-task classifier is defined as c ∗ m ( h ∗ 2 ( z ))
=
h ∗ 2 ( z ) .
The main-task classifier c m when using the incorrect encoder takes form c m ( h 2 ( z ))
=
h 2 ( z ) .
As stated in theorem statement, all the assumption of Lemma B.5 is satisfied.
Since Assm B.2 (one of the assumption of Lemma B.5) is satisfied, there exist an unit-vector in R dp such that concept-causal feature of margin point of main-task classifier using encoder h ∗ 2 is linearly separable w.r.t main-task label.
Let ˆ ϵ p in our constructed undesired encoder h α 2 (Eq. 106) be set to that unit vector such that:
[Equation]
107) where z Mp is the concept-causal feature of margin point z M of the main-task classifier when using encoder h ∗ 2 .
Now since all the assumption of Lemma B.5 is satisfied, the margin of main-task classifier when using undesired encoder h
α 2 ( z ) is bigger than when desired encoder h ∗ 2 is used for some α ∈ ( α 1 lb , 1) . Consequently we have:
[Equation]
108) where z M is the margin point of c m ( h ∗ 2 ) .
Since Assm 3.2 is satisfied, we have a fully predictive concept-causal feature z p for prediction of adversarial label y p such that for some unit vector ˆ w p ∈ R d p we have:
[Equation]
Next, since Assm 3.4 is also satisfied for this second part of theorem, we have y ip =
y
im for every margin point of the desired/correct main-task classifier using the desired/correct encoder h ∗ 2 ( z ) .
Thus we can assign ˆ ϵ p :
= ˆ w p which satisfies the inequality in Eq.
Thus our incorrect encoder h α 2 ( z ) take the following form:
[Equation]
Note that when α = 1 , we recover back the correct encoder h ∗ 2 .
Thus to perturb the h ∗ 2 , we set α close but less than 1.
Showing h ∗ 2 is not the equilibrium point.
From Eq.
105, we want to show that for some α ∈ [0 , 1) , the undesired encoder h
α 2 has bigger combined objective than desired encoder h ∗ 2 .
Since the combined adversarial objective for encoder h 2 ( z ) ( E ( h 2 ( z )) in Eq. 105) is evaluated on margin point the margin point of main-task and probing task classifier.
We use the following lemma to show that for small perturbing of optimal encoder ( α → 1 ), the margin point of main-task classifier and probing classifier when using perturbed encoder h α 2 is same or a subset of margin points when using desired encoder h ∗ 2 .
The proof of the lemma below is given after proof of current theorem.
Lemma D.1.
There exist an α 2 lb ∈ [0 , 1) s.t.
when α > α 2 lb we have: (i) margin point of probing classifier when using perturbed encoder h α 2 is same or is a subset of margin points when using desired encoder h ∗ 2 .
(ii)
margin point of main-task classifier when using perturbed encoder h α 2 is same or is a subset of margin points when using desired encoder h ∗ 2 . Let z M ∗ be one of the margin point of main-task classifier c m and z P ∗ be one of the margin point of probing classifier c p when using the correct encoder h ∗ 2 . Let z M α be one of the margin point of main-task classifier c m and z P α be one of the margin point of probing classifier c p when using the perturbed encoder h
α 2 . Thus we want to show that for all ( z M ∗ , z P ∗ , z M α , z P α ) tuple there exists some α close but less than 1 s.t.
we have: [Equation] [Equation]
[Equation] [Equation]
For β < 0 . From Lemma D.1, both z P α and z P ∗ are the margin point of probing classifier when using the desired encoder h ∗ 2 .
Thus we have:
[Equation]
From Eq.
109 we have y P α p ( ˆ w p · z P α p ) > 0 .
Since β < 0 , we have β ∥ w ∗ m ∥ y P α p ( ˆ w p · z P α p ) < 0 .
Adding this to LHS of above equation we get:
[Equation]
[Equation] [Equation]
From Eq.
119 the RHS of Eq.
113 is less than zero.
Also, from Eq. 108, for α ∈ ( α 1 lb , 1) we have m
[Equation]
where value of α 1 lb is given by Lemma B.5.
Thus the LHS of Eq.
113 is greater than 0 .
Thus the inequality in 113 is always satisfied when β < 0 and α ∈ ( max { α lb 1 , α 2 lb } , 1) . The constraint α > α 1 lb is enforced by Lemma B.5 when constructing the perturbed encoder and α > α 2 lb is enforced by Lemma D.1 which ensures z P α is also a margin point of probing classifier when using desired encoder h ∗ 2 . Thus we have shown that h ∗ 2 is not the equilibrium point since there exist perturbed undesired encoder h α 2 such that the combined encoder objective is greater in Eq.
and consequently the optimizer will try to move away from/change h ∗ 2 .
For β > 0 .
Next we have to show that we want to show that there exist α ∈ [0 , 1) , we have Eq.satisfied.
Thus we solve for allowed values of α :
[Equation]
From the second statement from Lemma D.1, z M α and z M ∗ both are margin point of main-task classifier using the desired encoder h ∗ 2 .
Thus we have m c m ( h ∗ 2 ( z M α ))
= m
c m ( h ∗ 2 ( z M ∗ ))
=
⇒ y M
α m h ∗ 2 ( z M α ) =
y M ∗ m h
[Equation]
2 ( z M ∗ ) =
⇒ y
[Equation]
α
[Equation]
z
[Equation]
= y M
[Equation]
( w ∗ m · z M ∗ m ) .
Substituting this observation in LHS of Eq. 120 we get m [Equation] z M
[Equation]
where for ease of exposition we have defined M :
= M α . Now again for RHS of Eq. 120, from Lemma D.1, z P α and z P ∗ both are margin point of probing classifier using the desired encoder h ∗ 2 .
Thus we have m
[Equation]
p
[Equation]
h [Equation] z [Equation]
P
[Equation]
m ) .
Substituting this observation in RHS of Eq.
120 we get m
[Equation]
p
[Equation]
where for ease of exposition we have defined P := P α .
Substituting RHS (Eq. 129) and LHS (Eq. 125) back in Eq.
120 and rearranging we get: [Equation] Now [Equation]
since Assm B.1 is satisfied
[Equation]
the main task feature
z
Mm is linearly separable w.r.t main-task label y Mm .
Thus we have y Mm ( ˆ w ∗ m · z Mm ) > 0 .
Case 1: Main-task feature is not fully predictive of probing label ( ∃ z s.t.
y p
( ˆ w ∗ m · z m ) < 0 ).
Since main-task feature is not fully predictive of the probing label y p , there will be some points which will be misclassified (will be on the opposite side of decision boundary) when probing classifier uses desired encoder c p ( h ∗ 2 ( z ))
=
βh ∗ 2 ( z )
=
w ∗ m · z m .
Thus margin for those points will be negative and one of them will be the margin point z P of the probing classifier.
Thus m c p ( h ∗ 2 ( z P )) =
y Pp β ( ˆ w ∗ m · z Pm ) < 0 .
Then the term (cid:0) y Mm ( ˆ w ∗ m · z Mm ) − y Pp β ( ˆ w ∗ m · z Pm ) (cid:1) > 0 in the above Eq.
Thus, rewriting the above equation we have:
[Equation]
[Equation]
Next, from Lemma D.1 both z M =
z M α and z P =
z P α are also the margin point of main-task and probing classifier respectively when the classifiers use the desired encoder h
α 2 . Then, since Assm D. is satisfied the numerator in LHS of above equation (cid:0)
y Mm ( ˆ w p · z Mp ) − y Pp β ( ˆ w p · z Pp ) (cid:1) > 0 .
Thus whole LHS in above equation is greater than zero.
Setting the LHS to γ ( z M , z P ) gives us:
[Equation]
[Equation]
Since γ 2 ( z M , z P ) > 0 , α 3 lb ( z M , z P )) < 1 .
Let α 3 lb = max
( z M , z P ) ( α 3 lb ( z M , z P ))) < 1 , which gives us the tight lower-bound on α such that Eq.
113 is satisfied for any pair of margin point z M and z P .
Case 2: Main-task is fully predictive of probing label.
( ∀ z , y p ( ˆ w ∗ m · z m ) > 0 ).
Since Assm B.
(from Lemma B.5) is satisfied, we have main-task feature fully predictive of main-task label i.e y m ( ˆ w ∗ m · z m ) > 0 for all z .
Thus for this case y m ( ˆ w ∗ m · z m ) > 0 and y p
( ˆ w ∗ m · z m ) > 0 =
⇒ y m
= y p for all z .
Also, for this case there will be no misclassified points for the probing classifier when using the desired encoder h ∗ 2 .
Thus the margin point for both main and the probing classifier is same i.e z M = z P .
Since Assm D.1 is satisfied, y m = y p for all z , y p ( ˆ w p · z ) > 0 for all z from Assm 3.
and z P =
z M we have:
[Equation] [Equation]
[Equation]
Thus, in this case the RHS in Eq. 130, could be simplified to :
y Mm ( ˆ w ∗ m · z Mm ) − y Pp β ( ˆ w ∗ m · z Pm ) =
y Mm ( ˆ w ∗ m · z Mm ) − βy Mm ( ˆ w ∗ m · z Mm ) =
(1 − β )
y Mm ( ˆ w ∗ m · z Mm ) > 0 since 0 < β < 1 from above Eq.and y p ( ˆ w ∗ m · z M m ) > 0 from Assm B.1.
Thus we can rewrite Eq.
130 as:
[Equation]
Again, from Lemma D.1 both z M =
z M α and z P =
z P α are also the margin point of main-task and probing classifier respectively when the classifiers use the desired encoder h
α 2 . Thus from Assm D.1, we have numerator of LHS in above equation greater than 0, thus we can follow the same steps from Eq.
132 to 136 to get the α 3 lb for this case.
So far we have three lower bounds on α needed for this proof, so lets define α lb = max { α 1 lb , α 2 lb , α 3 lb } , where α 1 lb is the enforced by Lemma B.5 on undesired encoder h
α 2 construction, α 2 lb is enforced by
Lemma D.1 and α 3 lb is enforced by Eq.
Thus, when α ∈ ( α lb , 1] we have bigger combined objective (Eq. 105) for h
α 2 than h ∗ 2 .
Thus, we can always perturb the desired encoder h
α 2 by choosing α ∈ ( α lb , 1] close but less than 1 to create h α 2 which will have better combined encoder objective.
Hence any optimizer will prefer to change the desired encoder h ∗ 2 , hence it is not an equilibrium solution to overall adversarial objective.
Proof of Lemma D.1. First we will prove the statement for the probing classifier. Let z M be one of the margin point of probing classifier when using the desired encoder h ∗ 2 and let z R be any other (non-margin) points. Then we have to show that margin-point of probing classifier when using perturbed encoder h α 2 cannot be z R .
This will imply that the margin points for probing classifier when using h α 2 has to be same or subset of margin points when using h ∗ 2 .
Since norm of parameters of both c p ( h α 2 ( z )) = βh
α 2 ( z ) and c p ( h ∗ 2 ( z )) =
βh ∗ 2 ( z ) is same and margin-point of a classifier is the point which have minimum margin we have to show that m c p ( h α 2 ) ( z R ) > m c p ( h α 2 ) ( z M ) .
We have:
[Equation]
[Equation]
[Equation]
Thus we have find an α s.t.:
[Equation]
z
[Equation]
Rearranging we get:
[Equation]
Since z M is the margin point of the probing classifier when using h ∗ 2 , we have m c p ( h ∗ 2 ) ( z R ) >
m c p
( h ∗ 2 ) ( z M ) .
Now, if y p (cid:110) β ( ˆ w p · z Rp ) − β ( ˆ w p · z Mp ) (cid:111) ≤ 0 , then above equation is trivially satisfied for all values of α ∈ (0 , 1) , since RHS of above equation is greater than 0 and LHS is less than 0.
For the case when y p (cid:110) β ( ˆ w p · z Rp ) − β ( ˆ w p · z Mp ) (cid:111) > 0 we need:
[Equation]
[Equation]
[Equation]
Since we have γ > 0 = ⇒ α plb ( z M , z p ) <
Lets define α plb := max ( z M , z p ) ( α plb ( z M , z p )) < 1 , which gives the tightest lower bound on α s.t.
when α ∈ ( α plb , 1) , the margin point of probing classifier when using the perturbed encoder is same or is a subset of margin point when using desired encoder h ∗ 2 .
This completes the first part of the proof.
Next we prove the second part of this lemma for main-task classifier.
Let z M be one of the margin point of the margin points of main-task classifier classifier when using the desired encoder h ∗and let z R be any other (non-margin) point.
Then we have to show that margin-point of maintask classifier when using perturbed encoder h α 2 cannot be z R .
Since norm of parameter of both c m ( h α 2 ( z ))
=
h
α 2 ( z ) and c m ( h ∗ 2 ( z )) =
h ∗ 2 ( z ) is same and margin-point of a classifier is the point which have minimum margin we have to show that m c m ( h α 2 ) ( z R ) >
m c m ( h α 2 ) ( z M ) .
We have:
[Equation]
[Equation]
Thus we have find an α s.t.:
[Equation]
Rearranging we get:
[Equation]
Since z M is the margin point of the probing classifier when using h ∗ 2 , we have m c m ( h ∗ 2 ) ( z R ) >
m c m ( h ∗ 2 ) ( z M ) .
If we notice, apart from y p being set to y m and β being set to 1 , the above equation is identical to Eq.
Since our argument (in Eq.
143 to 147) to derive the allowed value of α doesn’t depend on y p and β , we could follow the same argument to get a lower bound α mlb s.t.
the main-task classifier has the same margin point when using either perturbed or desired encoder.
Lets define α 2 lb = max { α plb , α mlb } .
Thus when α ∈ ( α 2 lb , 1) , both the statement of of this lemma is satisfied thus completing our proof.
E Experimental Setup E.1 Dataset Description
As described in §4, we demonstrate the failure of Null-Space Removal (§4.2) and Adversarial
Removal (§4.3) in removing the undesired concept from the latent representation on three real world datasets: MultiNLI [47], Twitter-PAN
[33] and Twitter-AAE [7]; and a synthetic dataset,
Synthetic-Text.
The detailed generation and evaluation strategies for each dataset are given below.
MultiNLI Dataset.
In MultiNLI dataset, given two sentences—premise and hypothesis—the main-task is to predict whether the hypothesis entails , contradicts or is neutral to the premise.
As described in §4, we simplify it to a binary task of predicting whether a hypothesis contradicts the premise.
The binary main-task label, y m = 1 when a given hypothesis contradicts the premise otherwise it is -1.
That is, we relabel the MNLI dataset by assigning label y m
= 1 to examples with contradiction labels and y m = − 1 to the example with neutral or entailment label.
It has been reported that the contradiction label is spuriously correlated with the negation words like nobody, no, never and nothing [18].
Thus, we created a ‘negation’ concept denoting the presence of these words in the hypothesis of a given (hypothesis, premise) pair.
The concept-label y p = 1 when the negation concept is present in the hypothesis otherwise it is − 1 .
The standard MultiNLI dataset 1 has approximately 90% of data points in the training set, 5% as publicly available development set and the rest of 5% in a separate held out validation set accessible through online competition leader-board not accessible to the public.
Thus, we create our own train and test split by subsamping 10 k examples from the initial training set, converting it into binary contradiction vs. non-contradiction labels, labelling the negation-concept label, and splitting them into 80-20 train and test split.
For pre-training a clean classifier that does not use the spurious-concept, we create a special training set following the method described in §E.2.
For evaluating the robustness of both null-space and adversarial removal methods, we create multiple datasets with different predictive-correlation as described in §E.3 .
Twitter-PAN16 Dataset.
In Twitter-PAN16 dataset [33], following [13], given a tweet, the main task is to predict whether it contains a mention to another user or not.
The dataset contains manually annotated binary Gender information (i.e Male or Female) of 436 Twitter user with atleast 1k tweets each.
The Gender annotation was done by assessing the name and photograph of the LinkedIn profile of each user [13].
The unclear cases were discarded in this process.
We consider “Gender” as a sensitive concept that should not be used for main-task prediction.
The dataset contains 160k tweets tweets for training and 10k tweets for test.
We merged the full dataset, subsampled 10k examples, and created a 80-20 train and test split.
For pre-training a clean classifier, we create a special training set following the method described in §E.2.
To generate datasets with different predictive correlation, we follow the method from E.3.
The dataset is acquired and processed using the code 2 made available by the [13].
According to Twitter’s policy the one has to download tweets from a personal account using
Twitter Academic Research access and cannot be released to public or used for commercial purpose.
We also adhere to this policy and don’t release any data to public or use it elsewhere.
AAE Dataset.
In Twitter-AAE dataset [7], again following [13], the main-task is to predict a binary sentiment (Positive or Negative) from a given tweet.
The dataset contain 59.2 million tweets by 2.8 million users.
Each tweet is associated with “race” information of the user which is labelled based on both words in the tweet and geo-location of the user.
We consider “race” as the sensitive concept which should not be used for the main task of sentiment prediction.
We use the
AAE (African America English) and SAE (Standard American English) as a proxy for non-Hispanic blacks and non-Hispanic whites automatically labelled using code made available by [13].
Again, we subsampled 10k examples with 80-20 split from the dataset and followed the method described in §E.2 and E.3 to generate a clean dataset for pre-training a clean classifier and datasets with different predictive correlation respectively.
The dataset is made publicly available online 3 only for research-purpose.
Synthetic Dataset.
To accurately evaluate the whether a classifier is using the spurious concept or not, we introduce a Synthetic-Text dataset where it is possible to change the text input based on the change in concept (thus implementing Def 2.1).
The main-task is to predict whether a sentence contains a numbered word (e.g. one, fifteen etc) or not, and the spurious concept is the length of the sentence which is correlated with the main task label.
To create a sentence with numbered words, we randomly sample 10 words from the following set and combine them to form the sentence
Numbered Words = one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen, sixteen, seventeen, eighteen, twenty, thirty, forty, fifty, sixty, seventy, eighty, ninety, hundred, thousand
Otherwise, a sentence is created by adding 10 non-numbered words randomly sampled from the following set.
Non-Numbered Words = nice, device, try, picture, signature, trailer, harry, potter, malfoy, john, switch, taste, glove, balloon, dog, horse, switch, watch, sun, cloud, river, town, cow, shadow, pencil, eraser
Next, we introduce the spurious concept (length) by increasing the length of the sentences which contain numbered words.
We do so by adding a special word “pad” 10 times.
In our experiments, we use 1k examples created using the above method and create 80-20 split for train and test set.
Again, we follow the method described in §E.2 and E.3 to generate a clean dataset for training a clean classifier and to generate datasets with different predictive correlations respectively.
E.2 Creating a “clean” dataset with no spurious correlation with main-label
To construct a new dataset with no spurious correlation between the main-task and the concept label, we subsample only those examples from the the given dataset which have a fixed value of the spurious-concept label ( y p ) .
Thus, if we train main-task classifier using this dataset, it cannot use the spurious-concept since they are not discriminative of the main task label [37].
In MultiNLI dataset, we select only those examples which have no negation words in the sentence for creating a clean dataset.
Similarly, for Twitter-PAN16 dataset, we only select those examples which have gender label y p = − 1 (Female) in the processed dataset.
And for Twitter-AAE dataset, we only select those examples which have non-Hispanic whites race label.
E.3 Creating datasets with spurious correlated main and concept label
In our experimental setup, both the main-task label ( y m ) and concept label ( y p ) are binary ( −or 1 ).
This creates 2 × 2 subgroups for each combination of ( y m , y p ).
In MultiNLI dataset, the contradiction label ( y m = 1) is correlated with the presence of negation words y p
= 1 , this implies that the not-contradiction label y m = − 1 is also correlated with absence of negation words in the sentence y p = − 1 .
Thus, the input example with ( y m = 1 , y p = 1) and ( y m = − 1 , y p = − 1) form the majority group, henceforth referred as S maj while groups ( y m = 1 , y p = − 1) and ( y m = − 1 , y p = 1) forms the minority group S min .
To evaluate the robustness of the removal methods, we create multiple datasets with different predictive correlation ( κ ) between the two labels y m and y p where κ = P ( y m · y p ) > 0 as defined in §4.
In other words, to create a dataset with a particular predictive correlation κ , we vary the size of S maj and S min .
More precisely, the predictive correlation can be equivalently defined in terms of the size of the these groups as:
[Equation]
Similarly for Twitter-PAN16, Twitter-AAE and Synthetic-Text dataset, we create datasets with different levels of spurious correlation between y m and y p by creating the S maj and S min to have the desired predictive correlation ( κ ).
E.4 Encoder for real datasets
For all the experiments on real datasets in §4 we used RoBERTa as default encoder h .
In §F, we report the results when using BERT instead of RoBERTa as input encoder.
RoBERTa We use the Hugging Face[48] transformers implementation of RoBERTa[27] robertabase model, starting with pretrained weights for encoding the text-input to latent representation.
We use default tokenizer and model configuration in our experiment.
BERT We use the Hugging Face[48] transformers implementation of BERT[11] bert-base-uncased model, starting with pretrained weights for encoding the text-input to latent representation.
We use default tokenizer and model configuration in our experiment.
For both BERT and RoBERTa, the parameters of the encoder were fine-tuned as a part of training the main-task classifier for null-space removal and then frozen.
For adversarial removal, the encoder, main-task classifier and the adversarial probing classifier are trained jointly.
For both BERT and
RoBERTa, we use the pooled output ( [ CLS ] token for BERT) from the the model, as the latent representation and is given to main-task and probing classifier.
E.5 Encoder for synthetic Dataset
nBOW: neural Bag of Word.
For Synthetic-Text dataset, we use sum of pretrained-GloVe embedding[32] of the words in the sentence to encode the sentence into latent representation.
We used
Gensim [38] library for acquiring the 100-dimensional GloVe embedding ( glove-wiki-gigaword-100 ).
Throughout all our experiments, the word embedding was not trained.
Post encoding, the latent representation were further passed through hidden layers consisting of a linear transformation layer followed by relu [2] non-linearity.
We will specify how many such hidden layers were used when discussing specific experiments in §F.
The hidden layer dimensions were fixed to 50 dimensional space.
E.6 Null-Space Removal Experiment Setup
For null-space removal (INLP) experiment on both real and synthetic dataset the following procedure is followed:
Pretraining Phase: A clean pretrained main-task classifier is trained using the clean dataset obtained by method described in §E.2.
This is to ensure that the main-task classifier does not use the spurious feature, so that the INLP method doesn’t have any effect on the main-task classifier.
The main-task classifier is a linear-transformation on the latent-representation provided by encoder followed by softmax layer for prediction.
Both the encoder and main-task classifier is fine-tuned during this process.
Removal Phase: Both the encoder and main-task classifier is frozen (made non-trainable).
Next, a probing classifier is trained from the latent representation of the encoder (refer §E. and E.5 for more details about encoder).
The probing classifier is also a linear transformation layer followed by softmax layer for prediction
The main-task classifier and encoder in the pretraining phase and the probing classifier in the removal phase is trained using cross-entropy loss for both real and synthetic datasets.
For the real dataset, a fixed learning rate of 1 × 10 − 5 is used when RoBERTa is used as encoder and 5 × 10 − 5 when using
BERT as encoder.
For synthetic experiments, a fixed learning rate of 5 × 10 − 3 is used when training both the nBOW encoder and main-task classifier in the pretraining stage and probing classifier in removal stage.
E.7 Adversarial Removal Experiment Setup
For adversarial removal (AR) experiment, for both real and synthetic datasets, first the input text is encoded to latent representation using the encoder (§E.4 and E.5).
Then for the main-task classifier a linear transformation layer followed by softmax layer is applied for the main-task prediction.
The same latent representation output from encoder is given to the probing classifier which is a separate linear transformation layer followed by softmax layer.
All components of the model, encoder, main-task classifier and probing classifier are trained using the following modified objective from
Eq.
[Equation]
where h is the encoder, c m is the main task classifier,
c p is the probing classifier, g − 1 is the gradient reversal layer with fixed reversal strength of − 1 .
The first term in the objective is for training the main task classifier and the second term is the adversarial objective for training the probing classifier using gradient reversal method [16, 13] .
The hyperparameter λ controls the strength of the adversarial objetive.
In our experiment we very λ ∈ { 0 . 00001 , 0 . 0001 , 0 .
When describing the experimental results in §F.3 we choose the λ which performs the best for all datasets with different predictive correlation κ in removing the undesired concept from the latent representation.
E.8 Metrics Description
For simplicity, in all our experiments we assume that both the main and the correlated attribute label are binary.
We measure the degree of spuriousness using the following two metrics:
Spuriousness Score:
As defined in §3.4, this metric help us quantify, how much a classifier is using the spurious feature.
∆ Probability: In Synthetic-Text dataset as described in E.1, we have the ability to change the input corresponding to the change in concept label (thus implementing Def 2.1) .
Thus we could measure if the main-classifier is using the spurious-concept by changing the concept in the input and measuring the corresponding change in the main-task classifier’s prediction probability.
Higher the change in prediction probability higher the main-task classifier is dependent on spurious-concept.
E.9 Compute and Resources
We used an internal cluster of Nvidia P40 and P100 GPUs for all our experiment.
Each experiment setting was run on three random seed and mean results with variance are reported in all the experiment.
F Additional Results F.1 Probing classifier Quality
Fig.
5 shows different failure modes of the probing classifier.
In Fig.
5a and 5b, a clean main-task classifier which doesn’t use the concept feature is trained on Synthetic-Text and MultiNLI dataset respectively using the method described in §E.2.
Thus the latent representation doesn’t have the concept feature.
Then, to test the presence of concept-causal feature in the latent representation we train a probing classifier to predict concept-label.
The first row show the accuracy of the probing classifier for testing the presence of concept in latent space.
When κ = 0 .
5 i.e no correlation between the main-task and the concept label, the probing accuracy is approximately 50% which correctly shows the absence of the concept-causal feature in the latent representation.
The accuracy increases as the correlation κ between the main and concept-causal feature increases in dataset.
This shows that even when concept-causal feature is not present in the latent representation, probing classifier will still claim presence of concept-causal feature if any correlated feature (main-task feature in this case) is present in the latent space.
In Fig.
5c, the latent space contains the concept-causal feature as shown by accuracy of approximately 94.5% when κ = 0 .
But as κ increases the probing classifier’s accuracy increases in the presence of correlated main-task feature which falsely increases the confidence of presence of the concept-causal feature.
The second row shows the spuriousness-score of conceptprobing classifier is increasing as the correlation between the main-task and concept-causal feature increases which implies that the probing classifier is using relatively large amount of correlated main-task feature for concept-label prediction in all settings.
[Figure]
when the latent representation doesn’t contain the probing concept-causal feature, the probing classifier is still has >50% accuracy when other correlated feature is present.
The accuracy increases as the correlation κ between the probing concept-causal feature and other correlated feature increases.
The first row Fig.
5c shows, that presence of correlated feature could increase the probing classifier’s accuracy thus increasing the confidence in the presence of concept-causal feature in latent representation. The second row of all the figures shows that the probing classifier is getting more spurious as the κ increases thus implying that the probing classifier is using some other correlated feature than concept-causal feature. For more discussion see §F.1. F.2 Extended Null-Space Removal Results
Fig.
6 and 7, shows the failure mode of null-space removal (INLP) in the real dataset when using
RoBERTa and BERT as encoder respectively.
Different columns of the figure are for three different real datasets — MultiNLI, Twitter-PAN16, and Twitter-AAE respectively.
The x-axis from step 8are different INLP removal steps.
The y-axis shows different metrics to evaluate the main-task and probing classifier.
Different colored lines shows the spurious correlation ( κ ) in the probing dataset used by INLP for removal of spurious-concept.
The pretrained classifier is clean, i.e., does not use the spurious concept-causal feature; hence INLP shouldn’t have any effect on main-classifier when removing concept-causal feature from the latent space.
The first row shows that as the INLP iteration progresses, the norm of latent representation, which is being cleaned of concept-causal feature, decreases.
This indicates that some features are being removed.
However, the results are against our expectation from the second statement of Theorem 3.2, which states that the norm of classifier will tend to zero as the INLP removal progresses.
The possible reason is that, from Theorem 3.2 the norm of latent representation will go to zero when the the latent representation only contains the spurious concept-causal feature and the other features correlated to it.
But, the encoder representation could have other features which are not correlated with concept-label and hence not removed.
Since, the pretrained classifier given for INLP was clean (using method described in §E.2), we do not expect the INLP to have any effect on the main-task classifier.
The second row in Fig.
6 and 7 shows that the main-classifier accuracy drops to random guess i.e 50% except for the case when probing dataset have κ = 0 .
5 i.e no correlation between the main and concept label.
Thus INLP method corrupted a clean-classifier and made it useless.
The reason behind this could be observed from the fourth and fifth row.
The fourth row show the accuracy of the probing classifier before the projection step.
We can see that at step 8 on x-axis κ = 0 .
5 , the probing classifier correctly has accuracy of 50% showing that the concept-causal feature is not present in the latent representation.
But for other value of κ , the probing classifier accuracy is proportional to value of κ implying that the probing classifier is using the main-task feature for its prediction.
Hence at the time of removal, it removes the main-task feature which leads drop in the main-task accuracy.
This can also be verified form the last row of Fig.
6 and 7, which shows that the spuriousness score of probing classifier is high; thus it is using the main-task feature for its prediction.
We observe similar results for Synthetic-Text dataset when using INLP in Fig.
For all the INLP experiment on Synthetic-Text dataset, there were no hidden layers after the nBOW encoder (see §E.5). [Table] L a t e n t [Table]
p . N [Table] r
[Table]
cc
[Table]
r
[Table]
c
[Table]
( A ll )
[Table]
i n -
[Table]
p u r i o u
[Table]
n
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
F.3 Extended Adversarial Removal Results
Adversarial removal failure in real-world datasets.
Fig.
9 shows the failure mode of adversarial removal AR on real-world datasets.
In x-axis we vary the predictive correlation κ between the main and the concept-label in different datasets and measure the performance of AR on different metrics on the y-axis.
The second row shows the spuriousness score of the main-task classifier after AR as we vary κ on the x-axis.
When using RoBERTa as the encoder, the orange curve in second row shows the spuriousness score of the main-task classifier when trained using the ERM loss.
The spuriousness score describes how much unwanted concept-causal feature the main-task classifier is using.
The blue curve shows that the AR method reduces the spuriousness of main-task though cannot completely remove it.
The reason for this failure can be attributed to probing classifier.
Even when AR has successfully removed the unwanted concept feature, the accuracy of concept-probing classifier will be proportion to κ due to presence of correlated main-task feature in the latent space.
This can be seen in the third row of the of Fig.
Thus we cannot be sure if the unwanted concept-causal feature has been completely removed from the latent space or just became noisy enough to have accuracy proportional to κ after AR converges.
In Fig.
9, for each dataset and encoder, we manually choose the hyperparameter described λ described in §E.7 which reduces the spuriousness score most for the main-task classifier while not hampering the main-task classifier accuracy.
In Fig.
10, we show the trend in spuriousness score is similar for all choices of hyperparameter λ in our search.
No value of λ is able to completely reduce the spuriousness score to zero.
Adversarial removal makes a classifier unfair.
Fig.
11 shows that when the adversarial classifier is initialized with a clean main-task classifier which doesn’t use unwanted-concept causal features, it makes matters worse by making the main-task classifier use the unwanted-concept feature.
For the Synthetic-Text dataset, since the word embedding are non-trainable, one single hidden layer is applied after the nBOW encoder so that AR methods could remove the unwanted-concept feature from the new latent representation.
F.4 Synthetic-Text dataset Ablations
Adversarial Removal Failure in Synthetic-Text dataset: Figure 12 shows the failure of AR on the synthetic dataset as we vary the noise in the main-task label and unwanted concept-label.
For the experiment, since the words embedding are non-trainable, one single hidden layer is applied after the nBOW encoder so that AR methods could remove the unwanted-concept feature from the new latent representation.
Dropout Regularization Helps AR method: Continuing on observation from Fig. 13a, 13b and 13c shows the ∆ -Prob of the main-task classifier after we apply the AR on Synthetic-Text dataset and how they changes as we increase the dropout regularization.
As we increase the dropout (drate in the figure), the ∆ -Prob of the main classifier decreases showing that the regularization methods could help improve the the removal methods.
G Comparison between Spuriousness Score and ∆ Prob
In this section we compare the Spuriousness Score proposed in §3.4 for measuring a classifier’s use of a binary spurious feature with the ideal, ground-truth metric, ∆ Probability ( ∆ Prob for short) defined in §E.8.
∆ Prob measures the reliance on a spurious feature by changing the spurious feature in the input space (when possible) and measuring the change in the prediction probability of the given classifier.
Hence ∆ Prob is a direct and intuitive measure of spuriousness in a given classifier.
But changing the spurious feature is difficult in the input space for real-world data, thus we only evaluate this metric on the Synthetic-Text dataset.
To do so, we use the result from Fig.
12 that showed failure of the adversarial removal method on the Synthetic-Text dataset under various noise settings (refer §F.4 for details).
For the setting with noise n = 0 . 0 , both Spuriousness Score and ∆ Prob curve for Adversarial Removal (marked as
ADV in Fig. 12) are identical (close to 0 for all values of κ with mean = 0 .
0 and standard-deviation = 0 . 0 ).
For the other settings with non-zero noise, we compute the Pearson correlation between the Spuriousness score and ∆ Prob for the ADV curve.
As Table 1 shows, we observe high Pearson correlation of 0 . 83 and 0 .
95 for the noise setting, n = 0 .
1 and n = 0 .
3 respectively.
The third column in the table shows p-value ( < 0 . 05 ) assuming a null hypothesis that the two metrics are uncorrelated.
These results suggest that Spuriousness-Score can be a good approximation for the ideal ∆ Prob metric.
[Table] [Table] [Table]
[Figure]
case when RoBERTa was used as encoder (see Fig. 6) .
Different columns of the figure are for three different real datasets — MultiNLI, Twitter-PAN16, and Twitter-AAE respectively.
The x-axis from steps 8-26 are different INLP removal steps.
The y-axis shows different metrics to evaluate the main-task and probing classifier.
Different colored lines shows the spurious correlation ( κ ) in the probing dataset used by INLP for removal of spurious-concept.
The pretrained classifier is clean i.e. doesn’t uses the spurious concept-causal feature, hence INLP shouldn’t have any effect on main-classifier when removing concept-causal feature from the latent space.
Against our expectation, the second row shows that the main-task classifier’s accuracy is decreasing even when it is not using the concept-feature.
The main reason for this failure to learn a clean concept-probing classifier.
This can be verified from the last row which shows that the concept-probing classifier has high spuriousness score thus implying that it is using the main-task feature for concept label prediction and hence during removal step, wrongly removing the main-task feature which leads to drop in main-task accuracy.
For more discussion see §F.2.
[Table]
i o
[Table]
n
[Table]
u r
i
o
[Table]
n
[Table]
c
[Table]
r e
[Table]
M a i n - - P r o b a b ili t y
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
P r o b e - A cc u r a c y ( A ll ) =0.5=0.6=0.7=0.
=0.9=
[Figure] [Figure]
P r o b e - Sp u
r
i
o u s n e ss S
c
o r e
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
∆ -Prob increases in the middle of INLP showing that the main-classifier which was not using the concept initially (as in iteration 21), started using the sensitive concept because of INLP removal.
Thus stopping INLP prematurely could lead to a more unclean classifier than before whereas running INLP longer removes all the correlated feature and could make the classifier useless.
For more discussion see §F.2.
[Figure]
i
[Figure]
r A cc u r a c y
[Figure]
r e
[Figure]
r
[Figure]
b e
A cc u r a c y ( A ll )
[Figure]
[Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure] [Figure]
[Figure] [Figure]