Probing Classifiers are Unreliable for Concept Removal and Detection
Abstract
Neural network models trained on text data have been found to encode undesired linguistic or sensitive attributes in their representation.
Removing such attributes is non-trivial because of a complex relationship between the attribute, text input, and the learnt representation.
Recent work has proposed post-hoc and adversarial methods to remove such unwanted attributes from a model’s representation.
Through an extensive theoretical and empirical analysis, we show that these methods can be counter-productive: they are unable to remove the attributes entirely, and in the worst case may end up destroying all task-relevant features.
The reason is the methods’ reliance on a probing classifier as a proxy for the attribute which we prove is difficult to train correctly in presence of spurious correlation.
1. Introduction
Neural network classifiers built using text data have been shown to learn spuriously correlated features (Gururangan et al., 2018a; McCoy et al., 2019) or embed sensitive attributes like gender or race (Bolukbasi et al., 2016; Conneau et al., 2018) at their representation layer.
Classifiers that use such sensitive or spurious concepts (henceforth concepts ) raise concerns of model unfairness and affects out-ofdistribution generalization (Sagawa et al., 2020a; Arjovsky et al., 2020).
Because the classifiers are based on hard-tointerpret deep neural networks, removing the influence of these concepts is non-trivial.
Since many concepts cannot be modified at the input tokens level, removal methods that work at the representation layer have been proposed: posthoc removal (Bolukbasi et al., 2016; Xu et al., 2017) on a pre-trained model (e.g., null space projection (Ravfogel et al., 2020)), and adversarial removal (Ganin & Lempitsky, 2015; Elazar & Goldberg, 2018) by jointly training the main task classifier with an (adversarial) classifier for the concept.
In this paper, we theoretically show that both classes of methods can be counter-productive in real-world settings where the main-task label is correlated with the concept.
Examples include natural language inference (main-task) where the presence of negative words (spurious concept) may be correlated with the “contradicts” label.
Our key result is based on the observation that both these methods internally use an auxiliary (probing) classifier (Shi et al., 2016;
Adi et al., 2017) that aims to predict the spurious feature based on the representation learnt by the main classifier.
We show that the auxiliary classifier cannot be a reliable signal on whether the representation includes features that are causally derived from the concept.
As previous work has argued (Tsipras et al., 2019; Sagawa et al., 2020c; Arjovsky et al., 2020; Belinkov, 2022), if the representation features causally derived from the concept are not predictive enough, the probing classifier for the concept can be expected to rely on correlated features to obtain a higher accuracy.
However, we show a stronger result by considering a setting where there is no potential accuracy gain and
the concept’s features are easily learnable.
Namely, even when the concept’s causally-related features alone can provide 100% accuracy, the probing classifier may still learn non-zero weights for the main-task relevant features.
Based on this result, under some simplifying assumptions, we prove that both posthoc and adversarial training methods can fail to remove the undesired concept, remove useful task-relevant features in addition to the undesired concept, or do both.
As an extreme case, we show how post-hoc methods can lead to a random main-task classifier, removing all task-relevant information from the representation 1 .
Our contributions include:
• Theoretical analysis of null space and adversarial methods, showing they fail to remove an unwanted concept from a model’s representation, even under favorable conditions.
• Empirical results on four datasets showing that the nullspace and adversarial removal methods are unable to remove concept-related features fully or end up unnecessarily removing task-relevant features.
• A practical spuriousness score motivated by theory for evaluating the output of removal methods.
2. Concept removal can be counter-productive
For a classification task, let ( x i , y im ) ni
=1 be set of examples in the dataset D m , where x i ∈
X is the input feature and y
im ∈ Y m the label.
We call this as the main task and label y
im as the main task label.
The main task classifier can be written as c m ( h ( x )) where h : X → Z is an encoder mapping the input X to the latent representation Z and c m : Z → Y m is the classifier on top of the representation
Z .
Additionally, we are given labels for a spurious or sensitive concept, y ip ∈ Y p , and our goal is to ensure that the representation h ( x ) learnt by the main classifier does not include concept-causal features defined as:
Definition 2.1.
A feature Z j ∈ Z ( j th dimension of h ( x ) ) at the representation layer is causally derived from a concept ( concept-causal for short) if upon changing the value of the concept label, the corresponding change in the input’s value x will lead to a change in the feature’s
( Z j ) value.
The above definition of concept-causal feature requires assumption of some underlying causal mechanism generating the text which is often not available.
Hence these latent space based removal methods (Sec 2.1) use an auxiliary probing classifier to estimate the concept-causal feature present in the latent representation from the concept label.
2.1. Challenge with learning a clean probing classifier
Given z = h ( x ) and the concept label y p , the goal of probing task is to learn a classifier c p ( z ) such that it only uses the concept-causal features and the accuracy for y p is maximized.
If there are features in z outside concept-causal that improve the accuracy for the classifier, a classifier trained on standard losses such as cross-entropy or max-margin is expected to use those features too, as argued in the probing literature (Hewitt & Liang, 2019;
Belinkov, 2022).
In this section, however, we create a setup that is the most favorable for the classifier to use only concept-causal features—no accuracy gain on using features outside concept-causal, disentangled representation so that no further representation learning is needed, and concept labels that are linearly separable using concept-causal features—yet we find that a trained classifier uses non-concept-causal features.
Assumption 2.1 (Disentangled Frozen Latent Representation) .
The latent representation z is disentangled and is of form [ z m , z p ] , where z p ∈ R d p are the features causally derived from the concept label and z m ∈ R
d m are the features causally derived from the main task label along with rest of the features in latent representation.
Here d m and d p are the dimensions of z m and z p respectively.
Also the encoder h ( x ) which maps the input X to latent representation Z is frozen or non-trainable.
While it may be less surprising that a spurious feature is used if the invariant feature (concept-causal) does not give 100% accuracy, we show that spurious feature will be used even if the invariant feature is fully predictive of the label, following (Nagarajan et al., 2021).
We formalize it as,
Assumption 2.2 (concept-causal feature Linear Separability) .
The concept-features of the latent representation ( z p ) for every point are linearly separable/fully predictive for the concept labels y p , i.e y ip · (ˆ ϵ p · z ip + b p ) > 0 for all datapoints ( x i , y ip ) for some ˆ ϵ p ∈ R d p .
For the case of zero-centered latent space we have b p = 0 .
In addition, we assume that there is some correlation between the concept and main-task label: a function of z m can also classify correctly a non-empty subset of points.
Assumption 2.3 (Spurious Correlation) .
For a subset of training points S ⊂ D p , z m is linearly-separable with respect to concept label y p i.e y ip · (ˆ ϵ m · z
im + b m ) > 0 for some ˆ ϵ m ∈ R d m and b m ∈ R .
For the case of zero-centered latent space we have b m = 0 .
Now we state the key lemma: if non-concept-causal features z m are linearly separable w.r.t.
to concept label y p for only a few special points, then the probing classifier c p ( z ) will use those features.
As in (Nagarajan et al., 2021), for ease of exposition we use max-margin training loss.
Lemma 2.1.
Let the latent representation be frozen and disentangled
such that z
=
[ z m , z p ] (Assm 2.1), where concept-features z p be fully predictive (Assm 2.2).
Let c ∗ p ( z )
= w p · z p be the desired clean/purely-invariant linear classifier trained using max-margin objective (Suppl C.1) which only uses z p for its prediction.
Let z m be the spurious feature s.t.
for the margin points of c ∗ p ( z ) , z m be linearlyseparable w.r.t.
probing task label y p
(Assm 2.3).
Then, assuming the latent space is centered around 0 , the conceptprobing classifier trained using max-margin objective will be of form c p ( z )
=
w p · z p + w m · z m where w m ̸ = 0 .
For detailed proof see Suppl C.2.
In Suppl C.3 we prove that the assumptions for Lemma 2.1 are both sufficient and necessary for a classifier to use non-concept-causal features z m when z p is 1-dimensional.
Lemma 2.1 generalizes a result from (Nagarajan et al., 2021) by using fewer assumptions (we do not assume that z m to be binary, that z m and z p are conditionally independent given y m or y p , and do not assume monotonicity of classifier norm with dataset size).
We also present similar result for main-task classifier in Suppl C.4.
Sagawa et al.
(2020b) shows that subsampling the dataset help in learning clean classifier by breaking spurious correlation.
Note that even after subsampling as in Sagawa et al. (2020b)
Assm 2.3 could still be satisfied if it is satisfied in the original dataset, since margin points are few and may not be subsampled.
[Figure]
M m
[Figure]
2.2. Failure mode of Null-space removal (INLP)
The null space method (Ravfogel et al., 2020; Elazar et al., 2021), henceforth referred as INLP , first estimates the subspace in the latent space discriminative of the concept we want to remove by training a probing classifier c p ( z i )
→ y ip , where y ip is the concept label.
Then the projection is done on the null-space of this probing classifier which is expected to be non-discriminative of the concept.
(Ravfogel et al., 2020) used a linear probing classifier ( c p ( z )) to ensure that the any linear classifier cannot recover the removed concept from modified latent representation z ′ and hence the main task classifier ( c m ( z ′ ) ) also become invariant to removed attribute.
Details for the method are in Suppl D.1.
Below we state the failure of null-space removal under the assumptions of Lemma 2.1 (proof in Suppl D.2).
We use z i ( k ) to denote the representation
z i after k steps.
Theorem 2.2.
Let the latent representation z be disentangled (Assm 2.1),
c m ( z ) be the pre-trained main-task classifier, and c p ( z ) the probing classifier used by INLP to remove the unwanted feature z p from the latent representation.
Let the assumptions of Lemma 2.1 be true for the probing classifier c p ( z ) , trained using max-margin objective.
Mixing:
After the first projection step of linear-INLP
, the dimensions of z gets mixed such that z i (
) =
[ g ( z im , x ip ) , f ( z ip , z im )]
̸
=
[ z im
z ip
Also
, this mixing
is non-
invertible
with subsequent
projection steps
Thus, the latent space is no longer disentangled and
removal
of concept
-causal
features will
also
lead to
removal
of
task-specific
features.
Removal: The L2-norm of the latent representation
z
decreases with every projection step as long as the pa-
rameters
of probing classifier at a step ( w k
does not lie completely in the space spanned by parameters
of previous probing classifiers i.e. span( w 1 , . . .
, w k − 1 ).
Thus, after sufficiently many steps, INLP can destroy all information in the representation, z i ( ∞ ) =
2.3. Failure mode of adversarial removal methods
To remove the unwanted features z p from the latent representation, the adversarial removal method jointly trains the main classifier c m ( z ) and the probing classifier c p : z → y p by specifying c p ’s loss as an adversarial loss (Elazar & Goldberg, 2018) .
For details refer to Suppl E.1.
Since the encoder h : x → z mapping the input the latent representation z is frozen ( Assm 2.1), to allow training for the main-task classifier we introduce additional representation layers after it.
For simplicity in the proof, we assume a linear transformation to the latent representation h 2 : z → ζ .
After this, we have a linear main-task classifier c m : ζ → y m , as before.
The probing classifier c p : ζ → y p is trained adversarially to remove z p from latent representation ζ .
Thus, the goal of the adversarial method can be stated as removing the information of z p from ζ .
Let the main-task classifier satisfy the assumption of the generalized version of Lemma 2.1 (Suppl C.4).
In addition we need an assumption on the hard-to-classify margin points which says that main-task labels and adversarial concept label are correlated on margin point of clean main-task classifier.
Assumption 2.4 (Label Correlation of Margin Point) .
For the margin points of clean classifier for the main task, the adversarial-probing label y p and the main task labels y m are correlated, i.e w.l.o.g.
y im =
y ip for all margins points of clean main-task classifier.
Theorem 2.3.
Let the latent representation Z satisfy
Assm 2.1, h 2 ( z ) be a linear transformation s.t.
h 2 : Z → ζ , main classifier be c m ( ζ ) =
w c m · ζ and the adversarial
[Figure]
, INLP
[Figure]
Removal
[Figure]
M a i n - T a s k A cc u r a c y ( A ll )
Dataset: MultiNLI =0.5=0.6=0.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
cc u r
[Figure]
c y (
[Figure]
ll )
Dataset
[Figure]
NLP =0.5=0.
[Figure]
=0.
[Figure]
=0.8=0.9=
[Figure] [Figure]
40 Pretraining:
[Figure] [Figure]
[Figure]
M a i n - P r o b a b ili t y ( a ll )
Dataset: Synthetic-NLP =0.5=0.6=0.
=0.8=0.9=0.
[Figure]
classifier be c p ( ζ ) =
w c p · ζ .
Let all the assumptions of generalized Lemma 2.1 (Suppl C.4) be satisfied for mainclassifier c m ( · ) when using Z directly as input and Assm 2.
be satisfied on Z w.r.t.
adversarial-task.
Let h ∗ 2 ( z ) be the desired encoder which is successful in removing z p from ζ .
Then there exists an undesired/incorrect encoder h 2 ( z ) such that ζ =
h 2 ( z ) =
f ( z m , z p ) for some function “ f ” and the main classifier using it c m ( h 2 ( z )) has bigger margin than c m ( h ∗ 2 ( z )) and has,
Acc ( c p ( h 2 ( z )) , y p ) =
Acc ( c p ( h ∗ 2 ( z )) , y p ) when adversarial probing classifier c p ( · ) is trained using any learning objective.
Thus the undesired encoder h 2 ( z ) is indistinguishable from desired encoder h ∗ 2 ( x ) in terms of adversarial task prediction accuracy but better for mainprediction task in terms of max-margin objective.
L ( c p ( h 2 ( z )) , y p ) >
L ( c p ( h ∗ 2 ( z )) , y p ) under Assm 2.
and Assm 2.4 where L is either max-margin or crossentropy loss used for training c p ( · ) .
Thus undesired encoder h 2 ( z ) is preferable over desired encoder h ∗ 2 ( z ) for both
main
and adversarial task objective.
Spuriousness Score.
We next define a new metric which will help us access the quality of a given classifier.
To define spuriousness score over binary labels and concepts, we consider two subgroups, S min and S maj where S maj contains the dominant correlation exhibited in dataset while
S min contains the rest of the data.
Definition 2.2.
Given a dataset, D m,p = S min ∪ S maj containing binary task label and binary concept, let
Acc c m ( S min ) be the accuracy of the given main-task classifier ( c m ) on the minority group S min and Acc ∗ ( S min ) be the accuracy on the minority group of a clean classifier that does not use the unwanted spurious concept.
We define spuriousness score of “ c m ” as: ψ ( c
[Equation]
= (cid:12)(cid:12)(cid:12)
[Equation]
− Acc cm
[Equation]
Acc ∗
[Equation]
To estimate Acc ∗ ( S min ) , we subsample the dataset such that y p takes a single value in the sample and train the main classifier on it, as done by (Ravichander et al., 2021).
Thus, probing label y p no longer is correlated with the main-task label y m .
There can be other ways to estimate it, e.g., by reweighting the data or using the accuracy on
S maj .
However, we found that the former had high variance and the latter requires an equal-noise assumption such that
Acc ( S maj ) =
Acc ( S min ) on a clean main-task classifier.
3. Experimental Results
Dataset.
We demonstrate the failure of null-space (INLP) and adversarial removal (AR) on three real-world dataset –
MultiNLI (Williams et al., 2018), Twitter-PAN16 (Rangel et al., 2016) and Twitter-AAE (Blodgett et al., 2016).
For brevity, only show the results for MultiNLI dataset in this section (see Suppl G for complete results).
In MultiNLI dataset, given two sentences–premise and hypothesis–the main task is to predict whether hypothesis entails , contradicts or is neutral to premise.
We simplify to a binary task of predicting whether a hypothesis contradicts the premise or not.
Since negation words like nobody,no,never and nothing have been reported to be spuriously correlated with with the contradiction label (Gururangan et al., 2018b), we create a ‘negation’ concept denoting the presence of these words.
The goal is to remove the negation concept from the
NLI model’s representation.
In general the negation feature might be useful for the task and should not be removed completely.
But for simplicity here we consider complete removal of this feature.
Synthetic-Text.
To understand the reasons for failure, we introduce a Synthetic-Text dataset where it is possible to change the input text based on a change in concept (thus implementing Def. 2.1).
Here we can directly evaluate whether the concept is being used by the main-task classifier by intervening on the concept (adding or removing) and observing the change in prediction scores.
The main-task is to predict whether a sentence contains a numbered word (e.g., one, fifteen, etc. ).
We introduce spurious concept (length) by increasing the length of sentences that contain numbered words.
Predictive correlation.
To assess robustness of removal methods, we create multiple datasets with different predictive correlation between two labels y m and y p .
The predictive correlation ( κ ) measures how informative one variable is for predicting the other, κ =
Pr ( y m · y p >
[Figure] [Figure]
, where N is the size of dataset and 1 [ · ] is the indicator function which is 1 if the argument is true otherwise 0 .
Predictive-correlation lies in κ ∈ [0 . 5 , 1] where κ = 0 .
5 means no-correlation and κ = 1 means the attributes are completely correlated.
Results: Null Space Removal
In general, for any model given as input to INLP, it may be difficult to verify whether the removal method removed the correct features.
Hence, we construct a benchmark where the input classifier is clean , i.e., it does not use the concept at all.
We do so by training on a subset of data with the same value of the property, as in (Ravichander et al., 2021).
In Fig 2a, iteration 7-26 shows individual steps of null-space removal for MultiNLI dataset.
Since, the given pre-trained classifier was clean i.e not using the sensitive features, nullspace removal shouldn’t have any effect on it.
Colored lines show different datasets used by INLP with different level of predictive correlation κ .
We observe that for most value of κ , the main-task classifier’s accuracy eventually goes to 50% random guess accuracy implying that the main-task related attribute has been removed by the removal process, as predicted by Theorem 2.2.
Higher the value of correlation κ , faster the removal of main-task attribute happens.
To avoid the full collapse, a stopping criteria of INLP is to stop when the main-task classifier’s performance drops (Ravfogel et al., 2020).
In Fig 2c we measure the sensitivity of
Synthetic-Text main-task classifier w.r.t.
to sensitive concept by changing the feature in input sentences corresponding to sensitive concept and measuring the change in main-task classifier prediction probability ( ∆ Prob).
At lower iterations of INLP, the change in main-task output due to change in sensitive concept’s value, ∆ Prob, is higher than that of the input classifier.
For example, for κ = 0 . 8 , the main-task classifier’s performance drops for the first time at iteration 27, but it has a high ∆ Prob ≈ 25% as shown in Fig 2c.
Hence it is possible that stopping prematurely will lead us to a classifier which is more unfair or reliant to sensitive concept than it was before, consistent with the first statement in Theorem 2.2 stating that INLP will lead to mixing of features in latent space.
Results: Adversarial Removal We compare the standard
ERM training of the main classifier with adversarial-removal (AR) method over the same number of epochs (20).
For
MultiNLI dataset, Fig 3b shows the spuriousness score (Def 2.2) of ERM and AR classifiers as we vary the predictive correlation ( κ ) between the main-task label and sensitive concept label in the training dataset.
While the spuriousness score for classifier trained using AR (blue curve) is lower than that of ERM for all values of κ , it is substantially away from zero.
Thus, the AR method fails to completely remove the sensitive/spurious concept completely from the latent representation.
Fig 3c shows the possible reason using the sensitive concept probing classifier accuracy for ERM and AR.
The probe accuracy after adversarial training doesn’t decrease to 50% but stops at accuracy proportional to the predictive correlation κ .
This is expected since even if the
AR would have been successful in removing the sensitive feature, the main-task feature would still be predictive of sensitive concept label by κ due to the spurious correlation between them.
However, the converse is not true: an accuracy of κ does not imply that sensitive concept is fully removed.
The results substantiate the first statement of Theorem 2.3: given two representations where one ( desired ) does not have concept features while the other ( undesired ) contains the sensitive concept features, the undesired one may be better for the main task loss even as both may have the same probing accuracy.
While previous empirical work ((Elazar & Goldberg, 2018)) showed a similar limitation for adversarial methods, we extend it through theory and empirical experiments that measure the use of unwanted concept by classifier through the spuriousness score, not just its presence in the representation.
4. Conclusion and Discussion
Using both theoretical analysis and empirical results over real-world datasets, we presented a critique of recent work that claims to remove spurious or sensitive concepts from a text classifier.
Given the practical relevance of the removal problem, we also provided an spuriousness metric to evaluate any classifier on its use of unwanted concepts.
A. Broader Impact and Ethical Consideration
Removal of spurious or sensitive concepts is an important problem to ensure that machine learning classifiers generalize better to new data and are fair towards all groups.
We found multiple limitations with current removal methods and recommend caution against the use of these methods in practice.
B. Limitations of our work
One of the limitations of our theoretical work is assuming frozen or non-trainable latent representation which makes the analysis of task-classifier trained on top of them relatively easier.
We address this limitation in our empirical work where we do not make such assumptions.
Also, our work addresses failure modes of probing based removal on two popular methods, post-hoc removal (Null-Space Removal) and Adversarial Removal and we conjecture that similar failure could be expected for other methods in those classes.
It would be interesting to show that any concept removal method based on a probing classifier will fail in general.
C. Probing and Main Classifier Failure Proofs C.1. Notation and Setup: Max-margin Classifier
Assm 2.1 says encoder h : X → Z , mapping the input to latent representation is frozen/non-trainable.
Thus for every input x i in the dataset D , we have a corresponding latent representation z i which is fixed.
Also, the latent representation
Z is disentangled i.e z
=
[ z m , z p ]
where z m are the feature causally derived from main task label and z p are the features causally derived from the concept label.
Let c p ( z )
=
w p · z p + w m · z m be the linear probing classifier which we train using max-margin objective.
The hyperplane c p ( z ) = 0 is the decision boundary of this linear classifier.
The points which falls on one side of the decision boundary ( c p ( z ) > 0 ) are assigned one label (say positive label 1) and the rest are assigned another label (say negative label -1).
The margin M c p of this hypothesis function ( c p ( z )) is the distance of the closest latent representation from the decision boundary.
The points which are closest to the decision boundary is called the margin points.
The distance of a given latent representation z
i having class label
y i , where y i ∈ {− 1 , 1 } , from decision boundary is given by
[Equation]
Max-Margin (MM):
Then the max-margin classifier is trained by optimizing the following objective:
[Equation]
For ease of exposition we could convert this objective into multiple equivalent forms.
To do this we observe that scaling the parameters of c p ( z ) by a factor of γ i.e w → γ w and b → γb does not change the distance of the point from the decision boundary ( M c p ( z i ) ).
MM-Denominator Version: We can use this freedom of scaling the parameters to set m c p ( z i )
= 1 for the closest point of any given hypothesis function, thus all the data points will satisfy the constraint:
[Equation]
and giving us the final max-margin objective:
[Equation]
under the constraint m c p ( z i ) ≥ 1 corresponding to all the points in the dataset.
MM-Numerator Version: Alternatively, one can choose γ such that ∥ w ∥
= c where c ∈ R is some constant value.
The the modified objective becomes:
[Equation]
under constrain ∥ w ∥ = c which is usually set to 1 .
We will use different formulation in our proofs later based on the ease of exposition and give a clear indication when we do so.
One can refer to Chapter 7 , Section 7 .
1 of (Bishop, 2006) for furthur details about max-margin classifier.
C.2. Proof of Sufficient Condition: Lemma 2.1 and Lemma C.
Lemma 2.1.
Let the latent representation be frozen and disentangled
such that z
=
[ z m , z p ] (Assm 2.1), where conceptfeatures z p be fully predictive (Assm 2.2).
Let c ∗ p ( z )
= w p · z p be the desired clean/purely-invariant linear classifier trained using max-margin objective (Suppl C.1) which only uses z p for its prediction.
Let z m be the spurious feature s.t.
for the margin points of c ∗ p ( z ) ,
z m be linearly-separable w.r.t.
probing task label y p
(Assm 2.3).
Then, assuming the latent space is centered around 0 , the concept-probing classifier trained using max-margin objective will be of form c p ( z )
=
w p · z p + w m · z m where w m ̸ = 0 .
In this section we prove that, given the assumption in Lemma 2.1 is satisfied, they are sufficient for a concept probing classifier c p ( z ) to use the spuriously correlated main-task feature z m .
See Sec C.1 for detailed setup and max-margin training objective.
Also, we could use the same line of reasoning to prove similar result for the main-task classifier i.e. when condition in Lemma C.5 is satisfied, the main-task classifier will use the spurious concept-feature z p .
To keep the proof general for both the lemmas, we prove the result for a general classifier c ( z ) trained to predict a task label y .
Here the latent representation be of form z =
[ z inv , z sp ] where z inv is the feature which is causally-derived from/invariant to task-label y and z sp be the feature spurious to the task label y .
With respect to concept-probing classifier c p ( z ) in Lemma 2.1 z inv :=
z p and z sp :=
z m .
Similarly, for the main-task classifier in Lemma C.5, z inv :=
z m and z sp :=
z p .
For ease of exposition, we define two categories of classifiers based on which features they use:
Definition C.1 (Purely-Invariant Classifier) .
Given a linear classifier of form c ( z ) =
w inv z inv + w sp z sp + b is called "purely-invariant" if it does not use the spurious features z sp i.e., w sp = 0 .
Definition C.2 (Spurious-Using Classifier) .
Given a linear classifier of form c ( z ) =
w inv z inv + w sp z sp + b is called "spurious-using" if it uses the spurious features z sp i.e., both w sp ̸ = 0 and w inv ̸ = 0 .
Proof of Lemma 2.1 and C.5.
Let c inv ( z ) =
w inv · z inv be the clean /purely-invariant classifier trained using the maxmargin objective using the MM-Denominator formulation given in Eq 4 such that w inv ̸ = 0 .
The classifier c inv ( z ) is 100% predictive of the task labels y
(from Assm 2.2 for the probing task or Assm C.1 for the main-task).
Here the bias term b =since we assume the latent representation z is centered.
The norm of this classifier is ∥ w inv ∥ and the distance of each input latent representation ( z i ) with class label y
i ( y i ∈ {− 1 , 1 } ) from the decision boundary ( c inv ( z ) = 0 ) is given by:
[Equation]
[Equation]
Since we have used the MM-Denominator version of max-margin to train C inv ( z ) from Eq 3 we have m inv ( z i )
=
for the margin-points and greater than 1 for rest of the points.
Next we will construct a new classifier parameterized by α ∈ [0 , 1] by perturbing the clean/purely-invariant classifier c inv ( z ) such that:
[Equation]
) where ˆ ϵ sp ∈ R d sp is a unit vector in spurious subspace of features, d sp is the dimension of the spurious subspace.
We observe that the norm of this perturbed classifier c
α
( x ) is ∥ w inv ∥ , which is same as the clean/purely-invariant classifier c inv ( x ) .
Thus the distance of any input z i with class label y i from the decision boundary of this perturbed classifier, c α ( z ) = 0 is given by:
[Equation]
The perturbed classifier will be spurious-using i.e use the spurious feature z sp when α ∈ [0 , 1) since ( ∥ w inv ∥ √ 1 − α 2 ) ̸
=for these setting of α .
Thus to show that there exist a spurious-using classifier which has a margin greater than the margin of the purely-invariant classifier, we need to prove that there exist an α ∈ [0 , 1) such that c α ( z ) has bigger margin than c inv ( z ) i.e. min z M α ( z ) >
min z M inv ( z ) .
Since norm of parameters of both the classifier is same, substituting the expression of
M α and M inv from Eq 6 and 8, we need to show m α ( z i ) > 1 for all z i .
We have:
[Equation]
Let S my denote the set of margin-points of purely-invariant classifier c inv ( z ) with class label y
having m inv ( z ) = 1 and S ry contain rest of non-margin points having m inv ( z ) > 1 with the label y .
Here “m” stands for margin-point in superscript of S and “r” stands for rest of point with label y .
Below we show that for all non-margin points we can always choose α ∈ [0 , 1) such that m α ( z i ) > 1 .
But we need spurious feature of margin point to be linearly separable for having m α ( z i ) > 1 , given by Assm 2.3 for probing task or C.2 for main-task.
Margin Points
For the margin-points in latent space, z m ∈ S
my we have m inv ( z m )
=
and we need to show that there exist α ∈ [0 , 1) such that m α ( z m ) > 1 for all z m ∈ S
my .
From Eq 10 we have:
[Equation] [Equation]
From Assm 2.3 for probing task or C.2 for the main-task, we know that spurious-feature z sp of margin-points are linearlyseperable w.r.t to task label y .
Thus there exist an unit vector ˆ ϵ sp ∈ R d sp such that y (cid:0) ˆ ϵ sp · z msp (cid:1) > 0 for all z m ∈ S my .
Also since α ∈ [0 , 1) , we have (1 − α ) > 0 and ( ∥ w inv ∥√ 1 − α 2 ) > 0 since ∥ w inv ∥ > 0 .
If Assm 2.3 or C.2 (corresponding to the task) wouldn’t have been satisfied then the above equation would have been inconsistent since right hand side is > 0 but left hand side wouldn’t have been always greater than 0.
Then achieving m α ( z m ) > 1 wouldn’t have been possible.
Let β :=
( y (cid:0) ˆ ϵ sp · z sp (cid:1) ) then squaring both sides and cancelling (1 − α ) since α ∈ [0 , 1) we get:
[Equation]
[Equation] [Equation]
After substituting back the value of β and rearranging we get:
[Equation]
[Equation]
Since ∥ w inv ∥ 2 · (cid:18) y (cid:0)
ˆ ϵ sp · x msp (cid:1)(cid:19) 2 > 0 , the right hand side of above equation
α 1 lb < 1 which sets a new lower bound on allowed value of α .
Thus there exist an α ∈ [0 , 1) such that m p ( x m ) > 1 for all x m ∈ S my .
Non-Margin Points
For the non-margin points z r ∈ S ry in the latent space we have m inv ( z r ) > 1 .
Let γ :=
min
z r ∈S ry (cid:0) m inv ( z r ) (cid:1) thus we also have γ > 1 .
Let α ̸ =
and we choose α such that:
[Equation]
Substituting the value of γ we get:
[Equation] [Equation]
Since γ > 1 , thus right hand side in above equation α 2 lb < 1 which sets a new lower bound on allowed values of α .
Since m inv ( z r ) > γ > 1 α for all x r ∈ S ry for α ∈ ( α 2 lb , 1) (Eq 20), we can write m inv ( z r )
= 1 α + η ( z r ) where η ( z r ) > 0 for all z r ∈ S ry .
Now we need to show that there exist an α ∈ ( α 2 lb , 1) such that m p ( z r ) > 1 for all z r ∈ S ry .
Thus from Eqwe have:
[Equation] [Equation]
Since α ∈ ( α 2 lb , 1) , we have ( α · η ( z r )) > 0 and ∥ w inv ∥√ 1 − α 2 > 0 .
Let’s define δ ( z r ) :
=
y (cid:0) ˆ ϵ sp · z sp (cid:1) .
Thus for the latent-points z r ∈ S ry which have δ ( z r ) ≥ 0 , Eq 23 is always satisfied since left side of inequality is greater than or equal to zero and right side is always less than zero.
For the points for which δ ( z r ) < 0 we have:
[Equation]
Now different z r will have different η ( z r ) which will give different lower bound of α .
Since the m α ( z r ) > 1 has to be satisfied for every point in z r ∈ S ry .
Thus it has to be satisfied for the point with minimum value η min =
min x r ∈S ry ( η ( x r )) which will give the tightest lower bound which α need to satisfy.
Thus we have:
[Equation]
Since η min > 0 and ∥ w inv ∥ 2 δ ( x r ) 2 > 0 , the lower bound α 3 lb < 1 in Eq 29.
Finally, combining Eq 17, Eq 20 and Eq 29 let the overall lower bound of α be α lb is:
[Equation]
[Equation]
This provides a way to construct a spurious-using classifier: given any purely-invariant , we can always choose α ∈ ( α lb , 1) and construct a perturbed spurious-using classifier from Eq 7 which has a bigger margin than purely-invariant .
Thus, given all the assumptions, there always exist a spurious-using classifier which has greater margin than the purely-invariant classifier completing our proof.
C.3. Proof of necessary condition
In this section we will show that the concept-probing classifier will use the spurious feature (
z m ) iff the spurious feature 2.3 is satisfied for the margin points of the clean concept-probing classifier when the concept-feature is 1-dimensional.
Also, same line of reasoning will hold for the main-task classifier where we will show that main-task classifier will use the spurious feature ( z p )
iff spurious feature satisfies C.2 for the margin point of clean main-task classifier.
Formally:
Lemma C.1 (Necessary Condition for concept-Probing Classifier) .
Let the latent representation be frozen/non-trainable and disentangled
such that z = [ z m , z p ] where z p is the concept-feature which is 1-dimensional and fully predictive (Assm 2.2) and z m ∈ R d m .
Let c ∗ p ( z )
= w p · z p be the desired clean/purely-invariant linear classifier trained using max-margin objective which only uses z p for prediction.
Then the concept-probing classifier trained using max-margin objective will be spurious-using i.e. c p ( z ) =
w p · z p + w m · z m where w m ̸ =
iff the spurious feature z m is linearly separable w.r.t to probing task label y p for the margin point of c ∗ p ( z ) (Assm 2.3) .
Lemma C.2 (Necessary Condition for Main-task Classifier) .
Let the latent representation be frozen/non-trainable and disentangled
such that z = [ z m , z p ] where z m is the main-task feature which is 1-dimensional and fully predictive (Assm C.1) and z p ∈ R d p .
Let c ∗ m ( z ) =
w m · z m be the desired clean/purely-invariant linear classifier trained using max-margin objective which only uses z m for prediction.
Then the main-task classifier trained using max-margin objective will be spurious-using i.e. c m ( z )
=
w m · z m + w p · z p where w p ̸ =
iff the spurious feature z p is linearly separable w.r.t to main task label y m for the margin point of c ∗ m ( z ) (Assm C.2) .
Since proof of both Lemma C.1 and C.2 follows same line of reasoning for brevity, following Sec C.2, we will prove the lemma for a general classifier c ( z ) trained using max-margin objective to predict the task-label y .
Let the latent representation be of form z = [ z inv , z sp ] where z inv ∈ R is the feature causally derived from the task-label y and z sp ∈ R dsp is the feature spuriously correlated to task label y .
With respect to concept-probing classifier c p ( z ) in Lemma C.
z inv :=
z p and z sp : =
z m .
Similarly, for the main-task classifier in Lemma C.2, z inv : =
z m and z sp : =
z p .
Proof of Lemma C.1 and C.2.
Now the goal is to show that Assm 2.3 for probing classifier or Assm C.2 for the main-task classifier is necessary for obtaining a spurious-using classifier for the case when z inv is one-dimensional.
We show this by assuming that optimal classifier is spurious-using even when Assm 2.3 or C.2 breaks and then show that this will lead to contradiction.
Contradiction Assumption: Formally, let’s assume that the optimal classifier for the given classification task is spurious-using c ∗ ( z ) , where:
[Equation]
31) where ∥ w ∗ sp ∥ ̸ = 0 and ˆ w ∗ sp ∈ R d sp is a unit vector in spurious subspace with dimension d sp .
Let c inv ( z ) =
w inv · z inv be the optimal purely-invariant classifier.
Let both c ∗ ( z ) and c inv ( z ) be trained using the max-margin objective using MM-Denominator formulation in Eq 4.
Thus from the constraints of this formulation (Eq 3) we have:
[Equation] [Equation]
From Assm 2.2 or C.1, the invariant feature z inv is 100% predictive and linearly separable w.r.t task label y .
Then without loss of generality let’s assume that:
[Equation]
From Eq 34 and 35 we have y · z inv > 0 thus from Eq 33 we get:
[Equation]
Also, from our contradiction-assumption the max-margin trained classifier is spurious-using , thus the norm of parameters of c ∗ ( z ) is less or equal to c inv ( z )
(Eq 4).
Thus we have:
[Equation]
Suppose Assm 2.3 for concept-probing task or Assm C.2 for the main-task breaks, then we have one of the following two case:
Opposite Side Failure: When the spurious part of margin point on opposite side of decision-boundary ( c ∗ ( z ) = 0 ) are not linearly-separable.
Formally, there exist two datapoint, P m + :=
[ z m + inv , z m + sp ] and P m − :=
[ z m − inv , z m − sp ] such that they are margin points of purely-invariant classifier c inv ( z ) where P m + has class label y
= +
and P
m − has class label y
= − 1 and ∀ ˆ ϵ sp ∈ R d sp , the spurious feature z sp of both the points lies on same side of ˆ ϵ sp i.e:
[Equation]
Same Side Failure: When the spurious part of margin point on same side of decision-boundary ( c ∗ ( z ) = 0 ) are always linearly-separable.
There exist two datapoints, P m 1 y :
[Equation]
[ x
[Equation]
inv , z m 1 sp ] and P m 2 y :=
[ x m 2 inv , z m 2 sp ] such that they are margin point of purely-invariant classifier c inv ( z ) and both points have same class label y and ∀ ˆ ϵ sp ∈ R d sp , w.l.o.g we have:
[Equation]
We will use the following two lemma to proceed with our proof:
Lemma C.3.
If Assm 2.3 or C.2 breaks by opposite-side failure mode it leads to contradiction.
Lemma C.4.
If Assm 2.3 or C.2 breaks by same-side failure mode it leads to contradiction.
This implies that our contradiction-assumption which said that the max-margin trained optimal classifier is spurious-using even when Assm 2.3 or C.2 breaks, is wrong.
Thus Assm 2.3 for concept-probing task or Assm C.2 for main-task is necessary for the optimal max-margin classifier to be spurious-using thus completing our proof.
Proof of Lemma C.3.
We have two point P m + :=
[ z m + inv , z m + sp ] and P m − :=
[ z m − inv , z m − sp ] which breaks the Assm 2.3 or
C.2.
From Eq 34, z inv > 0 for all the points with label y = 1 , thus we have x m + inv > 0 and using Eq 40 ( w ∗ inv < w inv ) we get:
[Equation] [Equation]
[Equation]
where the right hand side w inv · z m + inv = 1 since P m + is the margin-point of c inv ( z ) (Eq 33).
Similarly from Eq 35, z inv <for all the points with label y = − 1 , thus we have z m − inv < 0 and using Eq 40 ( w ∗ inv < w inv ) we get:
[Equation] [Equation]
[Equation]
48) where the right hand side ( − 1) · ( w pinv · x m − inv )
= 1 since P m − is the margin-point of c inv ( z ) (Eq 33).
Next from Eq 32 we have m ∗ ( z ) ≥ 1 for all “ z ” hence it is also true for P m + with y = 1 and P m − with y = − 1 .
Then:
[Equation]
where β m + = (cid:0) ˆ w ∗ sp · x m + sp (cid:1) .
Also we have:
[Equation]
where β m − = (cid:0) ˆ w ∗ sp · x m − sp (cid:1) .
From Eq 41 we have (cid:0) ( ˆ ϵ sp · x m + sp ) · ( ˆ ϵ sp · x m − sp ) (cid:1) ≥ 0 for all ˆ ϵ sp ∈ R d sp which states the opposite-side failure of Assm 2.3 or C.2.
Thus:
[Equation]
Now we will show that Eq 45, 48, 51 and 54 cannot be satisfied simultaneously for any allowed value of β m + and β m − (given by Eq 55) which are:
β m + > 0 and β m − > 0 : From Eq 54 we have − w ∗ inv x m − inv > 1 since ∥ w ∗ sp ∥ ̸ = 0 and β m − > 0 .
But from Eq 48 we have − w ∗ inv x m − inv < 1 which is a contradiction.
β m + < 0 and β m − < 0 :
From Eq 51 we have w ∗ inv x m + inv > 1 since ∥ w ∗ sp ∥ ̸ = 0 and β m + < 0 .
But from Eq 45 we have w ∗ inv x m + inv < 1 which is a contradiction.
β m + = 0 and β m − ∈ R :
From Eq 51 we have w ∗ inv x m + inv ≥
but from Eq 45 we have w ∗ inv x m + inv < 1 which is a contradiction.
β m + ∈ R
and
β m − =
: From Eq 54 we have − w ∗ inv x m − inv ≥ 1 but from Eq 48 we have − w ∗ inv x m − inv < 1 which is a contradiction.
Thus we have a contradiction for all the possible values β m +
and β m − could take, completing the proof of this lemma.
Proof of Lemma C.4.
We have two margin-points P m 1 y :=
[ z m 1 inv , z m 1 sp ] and P m 2 y :=
[ z m 2 inv , z m 2 sp ] which breaks Assm 2.3 or
C.2.
From Eq 34 and Eq 35 we have y · z m 1 inv > 0 and y · z m 2 inv > 0 .
Using Eq 40 ( w ∗ inv < w inv ) we get:
[Equation]
58) where j ∈ { 1 , 2 } and right hand side w inv · ( y · z mjinv ) = 1 since P mjy is the margin point of purely-invariant classifier c inv ( z ) (Eq 33).
From Eq 32 we have m ∗ ( x ) ≥ 1 for all “ z ” thus also true for P m 1 y and P m 2 y .
Then:
[Equation] [Equation]
where β m 1 = (cid:0) ˆ w ∗ sp · z m 1 sp (cid:1) .
Also we have:
[Equation] [Equation]
where β m 2 = (cid:0) ˆ w ∗ sp · z m 2 sp (cid:1) .
Now from Eq 42 we have β m 1 ≥ 0 and β m 2 ≤ 0 for all unit vector
ˆ ϵ sp ∈ R d sp which states the same-side failure mode of Assm 2.3 or C.2.
Now we will show that for all allowed value of β m 1 and β m 2 , Eq 58 61 and 64 will lead to contradiction.
Following are the cases for different allowed values of β m 1 and β m 2 : 1.
β m 1 = 0 and β m 2 ∈ R :
Substituting β m 1 = 0 in Eq 61 we get y · ( w ∗ inv z m 1 inv ) ≥ 1 , but from Eq 58 we have y · ( w ∗ inv z m 1 inv ) < 1 .
Thus we have a contradiction.
β m 1 ∈ R and β m 2 = 0 :
Substituting β m 2 = 0 in Eq 64 we get y · ( w ∗ inv z m 2 inv ) ≥ 1 , but from Eq 58 we have y · ( w ∗ inv z m 2 inv ) < 1 .
Thus we have a contradiction.
β m 1 > 0 , β m 2 < 0 and y =
(+1) :
Substituting β m 2 < 0 and y =
(+1) in Eq 64 we get y · ( w ∗ inv z m 2 inv ) ≥ 1 , but from Eq 58 we have y · ( w ∗ inv z m 2 inv ) < 1 .
Thus we have a contradiction.
β m 1 > 0 , β m 2 < 0 and y =
( − 1) :
Substituting β m 1 > 0 and y
=
( − 1) in Eq 61 we get y · ( w ∗ inv z m 1 inv ) ≥ 1 , but from Eq 58 we have y · ( w ∗ inv z m 1 inv ) < 1 .
Thus we have a contradiction.
Thus we have a contradiction for all the possible values β m 1 , β m 2 and y could take, completing the proof of this lemma.
C.4. Problem with learning a clean main-task classifier
In this section we will restate the assumptions and results of 2.1 for the main-task classifier (instead of the probing classifier) and show that the same results will hold.
Assm 2.1 remains the same since it is made on the latent-representation being disentangled and frozen/non-trainable.
Next, parallel to Assm 2.2 show that even when main-task feature is 100% predictive of main-task and a linearly separable, the trained main-task classifier will also use the concept-features.
Formally,
Assumption C.1 (main-task feature Linear Separability) .
The main-task features of the latent representation ( z m ) for every point are linearly separable/fully predictive for the main-task labels y m , i.e y
im · (ˆ ϵ m · z im + b m ) > 0 for all datapoints ( x i , y im ) for some ˆ ϵ m ∈ R d p .
For the case of zero-centered latent space, we have b m = 0 .
Next similar to Assm 2.3, we define the spurious correlation between main-task and concept label: a function using z p may also be able to classify correctly w.r.t.
main-task label on some non-empty subset of points.
Assumption C.2 (Main-Task Spurious Correlation) .
For a subset of training points S ⊂ D p , main-task label y m is linearly separable using z p
i.e y
im · (ˆ ϵ p · z ip + b p ) > 0 for all the points in the dataset, ˆ ϵ p ∈ R d p and b p ∈ R .
For the case of zero-centered latent space we have b p = 0 .
Next we rephrase Lemma 2.1 which shows that for only a few special points if the concept-features z p are linearly-separable w.r.t.
to main task classifier y m , then the main-task classifier c m ( z ) will use those features.
Lemma C.5 (Sufficient Condition for Main-task Classifier) .
Let the latent representation be frozen and disentangled
such that z = [ z m , z p ] (Assm 2.1), where main-task-features z m be fully predictive (Assm C.1).
Let c ∗ m ( z ) =
w m · z m be the desired/clean linear main-task classifier trained using max-margin objective (Suppl C.1) which only uses z m for its prediction.
Let z p be the spurious feature s.t.
for the margin points of c ∗ m ( z )
, z p be linearly-separable w.r.t.
task label y m (Assm C.2).
Then, assuming the latent space is centered around 0 , the main-task classifier trained using max-margin objective will be of form
c m ( z ) =
w m · z m + w p · z p where w p ̸ = 0 .
The proof of Lemma C.5 is identical to Lemma 2.1 and is provided in Suppl C.2.
D. Null-Space Removal Failure: Setup
and Proof of Theorem 2.
D.1. Null-Space Setup
As described in Sec 2, the given main-task classifier have an encoder h :
X → Z mapping the input X to latent representation Z .
Post that the main-task classifier c m : Z → Y m is used to predict the main-task label y
im from latent representation
z i for every input x i .
Given this (pre) trained main-task classifier the goal of post-hoc removal method to remove any undesired/sensitive/spurious concept from the latent representation Z without retraining the encoder h or main-task classifier c m ( z ) .
The null space method (Ravfogel et al., 2020; Elazar et al., 2021), henceforth referred as INLP , is one such post-hoc removal method which removes a concept from latent space by projecting the latent space to a subspace that is not discriminative of that attribute.
First, it estimates the subspace in the latent space discriminative of the concept we want to remove by training a probing classifier c p ( z ) → y p , where y p is the concept label.
(Ravfogel et al., 2020) used a linear probing classifier ( c p ( z )) to ensure that the any linear classifier cannot recover the removed concept from modified latent representation z ′ and hence the main task classifier ( c m ( z ′ ) ) also become invariant to removed attribute.
Let linear concept-probing classifier c p ( z ) be parametrized by a matrix W , and null-space of matrix W is defined as space N ( W )
= { z | W z = 0 } .
Give the basis vectors for the N ( W ) we can construct the projection matrix P N ( W ) such that W ( P N ( W ) z ) = 0 for all z .
This projection matrix is defined as the guarding operator g :=
P N ( W ) , when applied on the z will remove the features which are discriminative of undesired concept from z as estimated by c p ( z ) .
For the setting when Y p is binary we have:
[Equation]
where I is the identity matrix and ˆ w is the unit vector in the direction of parameters of classifier c p ( z ) ((Ravfogel et al., 2020)).
Also, the authors recommend running this removal step for multiple iterations to ensure the unwanted concept is removed completely.
Thus after S steps of removal, the final guarding function is:
[Equation]
where P iN ( W ) is the projection matrix at i th removal step.
Amnesic Probing ((Elazar et al., 2021)) builds upon this idea for testing whether concept is being used by a given pre-trained classifier or not.
The core idea is to remove the concept we want to test from the latent representation and if the prediction of the given classifier is influenced by this removal then the attribute was being used by the given classifier otherwise not.
D.2. Null-Space Removal Failure : Proof of Theorem 2.
Theorem 2.2.
Let the latent representation z be disentangled (Assm 2.1),
c m ( z ) be the pre-trained main-task classifier, and c p ( z ) the probing classifier used by INLP to remove the unwanted feature z p from the latent representation.
Let the assumptions of Lemma 2.1 be true for the probing classifier c p ( z ) , trained using max-margin objective.
Mixing: After the first projection step of linear-INLP, the dimensions of z gets mixed such that z
i
(1) =
[ g ( z im , x ip ) , f ( z ip , z im )]
̸ =
[ z im , z ip ] .
Also, this mixing is non-invertible with subsequent projection steps.
Thus, the latent space is no longer disentangled and removal of concept-causal features will also lead to removal of task-specific features.
Removal: The L2-norm of the latent representation z decreases with every projection step as long as the parameters of probing classifier at a step ( w k ) does not lie completely in the space spanned by parameters of previous probing classifiers i.e. span( w 1 , . . .
, w k − 1 ).
Thus, after sufficiently many steps, INLP can destroy all information in the representation, z i ( ∞ ) =
The proof of Theorem 2.2 proceeds in two steps:
We first show that after first step of null-space projection (INLP), both the main-task feature and concept-causal feature gets emph .
Then we show that the projection operation is non-invertible thus, mixing cannot be corrected in subsequent projection steps.
Next we show that the projection operation is lossy, i.e removes the norm of latent representation under some condition.
Hence after sufficient steps, INLP could destroy all the information in latent representation.
Proof of Theorem 2.2.
First Claim.
Let c p ( z ) =
w p z p + w m z m be the linear concept-probing classifier trained to predict the probing label y p from the latent representation z .
Since all the assumptions of Lemma 2.1 are satisfied for the probing classifier c p ( z ) , it is spurious using , i.e., w m ̸ = 0 and w p ̸ = 0 .
Since the concept-probing label y p is binary the projection matrix for the first step of INLP removal is defined as P 1 N ( W )
=
I − ˆ w ˆ w T where ˆ w T
=
[ ˆ w m , ˆ w p ] , ˆ w m and ˆ w p are the unit norm parameters of c p ( z ) i.e w m and w p respectively.
On applying this projection on the latent space representation z i we get:
[Equation]
Next we will show that g ( z m , z p ) ̸ =
z m
and h ( z m , x p ) ̸ = z p .
For g ( z m , z p ) =
z m we need:
[Equation]
We know that w m ̸ = 0 and since the probing classifier is trained using max-margin objective using the MM-Denominator formulation we have y ip · c p ( z i ) ≥ 1 from Eq 3.
Thus c p ( z i ) ∈ ( ∞ , − 1] ∪ [1 , ∞ ) and ˜ c p ( z i ) ∈ ( ∞ , − 1 ∥ w ∥ ] ∪ [ 1 ∥ w ∥ , ∞ ) ̸ = 0 .
Since w m ̸ = 0 and ˜ c p ( z i
[Equation]
0 , Eq 72 cannot be true hence g
( z m , z p ) ̸
= z m .
Similarly for h ( z m , x p ) =
z p we need:
[Equation]
And again since w p ̸ = 0 and ˜ c p ( z i ) ̸ = 0 , Eq 74 cannot be true hence h
( z m , z p ) ̸
= z p .
Hence both concept-feature z p and the main-task feature z m got mixed after the first step of projection.
Next, the following lemma proves that the above projection matrix ( P 1 N ( W ) ) is non-invertible.
The subsequent steps of
INLP applies projection transformation which can be combined into one single matrix P > 1 N ( W ) =
(cid:81) j> 1 P jN ( W ) .
In order for mixing to be reversed, we need P > 1 N ( W ) × P 1 N ( W )
=
I , thus we need P > 1 N ( W ) =
( P 1 N ( W ) ) − 1 which is not possible from
Lemma D.1.
Hence the mixing of main-task feature and the concept-feature which happened after first step of projection couldn’t be corrected in the subsequent steps of INLP thus completing the first part of our proof.
Lemma D.1.
The projection matrix P 1 N ( W ) at any projection step of INLP is non invertible.
Proof of Lemma D.1.
The projection matrix for binary target case is defined as P :
= P 1 N ( W ) =
I − A where A = ˆ w ˆ w T be a n × n matrix.
We can see that A is a symmetric matrix.
From Eq.
6 in (Weisstein)), every symmetric matrix is diagonalizable.
Hence, we can write A = Q Λ Q T , where Q is a some orthonormal matrix such that QQ T
=
I and Λ = diag ( λ 1 , . .
. , λ n ) be a n × n diagonal matrix where the diagonal entries ( λ 1 . . .
λ n ) are the eigen-values of A .
Since QQ T
=
I we can write
P =
I − A
=
QQ T − Q Λ Q T =
Q ( I − Λ) Q T .
For the projection matrix P to be invertible P − 1 should exist.
We have:
[Equation]
So projection matrix is only invertible when ( I − Λ) is invertible.
We will show next that ( I − Λ) is not invertible thus completing our proof.
We have I − Λ = diag (1 − λ 1 , . . .
, 1 − λ n ) , hence:
[Equation]
Now, if one of the eigen values of A is 1 , then the diagonal matrix ( I − Λ) is not invertible.
If one of the eigen values of A is 1 , then there exist an eigen-vector x such that A x = ˆ w ˆ w T × x
= 1 × x .
The vector x = ˆ w is the eigen vector of A with eigen-value 1 :
A ˆ w = ˆ w ˆ w T × ˆ w = 1 × ˆ w since ˆ w T × ˆ w = 1 as it is a unit vector.
Hence the projection matrix is not invertible.
Second Claim.
Now for proving these second statement, we will make use of the following lemma:
Lemma D.2.
After every projection step of INLP, the norm of every latent representation z i decreases, i.e., ∥ z i ( k ) ∥ <
∥ z i ( k − 1) ∥ for step k and k − 1 , if 1) z i ( k − 1) ̸
= 0 and 2) the parameters of probing classifier in step “ k ” i.e ˆ w k don’t lie in the space spanned by parameters of previous concept-probing classifier, span( ˆ w 1 , . . .
, ˆ w k − 1 ).
Proof of Lemma D.2.
After k − 1 -steps of INLP let the latent space representation
z i be denoted as z i ( k − 1) .
Let ˆ w k be the parameters of classifier c p ( z k − 1 ) trained to predict the concept–probing label y p which we want to remove at step k .
Then before the projection step in the k th iteration of the INLP we can write z k − 1 as:
[Equation]
where { ˆ w k , N ( ˆ w k ) } is the basis set and N ( ˆ w k ) is the null-space of ˆ w k .
The parameter w k in this new basis is:
[Equation]
where I ˆ w k is identity matrix with dimension d ˆ w k ×
d
ˆ w k .
Now, in this new basis when we project the z k − 1 to the null space of ˆ w k we have:
[Equation]
Thus the norm of ∥ z k ∥ =
(cid:112) ∥
z N ( ˆ w p ) ∥ + 0 is less that ∥
z k − 1 ∥ = (cid:113) ∥
z ˆ w k ∥ 2 + ∥
z N ( ˆ w k ) ∥ 2 if z ˆ w k ̸ = 0 .
Next we will show that z ˆ w k cannot be zero.
From Eq 84, we observe that at any step “ k ” INLP removes the part of the representation from z k − 1 which is in the direction of ˆ w k i.e. z ˆ w k .
Consequently, a sequence of removal steps with parameters ˆ w 1 , . . .
, ˆ w k −will remove the part of z which lies in the span( ˆ w 1 , . . .
, ˆ w k − 1 ).
Thus z ˆ w k = 0 if ˆ w k lies in the span of parameters of previous classifier i.e span( ˆ w 1 , . . .
, ˆ w k − 1 ) which violates the assumption 2) in Lemma D.2, thus completing our proof.
Given parameters of current step concept-probing classifier doesn’t lie in the span of previous probing classifier’s parameter, the norm of the latent representation is decreasing in every step.
Thus given large number of INLP projection iteration ∥
z i ( ∞ ) ∥ → 0 thus completing the proof of the second statement.
The proof of Lemma D.1 and Lemma D.2 is given below.
Remark.
The following lemma from (Ravfogel et al., 2020) tells us some of the sufficient condition when the parameters of probing classifier at current iteration will not be same as previous step:
Lemma D.3 (Lemma A.1 from (Ravfogel et al., 2020)) .
If the concept-probing classifier is being trained using SGD (stochastic gradient descent) and the loss function is convex then parameter of probing classifier at step k i.e w k is orthogonal to parameter at step k − 1 i.e w k − 1 .
We conjecture that Lemma D.3 will be true for any loss since after k − 1 steps of projection, the component of z in the direction of span( w 1 , . . .
, w k − 1 ) will be removed.
Hence the concept-probing classifier should be orthogonal to span( w 1 , . . .
, w k − 1 ) in order to have non-random guess accuracy on probing task.
E. Adversarial Removal: Setup and Proof E.1. Adversarial Setup
As described in Sec 2.3, let h :
X → z be an encoder mapping the input x
i to latent representation z i .
The main task classifier c m : Z → Y m is applied to predict the main task label y
im from the input latent representation
z i for every input x i .
As described in Sec 2.3, the goal of adversarial-removal method, henceforth referred as AR, is to remove any undesired/sensitive/spurious concept from the latent representation Z .
Once the concept is removed from latent representation any (main-task) classifier using the latent representation Z will not be able to use them (Ganin & Lempitsky, 2015; Xie et al., 2017; Elazar & Goldberg, 2018).
These methods train jointly trains the main-task classifier
c m ( z ) and the probing classifier c p : Z → Y p which is adversarially trained to predict the concept label y ip from latent representation z i .
Hence,
AR methods optimize the following two objectives simultaneously:
[Equation]
[Equation]
Here L ( · ) is the loss function which estimates the error given the ground truth y m /y
p and corresponding prediction c m ( z ) /c
p ( z ) .
A valid equilibrium point of the above min-max game (and the desired solution) is an encoder h such that it removes all the features from latent space useful for prediction of y p while keeping intact other features causally-derived from the main-task prediction.
In practice, the optimization to above min-max game is performed using gradient-reversal (GRL) layer ((Ganin & Lempitsky, 2015)).
It introduces an additional layer g λ between the latent representation h ( z ) and the adversarial classifier c p ( z ) .
The g λ layer acts as an identity layer (i.e., has no effect) during the forward pass but scales the gradient by ( − λ ) when back-propagating it during the backward pass.
Thus resulting combined objective is:
[Equation]
E.2. Adversarial Proof
Theorem 2.3.
Let the latent representation Z satisfy Assm 2.1, h 2 ( z ) be a linear transformation s.t.
h 2 : Z → ζ , main classifier be c m ( ζ ) =
w c m · ζ and the adversarial classifier be c p ( ζ )
=
w c p · ζ .
Let all the assumptions of generalized
Lemma 2.1 (Suppl C.4) be satisfied for main-classifier c m ( · ) when using Z directly as input and Assm 2.2 be satisfied on Z w.r.t.
adversarial-task.
Let h ∗ 2 ( z ) be the desired encoder which is successful in removing z p from ζ .
Then there exists an undesired/incorrect encoder h 2 ( z ) such that ζ =
h 2 ( z ) =
f ( z m , z p ) for some function “ f ” and the main classifier using it c m ( h 2 ( z )) has bigger margin than c m ( h ∗ 2 ( z )) and has, 1.
Acc ( c p ( h 2 ( z )) , y p ) =
Acc ( c p ( h ∗ 2 ( z )) , y p ) when adversarial probing classifier c p ( · ) is trained using any learning objective.
Thus the undesired encoder h 2 ( z ) is indistinguishable from desired encoder h ∗ 2 ( x ) in terms of adversarial task prediction accuracy but better for main-prediction task in terms of max-margin objective.
L ( c p ( h 2 ( z )) , y p ) >
L ( c p ( h ∗ 2 ( z )) , y p ) under Assm 2.2 and Assm 2.4 where L is either max-margin or cross-entropy loss used for training c p ( · ) .
Thus undesired encoder h 2 ( z ) is preferable over desired encoder h ∗ 2 ( z ) for both main and adversarial task objective.
Proof of Theorem 2.3.
Let the main classifier be of form
c m ( ζ ) =
w c m · ζ where w c m and ζ are d ζ dimensional vectors.
Since both parameters w c m and ζ are learnable, for ease of exposition we constrain w c m to be [1 , 0 , . . .
This constraint on w c m is w.l.o.g.
since w c m uses the latent representation ζ by projecting it into one specific direction of ζ encoded by h 2 ( z ) .
We constrain that direction to be the first dimension of ζ .
Since the encoder is trainable it could learn to encode that information in the first dimension of ζ .
Thus, effectively a single dimension of the representation ζ encodes the main-task information.
Thus the main classifier is effectively of form c m ( ζ ) =
w 0 c m × ζ 0 = 1 × ζ 0 where w 0 c m and ζ 0 are the first element of w c m and ζ respectively and w 0 c m = 1 .
We can now write the goal of the adversarial method as removing the information of z p from ζ 0 , because the other dimensions are not used by the main classifier .
The the adversarial classifier can be written effectively as c p ( ζ ) =
β × ζ 0 where β ∈ R is a trainable parameter.
Since both the main and adversarial classifier are using only ζ 0 , the second encoder can be simplified as ζ := ζ 0 :
=
h 2 ( z ) =
w m · z m + w p · z p where ζ ∈ R and w m and w p are the weights that determine the first dimension of ζ . .
Also, let the desired “correct" second encoder which is successful in removing the spurious-concept z p from ζ be h ∗ 2 ( z ) =
w ∗ m · z m .
First claim.
The ideal main-classifier with desired encoder can be written as, c m ( h ∗ 2 ( z ))
=
1 × h ∗ 2 ( z ) =
w ∗ m · z m .
Therefore, it can be trained using same MM-Denominator formulation of max-margin objective and would satisfy the constraint in Eq 3:
[Equation]
for all the points x i with latent representation z i and m ( . ) is the numerator of the distance of point from the decision boundary of classifier (Eq 1).
However, the main-task classifier which does not use the desired encoder is of the form, c m ( ζ ) =
× h 2 ( z ) =
w m · z m + w p · z p .
Since this main-task classifier is also trained using max-margin objective by MM-Denominator formulation, it would satisfy the constraint in Eq 3:
[Equation]
Since all assumptions of Lemma C.5 are satisfied, we use the lemma to conclude that the main-task classifier trained using max-margin objective will be spurious-using i.e w m ̸ = 0 and w p ̸ =
(Def C.2).
Hence there exists an undesired/incorrect second encoder h 2 ( z ) such that the main classifier c m ( h 2 ( z )) has bigger margin than c m ( h ∗ 2 ( z )) .
Given that there exists a main task classifier with an undesired encoder that has a higher accuracy than the one with the desired encoder, our next goal is to show that the accuracy of the adversarial classifier remains the same irrespective of whether the desired or undesired encoder h 2 ( z ) is used.
The accuracy of the adversarial classifier c p ( ζ ) =
β × ζ , using the desired/correct encoder ζ = h ∗ 2 ( z ) is given by:
[Equation]
where 1 ( · ) is an indicator function which takes the value 1 if the argument is true otherwise 0 , and sign ( γ )
= +1 if γ ≥and − 1 otherwise.
Combining Equations
and Eq 89, since y im ∈ {− 1 , 1 } , we see that whenever h 2 ( z i ) >
1 we also have h ∗ 2 ( z i ) > 1 and similarly whenever h 2 ( z i ) <
− 1 , we have h ∗ 2 ( z i ) < − 1 .
Thus,
[Equation]
From Eq 91, h ∗ 2 ( z i ) and h 2 ( z i ) has the same sign for every input z i .
Thus we can replace h ∗ 2 ( z i ) with h 2 ( z i ) in the above equation and we have:
[Equation]
thus completing the first part our proof.
Second claim.
In the first part of proof we have seen that from Lemma C.5 there exist an incorrect second-encoder h 2 ( z ) such that margin of main-task classifier c m ( h 2 ( z )) is larger than c m ( h ∗ 2 ( z )) .
Next, we show that adversarial classifier c p ( β · ζ ) when using incorrect/undesired second-encoder ζ = h 2 ( z ) for predicting concept-label y p will have a higher prediction loss as compared to the case when using correct/desired second-encoder ζ = h ∗ 2 ( z ) .
Under such a scenario, the incorrect encoder will be preferred by both the main and the adversarial task objective over the correct one.
Note that the adversarial-probing classifier prefers a higher prediction loss since this will imply that the latent representation generated by the second encoder is less predictive of the concept-causal feature we are trying to remove.
Construction of the undesired/incorrect encoder.
Let any margin point of main-task classifier using the correct encoder be denoted by z µ,i , where µ , denotes the points is margin-point of main-task classifier.
From Assm C.2 (in Lemma C.5), we know there exist some unit vector
ˆ ϵ p ∈ R d p such that spurious concept-feature ( z p ) of the margin points of the main-task classifier ( z µ,i ) using the correct encoder is linearly separable wrt main-task label such that:
[Equation] [Equation]
and from Lemma C.5 there exists an incorrect encoder h
α 2 ( z )
such that the main task
classifier
c m ( h 2 ( z )) has bigger margin than c m ( h ∗ 2 ( z )) when α ∈ ( α lb , 1) , where α lb is given by Eq 30 in Lemma C.5:
[Equation]
[Equation]
where w ∗ m is the parameter of the correct/desired encoder h ∗ 2 ( x )
= w ∗ m · z m .
We now use the assumptions in the theorem statement for the second claim.
From Assm 2.2, we have a fully predictive concept-feature z p for prediction of adversarial label y p such that for some unit vector ˆ w p ∈ R d p we have:
[Equation]
Now from Assm 2.4, we have y ip =
y
i m for every margin point of the desired/correct main-task classifier using the desired/correct encoder h ∗ 2 ( z ) .
Thus we can assign ˆ ϵ t :
=
ˆ w t which satisfies the inequality in Eq 92.
Thus our incorrect encoder h α 2 ( z ) take the following form:
[Equation]
We will use the following two lemmas to prove that the adversarial prediction loss will be bigger for the incorrect/undesired encoder than the correct/desired one.
Next the following lemmas will show that w.r.t.
max-margin and cross-entropy objective, the incorrect/undesired encoder’s loss is greater than the correct/desired encoder’s loss.
We state two simpler lemmas first and then define the two main lemmas.
Lemma E.1 (Negative adv-labels) .
For all the points with negative adversarial label i.e. y ip = − 1 , we have h α 2 ( z i ) <
h ∗ 2 ( z i )
Lemma E.2 (Positive adv-labels) .
For all the points with positive adversarial labels i.e y ip = +1 we can choose an α ∈ ( α lb , 1) such that h
α 2 ( z i ) >
h ∗ 2 ( z i ) .
Definition E.1 (Boundary-Distance Based Loss) .
A loss function L belongs to boundary-distance-based loss family, if the prediction loss increases as the distance from the boundary decreases.
The distance from the boundary of a classifier “ c ” for a point z
i is calculated by d c ( z i ) :
= M c ( z i ) (Eq 1).
For mis-classifed points the distance from decision boundary is negative and for correctly classified points the distance from decision boundary is positive.
Lemma E.3.
Given Lemma E.1 and Lemma E.2 holds, if the loss function L is boundary-distance-based (Def E.1), then L ( c p ( h 2 ( z i )) , y ip ) >
L ( c p ( h ∗ 2 ( z i )) , y ip ) for every example ( z i , y ip ) .
Cross-Entropy and Max-margin loss belongs to boundary-distance-based loss.
Lemma E.4.
Max-margin and Cross-Entropy Loss belongs to the boundary-distance-based loss family (Def E.1) when the norm of parameters of classifier is fixed.
Lemma E.3 and Lemma E.4 state that there exists an incorrect/undesired encoder h α 2 ( z ) which has the same norm as correct/desired encoder h ∗ 2 ( z ) , such that L ( c p ( h α 2 ( z i )) , y ip ) >
L ( c p ( h ∗ 2 ( z i )) , y ip ) where L is max-margin or cross-entropy loss for all z i .
Hence the overall loss which is the sum of the loss for every sample is also greater for adversarial classifier with incorrect encoder h
α 2 ( z ) .
Thus the incorrect encoder is preferred over the correct encoder for both the main task (bigger margin) and the adversarial task objective (bigger prediction loss).
We now prove all the required lemmas stated in the above proof.
Proof of Lemma E.1.
From Eq 94, for the points having adversarial label y ip = − 1 , we have ˆ w p · z ip < 0 .
Now, since
0 < α lb < α <
we have √ 1 − α 2 ∥
w ∗ m ∥ > 0 , and can write,
[Equation]
since √ 1 − α 2 ∥ w ∗ m ∥ (cid:0) ˆ w p · z ip (cid:1) < 0 for points with y ip = − 1 , thus completing our proof.
Proof of Lemma E.2.
We have to show that there exist an α ∈ ( α lb , 1)
such that h α 2 ( z i ) >
h ∗ 2 ( z i ) for every z i in the dataset.
For h α 2 ( z i ) > h ∗ 2 ( z i ) we need:
[Equation]
+
[Equation]
since α ̸ = 1 .
We defined γ ( z i ) =
ˆ w p · z ip ˆ w ∗ m · z im , where the denominator ( ˆ w ∗ m · z im ) ̸ = 0 since y im ( ˆ w ∗ m · z im ) ≥ 1 and y im ∈ {− 1 , 1 } (Eq 88).
Also, from Eq 94 we have ( ˆ w p · z ip ) ̸ = 0 , thus
γ 2 ( z i ) > 0 , hence we have 0 < 1 − γ 2 ( z i ) 1+ γ
( z i ) < 1 .
Hence we can choose an α to construct h 2 ( z )) such that,
[Equation]
Then we have 1) h
α 2 ( z i ) >
h ∗ 2 ( z i ) for all z i and 2) margin of main classifier using incorrect/undesired encoder h α 2 ( z ) is bigger than correct/desired encoder ( h ∗ 2 ( z ) ), thus proving our lemma.
Proof of Lemma E.3.
From Lemma E.2 we have h α 2 ( z ) > h ∗ 2 ( z ) for all the points which have adversarial label y ip =
+ and the value of α given by Eq 106.
Since, β is a trainable parameter, the adversarial classifier could choose any value for it which will help it increase the adversarial loss.
Let’s set trainable parameter β
= ( − 1) for the adversarial classifier c p ( h α 2 ( z ))
=
β · h α 2 ( z ) = − h
α 2 ( z ) and c p ( h ∗ 2 ( z ))
=
β · h ∗ 2 ( z ) =
− h ∗ 2 ( z ) .
Since by construction the norm of parameters of both h α 2 ( z ) and h ∗ 2 ( z )
=
w ∗ m (Eq 93) , we have:
[Equation]
for all the points with y ip = +1 .
Similarly for the points with adversarial label y ip = − 1 , from Lemma E.1 we have h α 2 ( z ) <
h ∗ 2 ( z ) and:
[Equation]
for all points with y ip = − 1 .
In Def E.1, we have defined the distance of an input from decision boundary of classifier “ c ” as d c ( z i ) :
= M c ( z i ) .
Hence from Eq 111 and 116 we have
[Equation]
) for all z i .
So, the distance of all the latent-points from margin is smallar when adversarial classifier c p uses incorrect/undesired encoder h α 2 ( z ) than correct/desired h ∗ 2 ( z ) .
If the loss function “ L ” is boundary-distance-based loss (Def E.1), then L ( c h
α 2 ( z i ) , y ip ) >
L ( c h ∗ 2 ( z i ) , y ip ) for every z i , y ip , thus completing our proof.
Proof of Lemma E.4.
Max-Margin:
From Eq 2, the max-margin loss (to be minimized) for a classifier “ c ” taking input z
i is defined as:
[Equation] [Equation]
where ∥ w c ∥ are the parameters of the classifier “ c ”.
If the distance of all the point from decision boundary of classifier “ c ” i.e. d c ( z i ) :
= M c ( z i ) is decreasing, then min i M c
( z i ) is decreasing hence ( − 1) · (cid:8) min i M c
( z i ) (cid:9) is increasing.
Max-margin loss in Eq 118 is increasing as the distance of points from boundary of classifier is decreasing.
Thus from (Def E.1) L ( max-margin ) belongs to boundary-distance-based family of losses.
Cross-Entropy: Next for the binary-cross-entropy loss for a linear classifier c ( z i ) =
σ ( w c · z i ) taking input z
i is defined as
[Equation]
when y i =
[Equation] [Equation]
when y i =
[Equation] [Equation]
120) where σ ( x ) =
1 1+exp − x is the sigmoid function.
In E.1 the distance of a point from the decision boundary is defined as:
[Equation]
When y i = 1 , keeping ∥ w c ∥ as fixed, if the distance of the point decreases the from Eq 121, c ( z i ) decreases.
Hence the cross entropy loss given by:
[Equation]
increases since − log( c ( z i )) is a decreasing function.
Similarly for the points with y i = − 1 , keeping ∥ w c ∥ fixed, if the distance of the point decreases, from Eq 121, c ( z i ) increases.
Then the cross-entropy loss given by:
[Equation]
123) increases, since − log(1 − c
( z i )) is an increasing function.
Cross-entropy loss with fixed parameter norm is increasing as the distance of points from the boundary of classifier is increasing.
Thus from (Def E.1) L ( max-margin ) belongs to boundary-distance-based family of losses.
F. Extension to real-world data: A metric for quantifying degree Of spuriousness
Our theoretical analysis shows that probing-based removal methods fail to make the main task classifier invariant to unwanted concept.
However, to verify whether the final classifier is using the concept or not, the theorem statements require knowledge of the concept’s features z p .
For practical usage, we propose a metric that quantifies the degree of failure or spuriousness for both the main and probing classifier.
For simplicity, we define it assuming that both main task and concept are binary.
Let D m,p be the dataset where for every input x i we have both the main task label y m and the concept label y p .
We define 2 × 2 groups, one for each combination of ( y m , y p ) .
Without loss of generality, assume that the main-task label y m = 1 is spuriously correlated with concept label y p = 1 and similarly y m = 0 is correlated with y p = 0 .
Thus, ( y m = 1 , y p = 1) and ( y m = 0 , y p = 0) are the majority group S maj while groups ( y m = 1 , y p = 0) and ( y m = 0 , y p
= 1) make up the minority group ( S min ).
We expect the main classifier to exploit this correlation and hence perform badly on S min where the correlation breaks.
Following (Sagawa et al., 2020a), we posit that minority group accuracy i.e Acc ( S min ) can be a good metric to evaluate the degree of spuriousness .
We bound the metric by comparing it with the accuracy on S min of a “clean” classifier that does not use the concept features.
Definition 2.2.
Given a dataset, D m,p = S min ∪ S maj containing binary task label and binary concept, let Acc c m ( S min ) be the accuracy of the given main-task classifier ( c m ) on the minority group S min and Acc ∗ ( S min ) be the accuracy on the minority group of a clean classifier that does not use the unwanted spurious concept.
We define spuriousness score of “ c m ” as: ψ ( c m ) =
(cid:12) (cid:12)
[Table]
To estimate Acc ∗ ( S min ) , we subsample the dataset such that y p takes a single value in the sample and train the main classifier on it, as done by (Ravichander et al., 2021).
Thus, probing label y p no longer is correlated with the main-task label y m .
There can be other ways to estimate it, e.g., by reweighting the data or using the accuracy on S maj .
However, we found that the former had high variance and the latter requires an equal-noise assumption such that Acc ( S maj ) =
Acc ( S min ) on a clean main-task classifier.
G. Experimental Results
Theorems 2.2 and 2.3 show the failure of concept removal methods under a simplified setup and max-margin loss.
But current deep-learning models are not trained using max-margin objective and might not satisfy the required assumptions (Assm 2.1,2.2,2.3,2.4).
Thus, we now verify the failure modes on three real-world and one synthetic datasets, without making any restrictive assumptions.
We use RoBERTa ((Liu et al., 2019)) as default encoder and fine-tune it over each real-world dataset.
For Synthetic-Text dataset we use sum of pre-trained GloVe embeddings ((Pennington et al., 2014)) of words in sentence as default encoder.
For more details of experimental setup refer Suppl H.
G.1. Datasets: Main task and spurious/sensitive concept
Real-world data.
We use three different real-world datasets: MultiNLI (Williams et al., 2018),
Twitter-PAN16 (Rangel et al., 2016) and Twitter-AAE (Blodgett et al., 2016).
In MultiNLI dataset, given two sentences—premise and hypothesis–the main task is to predict whether hypothesis entails , contradicts or is neutral with respect to premise.
We simplify to a binary task of predicting whether a hypothesis contradicts the premise or not.
Since negation words like nobody,no,never and nothing have been reported to be spuriously correlated with with the contradiction label (Gururangan et al., 2018b), We create a ‘negation’ concept denoting the presence of these words.
The goal is to remove the negation property from the NLI model’s representation space.
In Twitter-PAN16 dataset, the main task is to detect whether a tweet mentions another user or not, as in (Elazar & Goldberg, 2018).
The dataset contains gender label for each tweet, which we consider as the sensitive concept to be removed from the main model’s representation.
In Twitter-AAE dataset, again following (Elazar & Goldberg, 2018), the main-task is to predict binary sentiment labels from a tweet’s text.
The tweets are associated with race of the author, the sensitive concept that should be removed from the main model’s representation.
Synthetic-Text.
To understand the reasons for failure, we introduce a Synthetic-Text dataset where it is possible to change the input text based on a change in concept Here we can directly evaluate whether the concept is being used by the main-task classifier by intervening on the concept (adding or removing) and observing the change in prediction scores.
The main-task is to predict whether a sentence contains a numbered word (e.g., one, fifteen, etc. ).
We introduce spurious concept (length) by increasing the length of sentences that contain numbered words.
Predictive correlation.
To assess robustness of removal methods, we create multiple datasets with different predictive
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Figure]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
M a
[Figure]
n - T a s k
[Figure]
cc
[Figure]
c
[Figure]
( A ll )
Dataset: Twitter-PAN16 =0.5=0.
=0.7=0.8=0.9=0.
(e) Main-Task Acc(All) 7 9 11 13 15 17 19 21
Pretraining: 1-6, INLP-Removal: 70.
M a i n - T a s k A cc u r a c y ( A ll )
Dataset: Twitter-AAE =0.5=0.6=0.7=0.
=0.9=0.
(f) Main-Task Acc(All) correlation between two labels y m and y p .
The predictive correlation ( κ ) measures how informative one variable is for predicting the other, κ = Pr
[Equation]
N , where N is the size of dataset and 1 [ · ] is the indicator function which is 1 if the argument is true otherwise 0 .
Predictive-correlation lies in κ ∈ [0 . 5 , 1] where κ = 0 .
5 means no-correlation and κ = 1 means the attributes are completely correlated.
For complete description of all datasets and how we vary the predictive-correlation, refer to Suppl H; and for additional results refer Suppl I.
G.2. Results: Null Space Removal
In general, for any model given as input to INLP, it may be difficult to verify whether the removal method removed the correct features.
Hence, we construct a benchmark where the input classifier is clean , i.e., it does not use the concept at all.
We do so by training on a subset of data with the same value of the property, as in (Ravichander et al., 2021).
Since the input classifier does not have any concept features in its representation, we should expect that INLP should not have any effect on the main task classifier.
Eventually all task-relevant features are destroyed.
We start with the Synthetic-Text dataset by training a clean classifier on the main-task and input it to INLP for removing the sensitive concept.
To keep the conditions favorable for INLP, both the main task and sensitive-concept probing task can achieve 100% accuracy using their causally derived features respectively.
In Fig 4a, iteration 21-40 shows individual steps of null-space removal.
Since, the given pre-trained classifier was clean i.e no using the sensitive features, null-space removal shouldn’t have any effect on it.
Colored lines show different datasets used by INLP with different level of predictive correlation κ .
We observe that for all value of κ , the main-task classifier’s accuracy eventually goes to 50% random guess accuracy implying that the main-task related attribute has been removed by the removal process, as predicted by Theorem 2.2.
Higher the value of correlation κ , faster the removal of main-task attribute happens.
We obtain a similar pattern over the real-world datasets, MultiNLI, Twitter-PAN16 and Twitter-AAE.
Fig 4d,4e and 4f show a decrease in the main-task accuracy to random-guess even when the input classifier for each dataset is ensured to be clean , thus demonstrating the failure of INLP method.
Early stopping increases the reliance on spurious features.
To avoid the full collapse, a stopping criteria of INLP is to stop when the main-task classifier’s performance drops (Ravfogel et al., 2020).
In Fig 4b we measure the sensitivity of
Synthetic-Text main-task classifier w.r.t.
to sensitive concept by changing the feature in input sentences corresponding to sensitive concept and measuring the change in main-task classifier prediction probability ∆ Prob.
At lower iterations of
INLP, the change in main-task output due to change in sensitive concept’s value, ∆ Prob, is higher than that of the input classifier.
For example, for κ = 0 . 8 , the main-task classifier’s performance drops at iteration 27, but it has a high ∆ Prob ≈ 25% as shown in Fig 4b.
Hence it is possible that stopping prematurely will lead us to a classifier which is more unfair or reliant to sensitive concept than it was before, consistent with the first statement in Theorem 2.2 stating that INLP will lead
[Figure]
a s
[Figure]
c y
[Figure]
Sp u n
[Figure]
[Figure]
to mixing of features in latent space.
Failure of causally-inspired probing like Amnesic probing.
Amnesic Probing (Elazar et al., 2021) declares that a sensitive concept is being used by the model if, after removal of the concept from from the latent representation using INLP, there is a drop in the main-task performance.
But Fig 4a, 4d, 4e and 4f show that even when the input classifier for their corresponding main-task is clean, i.e., does not use the sensitive concept, INLP leads to drop in performance of the main-classifier.
Hence, removal-based methods like Amnesic probing will falsely claim that the sensitive concept is being used.
G.3. Results: Adversarial Removal Method
We demonstrate the failure of adversarial-removal method (AR) in removing the spurious/unwanted concept from the main classifier.
Here we compare the standard ERM training of the main classifier with AR method over the same number of epochs (20).
We follow the training procedure of (Elazar & Goldberg, 2018), and conduct a hyper-parameter sweep on the adversarial training strength to select the value which is most effective in removing the unwanted property.
For more details, refer to Suppl H.
Cannot remove the spurious concept fully.
For MultiNLI, Fig 5b shows the spuriousness score (Def 2.2) of ERM and AR classifiers as we vary the predictive correlation ( κ ) between the main-task label and sensitive concept label in the training dataset.
While the spuriousness score for classifier trained using AR (blue curve) is lower than that of ERM for all values of κ , it is substantially away from zero.
Thus, the AR method fails to completely remove the sensitive/spurious concept completely from the latent representation.
Fig 5c shows the possible reason using the sensitive concept probing classifier accuracy for ERM and AR.
The probe accuracy after adversarial training doesn’t decrease to 50% but stops at accuracy proportional to the predictive correlation κ .
This is expected since even if the AR would have been successful in removing the sensitive feature, the main-task feature would still be predictive of sensitive concept label by κ due to the spurious correlation between them.
However, the converse is not true: an accuracy of κ does not imply that sensitive concept is fully removed.
The results substantiate the first statement of Theorem 2.3: given two representations where one ( desired ) does not have concept features while the other ( undesired ) contains the sensitive concept features, the undesired one may be better for the main task loss even as both may have the same probing accuracy.
Fig 5d, 5e and 5f show the spuriousness score of AR in comparison to classifier trained with ERM on Twitter-PAN16, Twitter-AAE and Synthetic-Text datasets respectively.
The failure of AR is worse here: there is no significant reduction in spuriousness score for AR in comparison to ERM.
In case of Synthetic-Text dataset, ERM has zero spuriousness score but AR have non-zero score.
We expand more on this observation in Suppl I.3.
Comparison with previous work.
Our experimental results extend the observations from previous work on adversarial removal failure (Elazar & Goldberg, 2018).
If post-removal the latent representation used by the main-task classifier is still predictive of the removed concept, (Elazar & Goldberg, 2018) claimed it as a failure of the removal method.
However, this claim may not be correct since a feature could be present in the latent space and yet not used by model (Ravichander et al., 2021).Therefore, we propose the spuriousness score for the main classifier as a metric to verify if a feature is being used.
We also confirm the observation from (Elazar & Goldberg, 2018) that adversarial removal can fail even if there is no correlation between concept and task label ( κ = 0 . 5 ), and extend their work by showing failure modes on multiple values of κ .
For more details on verifying this observation with other metrics and other ablations, see Suppl I.3.
Ablations.
In Suppl.
, we report results on using BERT instead of RoBERTa as the input encoder (Suppl.
I.2, I.3), the effect of using different modeling choices like loss-function, regularization, e.t.c.
(I.4), and the behavior of probing classifiers when concept is not there in latent space (I.1).
H. Experimental Setup H.1. Dataset Description
As described in Sec G, we demonstrate the failure of Null-Space Removal (Sec G.2) and Adversarial Removal (Sec G.3) in removing the undesired concept from the latent representation on three real world datasets: MultiNLI (Williams et al., 2018),
Twitter-PAN16 (Rangel et al., 2016) and Twitter-AAE (Blodgett et al., 2016); and a synthetic dataset, Synthetic-Text.
The detailed generation and evaluation strategies for each dataset are given below.
MultiNLI Dataset.
In MultiNLI dataset, given two sentences—premise and hypothesis—the main-task is to predict whether the hypothesis entails , contradicts or is neutral to the premise.
As described in Sec G, we simplify it to a binary task of predicting whether a hypothesis contradicts the premise.
The binary main-task label, y m = 1 when a given hypothesis contradicts the premise otherwise it is -1.
That is, we relabel the MNLI dataset by assigning label y m
= 1 to examples with contradiction labels and y m = − 1 to the example with neutral or entailment label.
It has been reported that the contradiction label is spuriously correlated with the negation words like nobody,no,never and nothing (Gururangan et al., 2018b).
Thus, we created a ‘negation’ concept denoting the presence of these words in the hypothesis of a given (hypothesis, premise) pair.
The concept-label y p = 1 when the negation -concept is present in the hypothesis otherwise it is − 1 .
The standard MultiNLI dataset has approximately 90% of example in training set, 5% as publicly available development set and the rest of 5% in a separate held out validation set accessible through online competition leader-board not accessible to the public.
Thus, we create our own train and test split by subsamping 10 k examples from the initial training set, converting it into binary contradiction vs. non-contradiction labels, labelling the negation-concept label, and splitting them into 80train and test split.
For pre-training a clean classifier that does not use the spurious-concept, we create a special training set following the method described in Sec H.2.
For evaluating the robustness of both null-space and adversarial removal methods, we create multiple datasets with different predictive-correlation as described in Sec H.3 .
Twitter-PAN16 Dataset.
In Twitter-PAN16 dataset (Rangel et al., 2016), following (Elazar & Goldberg, 2018), given a tweet, the main task is to predict whether it contain a mention to another user or not.
The dataset contains manually annotated binary Gender information (i.e Male or Female) of 436 Twitter user with atleast 1k tweets each.
The Gender annotation was done by assessing the name and photograph of the LinkedIn profile of each user (Elazar & Goldberg, 2018).
The unclear cases were discarded in this process.
We consider “Gender” as a sensitive concept that should not be used for main-task prediction.
The dataset contains 160k tweets tweets for training and 10k tweets for test.
We merged the full dataset, subsampled 10k examples, and created a 80-20 train and test split.
For pre-training a clean classifier, we create a special training set following the method described in Sec H.2.
To generate datasets with different predictive correlation, we follow the method from H.3.
The dataset is acquired and processed using the code 2 made available by the (Elazar &
Goldberg, 2018).
Twitter-AAE Dataset.
In Twitter-AAE dataset (Blodgett et al., 2016), again following (Elazar & Goldberg, 2018), the main-task is to predict a binary sentiment (Positive or Negative) from a given tweet.
The dataset contain 59.2 million tweets by 2.8 million users.
Each tweet is associated with “race” information of the user which is labelled by taking both words in tweet and geo-location of user into account.
We consider “race” as sensitive concept which should not be used for the main task of sentiment prediction.
Following (Elazar & Goldberg, 2018), we use the AAE (African America English) and SAE
(Standard American English) as a proxy for non-Hispanic blacks and non-Hispanic whites race.
Again, we subsampled 10k examples with 80-20 split from the dataset and followed the method described in Sec H.2 and H.3 to generate a clean dataset for pre-training a clean classifier and generating a dataset with different predictive correlation respectively.
Synthetic Dataset.
To accurately evaluate the whether a classifier is using the spurious concept or not, we introduce a
Synthetic-Text dataset where it is possible to change the text input based on the change in concept (thus implementing
Def 2.1).
The main-task is to predict whether a sentence contains a numbered word (e.g. one, fifteen etc) or not, and the spurious concept is the length of the sentence which is correlated with the main task label.
To create a sentence with numbered words, we randomly sample 10 words from the following set and combine them to form the sentence.
Numbered Words = one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen, sixteen, seventeen, eighteen, twenty, thirty, forty, fifty, sixty, seventy, eighty, ninety, hundred, thousand otherwise 10 non-numbered words randomly sampled from the following set were added to sentence :
Non-Numbered Words = nice, device, try, picture, signature, trailer, harry, potter, malfoy, john, switch, taste, glove, balloon, dog, horse, switch, watch, sun, cloud, river, town, cow, shadow, pencil, eraser
Next, we introduce the spurious concept (length) by increasing the length of the sentences which contain numbered words.
We do so by adding a special word “pad” 10 times.
In our experiment we used 1k examples created using the above described method and create 80-20 split for train and test set.
Again, we follow the method described in Sec H.2 and H.3 to generate a clean dataset for training a clean classifier and to generate datasets with different predictive correlations respectively.
H.2. Creating a “clean” dataset with no spurious correlation with main-label
To construct a new dataset with no spurious correlation between the main-task and the concept label, we subsample only those examples from the the given dataset which have a fixed value of the spurious-concept label ( y p ) .
Thus, if we train main-task classifier using this dataset, it cannot use the spurious-concept since they are not discriminative of the main task label (Ravichander et al., 2021).
In MultiNLI dataset we select only those examples which have no negation words in the sentence for the main-task for creating a clean dataset.
Similarly, for Twitter-PAN16 dataset, we only select those examples which have gender label y p = − 1 in the processed dataset. .
And for Twitter-AAE dataset, we only select those examples which have non-Hispanic whites race label.
H.3. Creating datasets with spurious correlated main and concept label
Since in our experiment, both the main-task label ( y m ) and concept label ( y p ) are binary ( − 1 or 1 ), they divide the dataset into 2 × 2 subgroups for each combination of ( y m , y p ).
In MultiNLI dataset, the contradiction label ( y m = 1) is correlated with the presence of negation words y p
= 1 , this implies that the not-contradiction label y m = − 1 is also correlated with absence of negation words in the sentence y p = − 1 .
Thus, the input example with ( y m = 1 , y p = 1) and ( y m = − 1 , y p = − 1) form the majority group, henceforth referred as S maj while groups ( y m = 1 , y p = − 1) and ( y m = − 1 , y p = 1) forms the minority group S min .
To evaluate the robustness of the removal methods, we create multiple datasets with different predictive correlation ( κ ) between the two labels y m and y p where κ = P ( y m · y p ) > 0 as defined in Sec G.
In other words, to create a dataset with a particular predictive correlation κ , we vary the size of S maj and S min .
More precisely, the predictive correlation can be equivalently defined in terms of the size of the these groups as:
[Equation]
Similarly for Twitter-PAN16, Twitter-AAE and Synthetic-Text dataset, the we artificially create spurious correlation between y m and y p by creating the S maj and S min to have the desired predictive correlation ( κ ).
H.4. Encoder for real datasets
For all the experiments on real datasets in Sec G we used RoBERTa as default encoder h .
In Sec I, we report the results when using BERT instead of RoBERTa as input encoder.
RoBERTa We use the Hugging Face(Wolf et al., 2019) transformers implementation of RoBERTa(Liu et al., 2019) roberta-base model, starting with pretrained weights for encoding the text-input to latent representation.
We use default tokenizer and model configuration in our experiment.
BERT We use the Hugging Face(Wolf et al., 2019) transformers implementation of BERT(Devlin et al., 2019) bert-baseuncased model, starting with pretrained weights for encoding the text-input to latent representation.
We use default tokenizer and model configuration in our experiment.
For both BERT and RoBERTa, the parameters of the encoder were fine-tuned as a part of training the main-task classifier for null-space removal and then frozen.
For adversarial removal, the encoder, main-task classifier and the adversarial probing classifier are trained jointly.
For both BERT and RoBERTa, we use the pooled output ( [ CLS ] token for BERT) from the the model, as the latent representation and is given to main-task and probing classifier.
H.5. Encoder for synthetic Dataset
nBOW: neural Bag of Word.
For Synthetic-Text dataset, we use sum of pretrained-GloVe embedding(Pennington et al., 2014) of the words in the sentence to encode the sentence into latent representation.
We used Gensim (Rehurek &
Sojka, 2011) library for acquiring the 100-dimensional GloVe embedding ( glove-wiki-gigaword-100 ).
Throughout all our experiments, the word embedding was not trained.
Post encoding, the latent representation were further passed through hidden layers consisting of a linear transformation layer followed by relu (Agarap, 2018) non-linearity.
We will specify how many such hidden layers were used when discussing specific experiments in Sec I.
The hidden layer dimensions were fixed to 50 dimensional space.
H.6. Null-Space Removal Experiment Setup
For null-space removal (INLP) experiment on both real and synthetic dataset the following procedure is followed:
Pretraining Phase: A clean pretrained main-task classifier is trained using the clean dataset obtained by method described in Suppl H.2.
This is to ensure that the main-task classifier does not use the spurious feature, so that the INLP method doesn’t have any effect on the main-task classifier.
The main-task classifier is a linear-transformation on the latent-representation provided by encoder followed by softmax layer for prediction.
Both the encoder and main-task classifier is fine-tuned during this process.
Removal Phase: Both the encoder and main-task classifier is frozen (made non-trainable).
Next, a probing classifier is trained from the latent representation of the encoder (refer Suppl H.4 and H.5 for more details about encoder).
The probing classifier is also a linear transformation layer followed by softmax layer for prediction.
The main-task classifier and encoder in the pretraining phase and the probing classifier in the removal phase is trained using cross-entropy loss for both real and synthetic datasets.
For the real dataset, a fixed learning rate of 1 × 10 − 5 is used when RoBERTa is used as encoder and 5 × 10 − 5 when using BERT as encoder.
For synthetic experiments, a fixed learning rate of 5 × 10 − 3 is used when training both the nBOW encoder and main-task classifier in the pretraining stage and probing classifier in removal stage.
H.7. Adversarial Removal Experiment Setup
For adversarial removal (AR) experiment, for both real and synthetic datasets, first the input text is encoded to latent representation using the encoder (Suppl H.4 and H.5).
Then for the main-task classifier a linear transformation layer followed by softmax layer is applied for the main-task prediction.
The same latent representation output from encoder is given to the probing classifier which is a separate linear transformation layer followed by softmax layer.
All components of the model, encoder, main-task classifier and probing classifier are trained using the following modified objective from Eq 87:
[Equation]
where h is the encoder, c m is the main task classifier,
c p is the probing classifier, g − 1 is the gradient reversal layer with fixed reversal strength of − 1 .
The first term in the objective is for training the main task classifier and the second term is the adversarial objective for training the probing classifier using gradient reversal method (Ganin & Lempitsky, 2015; Elazar & Goldberg, 2018) .
The hyperparameter λ controls the strength of the adversarial objetive.
In our experiment we very λ ∈ { 0 . 00001 , 0 . 0001 , 0 .
When describing the experimental results in Sec I.3 we choose the λ which performs the best for all datasets with different predictive correlation κ in removing the undesired concept from the latent representation.
H.8. Metrics Description
For simplicity, in all our experiments we assume that both the main and the correlated attribute label are binary.
We measure the degree of spuriousness using the following two metrics:
Spuriousness Score:
As defined in Sec F, this metric help us quantify, how much a classifier is using the spurious feature.
∆ Probability
: In Synthetic-Text dataset as described in H.1, we have the ability to change the input corresponding to
the change
in concept label (thus implementing Def 2.1) .
Thus we could measure if the main-classifier is using the spurious-concept by changing the concept in the input and measuring the corresponding change in the maintask classifier’s prediction probability.
Higher the change in prediction probability higher
the
main-task classifier
is dependent on spurious-concept.
I. Additional Results I.1. Probing classifier Quality
Fig 6 shows different failure modes of the probing classifier.
In Fig 6a and 6b, a clean main-task classifier which doesn’t use the concept feature is trained on Synthetic-Text and MultiNLI dataset respectively using the method described in Sec H.2.
Thus the latent representation doesn’t have the concept feature.
Then, to test the presence of concept-causal feature in the latent representation we train a probing classifier to predict concept-label.
The first row show the accuracy of the probing classifier for testing the presence of concept in latent space.
When κ = 0 .
5 i.e no correlation between the main-task and the concept label, the probing accuracy is approximately 50% which correctly shows the absence of the concept-causal feature in the latent representation.
The accuracy increases as the correlation κ between the main and concept-causal feature increases in dataset.
This shows that even when concept-causal feature is not present in the latent representation, probing classifier will still claim presence of concept-causal feature if any correlated feature (main-task feature in this case) is present in the latent space.
In Fig 6c, the latent space contains the concept-causal feature as shown by accuracy of approximately 94.5% when κ = 0 .
But as κ increases the probing classifier’s accuracy increases in the presence of correlated main-task feature which falsely increases the confidence of presence of the concept-causal feature.
The second row shows the spuriousness-score of concept-probing classifier is increasing as the correlation between the main-task and concept-causal feature increases which implies that the probing classifier is using relatively large amount of correlated main-task feature for concept-label prediction in all settings.
I.2. Extended Null-Space Removal Results
Fig 7 and 8, shows the failure mode of null-space removal (INLP) in the real dataset when using RoBERTa and BERT as encoder respectively.
Different columns of the figure are for three different real datasets — MultiNLI, Twitter-PAN16, and
[Figure]
Probing classifier:
The first row in Fig 6a and 6b shows that even when the latent representation doesn’t contain the probing concept-causal feature, the probing classifier is still has >50% accuracy when other correlated feature is present.
The accuracy increases as the correlation κ between the probing concept-causal feature and other correlated feature increases.
The first row Fig 6c shows, that presence of correlated feature could increase the probing classifier’s accuracy thus increasing the confidence in the presence of concept-causal feature in latent representation.
The second row of all the figures shows that the probing classifier is getting more spurious as the κ increases thus implying that the probing classifier is using some other correlated feature than concept-causal feature.
For more discussion see Suppl I.1.
Twitter-AAE respectively.
The x-axis from step 8-26 are different INLP removal steps.
The y-axis shows different metrics to evaluate the main-task and probing classifier.
Different colored lines shows the spurious correlation ( κ ) in the probing dataset used by INLP for removal of spurious-concept.
The pretrained classifier is clean, i.e., does not use the spurious concept-causal feature; hence INLP shouldn’t have any effect on main-classifier when removing concept-causal feature from the latent space.
The first row shows that as the INLP iteration progresses, the norm of latent representation, which is being cleaned of concept-causal feature, decreases.
This indicates that some features are being removed.
However, the results are against our expectation from the second statement of Theorem 2.2, which states that the norm of classifier will tend to zero as the INLP removal progresses.
The possible reason is that, from Theorem 2.2 the norm of latent representation will go to zero when the the latent representation only contains the spurious concept-causal feature and the other features correlated to it.
But, the encoder representation could have other features which are not correlated with concept-label and hence not removed.
Since, the pretrained classifier given for INLP was clean (using method described in Sec H.2), we do not expect the INLP to have any effect on the main-task classifier.
The second row in Fig 7 and 8 shows that the main-classifier accuracy drops to random guess i.e 50% except for the case when probing dataset have κ = 0 .
5 i.e no correlation between the main and concept label.
Thus INLP method corrupted a clean-classifier and made it useless.
The reason behind this could be observed from the fourth and fifth row.
The fourth row show the accuracy of the probing classifier before the projection step.
We can see that at step 8 on x-axis κ = 0 .
5 , the probing classifier correctly has accuracy of 50% showing that the concept-causal feature is not present in the latent representation.
But for other value of κ , the probing classifier accuracy is proportional to value of κ implying that the probing classifier is using the main-task feature for its prediction.
Hence at the time of removal, it removes the main-task feature which leads drop in the main-task accuracy.
This can also be verified form the last row of Fig 7 and 8, which shows that the spuriousness score of probing classifier is high; thus it is using the main-task feature for its prediction.
We observe similar results for Synthetic-Text dataset when using INLP in Fig 9.
For all the INLP experiment on Synthetic-Text dataset, there were no hidden layers after the nBOW encoder (see Sec H.5).
I.3. Extended Adversarial Removal Results
Adversarial removal failure in real-world datasets.
Fig 10 shows the failure mode of adversarial removal AR on real-world datasets.
In x-axis we vary the predictive correlation κ between the main and the concept-label in different datasets and measure the performance of AR on different metrics on the y-axis.
The second row shows the spuriousness score of the main-task classifier after AR as we vary κ on the x-axis.
When using RoBERTa as the encoder, the orange curve in second row shows the spuriousness score of the main-task classifier when trained using the ERM loss.
The spuriousness score describes how much unwanted concept-causal feature the main-task classifier is using.
The blue curve shows that the AR method reduces the spuriousness of main-task though cannot completely remove it.
The reason for this failure can be attributed to probing classifier.
Even when AR has successfully removed the unwanted concept feature, the accuracy of concept-probing classifier will be proportion to κ due to presence of correlated main-task feature in the latent space.
This can be seen in the third row of the of Fig 10.
Thus we cannot be sure if the unwanted concept-causal feature has been completely removed from the latent space or just became noisy enough to have accuracy proportional to κ after AR converges.
In Fig 10, for each dataset and encoder, we manually choose the hyperparameter described λ described in Sec H. which reduces the spuriousness score most for the main-task classifier while not hampering the main-task classifier accuracy.
In Fig 11, we show the trend in spuriousness score is similar for all choices of hyperparameter λ in our search.
No value of λ is able to completely reduce the spuriousness score to zero.
Adversarial removal makes a classifier unfair.
Fig 12 shows that when the adversarial classifier is initialized with a clean main-task classifier which doesn’t use unwanted-concept causal features, it makes matters worse by making the main-task classifier use the unwanted-concept feature.
For the Synthetic-Text dataset, since the word embedding are non-trainable, one single hidden layer is applied after the nBOW encoder so that AR methods could remove the unwanted-concept feature from the new latent representation.
I.4. Synthetic-Text dataset Ablations
Adversarial Removal Failure in Synthetic-Text dataset: Figure 13 shows the failure of AR on the synthetic dataset as we vary the noise in the main-task label and unwanted concept-label.
For the experiment, since the words embedding are non-trainable, one single hidden layer is applied after the nBOW encoder so that AR methods could remove the unwanted-concept feature from the new latent representation.
Dropout Regularization Helps AR method: Continuing on observation from Fig 14a, 14b and 14c shows the ∆ -Prob of the main-task classifier after we apply the AR on Synthetic-Text dataset and how they changes as we increase the dropout regularization.
As we increase the dropout (drate in the figure), the ∆ -Prob of the main classifier decreases showing that the regularization methods could help improve the the removal methods.
[Table]
Sp u i o u
[Table]
ss
[Table]
[Figure]
[Table]
M a i n -
[Table]
u r i o u
[Table]
n
[Table]
cc
[Table]
A
[Table]
ss
[Table]
[Figure]
[Table] [Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
, how much sensitive the main-task classifier is wrt.
to concept feature.
We observe that the ∆ -Prob increases in the middle of INLP showing that the main-classifier which was not using the concept initially (as in iteration 21), started using the sensitive concept because of INLP removal.
Thus stopping INLP prematurely could lead to a more unclean classifier than before whereas running INLP longer removes all the correlated feature and could make the classifier useless.
For more discussion see Suppl I.2.
[Table]
a i n
C l a ss i f
i e r A cc u r a c y
[Table] [Table]
a i n
[Table]
u r i o u s n
[Table]
ss
S c
o r
[Table]
P r o b e
A cc u r a c y ( A
[Table]
[Figure]
u
[Figure] [Figure]
M a i n
C l a ss i f
i e r
A cc u r a c y (
A ll )
[Figure]
M a i n
Sp u
r
i
o u s n e ss S
c
o r e
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
s k
[Figure]
cc u r a c y ( A ll )
Dataset: MultiNLI
[Figure] [Figure]
[Figure] [Figure]
M a i n - Sp u r i o u s n e ss
S c
o r e
[Figure]
=0.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
S c
[Figure]
r e
[Figure] [Figure]
[Table]
r i
[Table]
M a i n - P r o b a b ili t y ( a ll )
[Table] [Table]
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]