Probing Classifiers are Unreliable for Concept Removal and Detection
Anonymous Author(s)
AffiliationAddress
Abstract
Neural network models trained on text data have been found to encode undesiredlinguistic or sensitive attributes in their representation.
Removing such attributes isnon-trivial because of a complex relationship between the attribute, text input, andthe learnt representation.
Recent work has proposed post-hoc and adversarial methods to remove such unwanted attributes from a model’s representation.
Throughan extensive theoretical and empirical analysis, we show that these methods canbe counter-productive: they are unable to remove the attributes entirely, and inthe worst case may end up destroying all task-relevant features.
The reason is themethods’ reliance on a probing classifier as a proxy for the attribute.
Even underthe most favorable conditions when an attribute’s features in representation spacecan alone provide 100% accuracy for learning the probing classifier, we prove thatpost-hoc or adversarial methods will fail to remove the attribute correctly.
Thesetheoretical implications are confirmed by empirical experiments on models trainedon synthetic, Multi-NLI, and Twitter datasets.
For sensitive applications of attributeremoval such as fairness, we recommend caution against using these methods andpropose a spuriousness metric to gauge the quality of the final classifier.
Introduction
Neural network classifiers built using text data have been shown to learn spuriously correlatedfeatures [12, 19] or embed sensitive attributes like gender or race [7, 6, 5] at their representationlayer.
Classifiers that use such sensitive or spurious concepts (henceforth concepts ) raise concernsof model unfairness out-of-distribution generalization [28, 2, 11].
Because the classifiers are basedon hard-to-interpret deep neural networks, removing the influence of these concepts is non-trivial.
Since many concepts cannot be modified at the input tokens level, removal methods that work atthe representation layer have been proposed: post-hoc removal [6, 36, 10] on a pre-trained model(e.g., null space projection [23]) and adversarial removal [11, 35, 8] by jointly training the main taskclassifier with an (adversarial) classifier for the concept.
In this paper, we theoretically show that both classes of methods can be counter-productive in realworld settings where the main task label is often correlated with the concept.
Examples includenatural language inference (main task) where the presence of negative words (spurious feature) maybe correlated with the “contradicts” label; or tweet sentiment classification (main task) where theauthor’s gender (sensitive feature) may be correlated with the sentiment label.
Our key result is basedon the observation that both these methods internally use an auxiliary (or probing) classifier [1, 30]that aims to predict the spurious feature based on the representation learnt by the main classifier.
We show that the auxiliary classifier cannot be a reliable signal on whether the representation includesfeatures that are causally derived from the concept.
As previous work has argued [3, 32, 29, 2], ifthe representation features causally derived from the concept are not predictive enough, the probingclassifier for the concept can be expected to rely on correlated features to obtain a higher accuracy.
However, we show a stronger result by considering a setting where there is no potential accuracy gainand
the concept’s features are easily learnable.
Namely, even when the concept’s causally-relatedfeatures alone can provide 100% accuracy and are linearly separable from other features , the probingclassifier may still learn non-zero weights for the main-task relevant features.
Based on this result,under some simplifying assumptions, we prove that both post-hoc and adversarial training methodscan fail to remove the undesired concept, remove useful task-relevant features in addition to theundesired concept, or do both . As an extreme case, we show how post-hoc removal methods canlead to a random main-task classifier, removing all task-relevant information from the representation.
Empirical results on four datasets—natural language inference, sentiment analysis, tweet-mentiondetection and synthetic-text classification—confirm our claims.
Across all datasets, as the correlationbetween the main task and the concept increases, we find that null space removal removes a higheramount of the main-task features, eventually leading to a random-guess classifier.
In particular, fora pre-trained classifier that does not use the concept at all, the method modifies the representationto yield a lower main-task accuracy, irrespective of the correlation between the main task and theconcept.
For the adversarial removal method, we find a similar result that adversarial classifier doesnot remove all concept-related features.
For most datasets, the concept-related features left within theadversarial classifier’s representation is comparable to that for a standard main-task classifier.
Forpractical usage of the removal methods, we propose a spuriousness metric that can be used to gaugethe quality of the final classifier.
Our theoretical analysis complements empirical critique of the adversarial removal method [8].
It alsoextends the literature on probing classifiers and their unreliability [3].
Adding to known limitations ofstandard methods [14, 26] that evaluate accuracy of a probing classifier, our results show that recentcausally-inspired methods like amnesic probing [9] are also flawed because they depend on access toa good quality classifier for the concept.
Our contributions include:
• Theoretical analysis of null space and adversarial removal methods showing that they fail to removean undesirable concept from a model’s representation, even under favorable conditions.
• Empirical results on four datasets showing that the methods are unable to remove concept-relatedfeatures fully and end up unnecessarily removing task-relevant features.• A practical spuriousness score motivated by theory for evaluating the output of removal methods.
Concept removal: Background and problem statement
For a classification task, let ( x i , y im ) ni
=1 be set of examples in the dataset D m , where x i ∈
X is theinput feature and y im ∈ Y m the label.
We call this as the main task and label y
im as the main task label.
The main task classifier can be written as c m ( h ( x )) where h : X → Z is an encoder mapping theinput X to the latent representation Z and c m : Z → Y m is the classifier on top of the representation
Z .
Additionally, we are given labels for a spurious or sensitive concept, y ip ∈ Y p , and our goal is toensure that the representation h ( x ) learnt by the main classifier does not include features causallyderived from the concept.
Below we define what it means to be “causally derived”: the representationshould not change under an intervention on concept.
Definition 2.1.
( concept-causal feature )
A feature Z j ∈ Z (jth dimension of h ( x ) ) at the representation layer is said to be causally derived from a concept (concept-causal for short) if upon changingthe value of the concept label,
the corresponding change in the input’s value x will lead to a changein the feature’s ( Z j ) value.
A concept removal algorithm is said to be successful if it produces a clean representation h ′ ( x ) to beused by the main-classifier that has no concept-causal features.
If the representation does not containsuch features, the main classifier cannot use them [8].
Existing concept removal methods: Adversarial training and null space projection.
Whenthe text input can be changed based on changing the sensitive property’s value, methods like dataaugmentation [16] have been proposed for concept removal.
However, for most sensitive or spuriousproperties, it is not possible to know the correct change to apply at the input level corresponding to achange in the property’s value.
Hence, the above approaches cannot be applied directly.
Instead, methods based on the representation layer have been proposed.
To determine whetherfeatures in a representation are causally derived by the concept, these methods train an auxiliary, probing classifier c p : Z → Y p where y ip ∈ Y p is the label of the concept we want to remove fromthe latent space z i ∈
Z .
The accuracy of the classifier denotes the predictive information about theconcept embedded in the representation.
This probing classifier is then used to remove the sensitiveconcept from the latent representation which will ensure that the main-task classifier cannot use them.
Two kinds of feature removal methods have been proposed: 1) post-hoc methods such as null spaceremoval ([23]), where removal is done after the main-task classifier is trained; 2) adversarial methodsthat jointly train the main task with the probing classifier as the adversary (e.g., [11, 35, 24, 25]).
Empirical results, however, have cast doubt on whether these methods (in particular, adversarial) canfully remove the sensitive concept from the model’s representation [8].
In this paper, we complementsuch empirical work through a theoretical analysis of both adversarial and post-hoc removal methods.
We also empirically investigate the null space method, and extend the analysis on adversarial methodsby characterizing the effect of correlation between the main task and concept labels.
Attribute removal using probing classifier can be counter-productive
As mentioned above, both removal methods internally use a probing classifier as a proxy for theconcept’s features.
In Section 3.1, we start off by showing that for any classification task be itprobing or main-task classification, it is difficult to learn a clean classifier which doesn’t use anyspuriously correlated feature (Lemma 3.1).
Hence the key assumption driving the use of predictiveclassifiers within both removal methods is incorrect.
Next in Section 3.2 and 3.3, we will show howthese individual components’ failure leads to overall failure of both removal methods.
Finally, in Section 3.4, we propose a practical spuriousness score to assess the output classifier from the removalmethods.
Throughout this section, we assume that both the main task label y m and probing tasklabels y p is binary taking value from {− 1 , 1 } and there is a basic, fixed encoder h converting the textinput to features in the representation space (e.g., it can a pre-trained model like BERT).
3.1 Problem with learning a clean classifier: Main and Probing Classifier
Given z = h ( x ) and the concept label y p , the probing task is to learn a classifier c p ( z ) such thatit only uses the concept-causal features and the accuracy for y p is maximized.
If there are featuresin z outside concept-causal that help improve the accuracy for the classifier, a classifier trained onstandard losses such as cross-entropy or max-margin is expected to use those features too, as arguedin the probing literature [14, 3].
In this section, however, we create a setting that is the most favorablefor the classifier to use only concept-causal features—no accuracy gain on using features outsideof concept-causal, disentangled representation so that no further representation learning is required,and class labels that are linearly separable using concept-causal features—yet we find that a trainedclassifier would use non-concept-causal features.
Specifically, we make the following assumptions.
Assumption 3.1 (Disentangled Frozen Latent Representation) .
The latent representation z is disentangled and is of form [ z m , z p ] , where z p ∈ R d p are the features causally derived from the conceptlabel and z m ∈ R d m are the features causally derived from the main task label along with rest of thefeatures in latent representation.
Here d m and d p are the dimensions of z m and z p respectively.
Alsothe encoder h ( x ) which maps the input X to latent representation Z is frozen or non-trainable.
Assumption 3.2 (concept-feature Linear Separability) .
The concept-features of the latent representation ( z p ) for every point are linearly separable/fully predictive for the concept labels y p , i.ey i · (ˆ ϵ p · z ip + b p ) ≥ 1 for all datapoints ( x i , y i ) for some ˆ ϵ p ∈ R d p .
For the case of zero-centeredlatent space we have b p = 0 .
In addition, we assume that there is some correlation between the concept and main-task label: afunction using z m may also be able to classify correctly some non-empty subset of points.
Assumption 3.3 (Spurious Correlation) .
For a subset of training points S ⊂ D p , z m linearlyseparable with respect to concept label y p i.e y i p · ( ˆ ϵ m ·
z i m + b m ) ≥ 1 for some ˆ ϵ m ∈ R d m andb m ∈ R .
For the case of zero-centered latent space we have b m = 0 .
Now we are ready to state the key lemma which will show that for only a few special points ifnon-concept-features
z m are linearly separable then the probing classifier c p ( z ) will use thosefeatures.
Following [20], for ease of exposition we assume max-margin as training loss. Under some [Figure] M
[Figure] m [Figure] mild conditions on separable data, a classifier trained using logistic/exponential loss converges tomax-margin classifier given infinite training time [31, 15].
Lemma 3.1.
Let the latent representation be frozen and disentangled
such that z
=
[ z m , z p ](Assumption 3.1), where concept-features z p be fully predictive (Assumption 3.2).
Let c ∗ p ( z )
=w p · z p be the desired clean/purely-invariant linear classifier trained using max-margin objective(Appendix C.1) which only uses z p for its prediction.
Let z m be the spurious feature s.t.
for the marginpoints of c ∗ p ( z ) , z m be linearly-separable w.r.t.
task label y p (Assumption 3.3).
Then, assuming thelatent space is centered around 0 , the concept-probing classifier trained using max-margin objectivewill be of form c p ( z ) =
w p · z p + w m · z m where w m ̸ = 0 .
Proof Sketch.
We can perturb c ∗ p ( z ) to have w m ̸ = 0 and show that there always exist some perturbedclassifier which have bigger margin than c ∗ p ( z ) .
For detailed proof see Appendix C.2.
Our result shows that not just accuracy, even geometric skews in the dataset can yield an incorrectprobing classifier.
In Appendix C.3 we prove that the assumptions for Lemma 3.1 are both sufficientand necessary for a classifier to use non-concept-features z m when z p is 1-dimensional.
Lemma 3.1generalizes a result from [20] by using fewer assumptions (we do not restrict z m to be binary, do notassume that z m and z p are conditionally independent given y , and do not assume monotonicity ofclassifier norm with dataset size).
Since we did not make any assumptions specific to concept-features,the same result is applicable for main-task classifier, as shown in Appendix C.4.
3.2 Failure mode of post-hoc removal methods: Null-space removal (INLP)
The null space method [23, 9], henceforth referred as INLP , removes a concept from latent space byprojecting the latent space to a subspace that is not discriminative of that attribute.
First, it estimatesthe subspace in the latent space discriminative of the concept we want to remove by training a probingclassifier c p ( z i ) → y i p , where y p is the concept label.
Then the projection is done on the null-spaceof this probing classifier which is expected to be non-discriminative of the concept.
[23] used a linearprobing classifier ( c p ( z )) to ensure that the any linear classifier cannot recover the removed conceptfrom modified latent representation z ′ and hence the main task classifier ( c m ( z ′ ) ) also becomeinvariant to removed attribute.
Also, they recommend running this removal step for multiple iterationto ensure the unwanted concept is removed completely.
Details for the method are in Appendix D.1.
Failure Mode: We use a 2-D latent feature representation to illustrate how null space removal canfail.
One of the dimensions correspond to profession (main task) and the other gender.
The inputpre-trained classifier is clean, i.e., it uses only the profession feature for its prediction (Figure 1a).
Next we apply the null-space removal to remove the gender concept.
From Lemma 3.1, the obtainedgender-probing classifier can also use profession feature, as shown by the projection direction—null-space of the gender classifier—being slanted (Figure 1a).
As a result, Figure 1b shows thatany two points (green points numbered 1) having the same profession feature but different genderfeature, after projection to the null-space (slanted line, shown by number 2), will have their professionfeature reversed (shown by number 3).
Even for a clean classifier, we see that null-space projectioncorrupts the task-specific features and thus the null-space projection step made a clean/fair classifierunclean/unfair.
Note that the main-task classifier’s accuracy will drop, affecting INLP based probing methods [9] that interpret a drop in main classifier’s accuracy after projection as evidence that thesensitive concept was being used.
Next we formalize these observations using z i ( k ) to denote therepresentation
z i after k steps.
Theorem 3.2 (INLP Failure Mode) .
Let the latent representation z satisfies Assumption 3.1 andc m ( z ) is the pre-trained main-task classifier and c p ( z ) be the probing classifier used by INLP toremove the unwanted feature z p from the latent representation.
Let the assumptions of Lemma 3.1 besatisfied for the probing classifier c p ( z ) which is trained using max-margin objective.
Then,
Mixing: After the first projection step of linear-INLP, the dimensions of z will get mixed suchthat z i (1) =
[ g ( z im , x ip ) , f ( z ip , z im )]
̸ =
[ z im , z ip ] .
Additionally, this mixing is non-invertible with subsequent projection steps of INLP.
Removal: The L2-norm of the latent representation z decreases with every projection step aslong as the the probing classifier at a step does not lie completely in the null-space of previousprobing classifiers.
Thus, after sufficiently many steps, INLP can destroy all information in therepresentation, z i ( ∞ ) =
Proof Sketch.
From Lemma 3.1, probing classifier will have non-zero weight on the spurious featurez m .
Hence, after projection operation they get mixed up and we show projection operator is noninvertible.
Then we show projection operation is lossy i.e removes some norm of z when null-spaceof c p ( z ) is not orthogonal to z .
For detailed proof see Appendix D.2.
3.3 Failure mode of adversarial removal methods
To remove the unwanted attribute z p from the latent representation, the adversarial removal methodjointly trains the main classifier c m ( z ) and the probing classifier c p : z → y p by specifying c p ’s lossas an adversarial loss.
For details refer to Appendix E.1.
Since the encoder h : x → z mapping the input the latent representation z is frozen ( Assumption 3.1),to allow training for the main-task classifier we introduce additional representation layers after it.
Forsimplicity in the proof, we assume a linear transformation to the latent representation h 2 : z → ζ .
After this, we have a linear main-task classifier c m : ζ → y m , as before.
The probing classifierc p : ζ → y p is trained adversarially to remove z p from latent representation ζ .
Thus, the goal ofthe adversarial method can be stated as removing the information of z p from ζ .
Let the main-taskclassifier satisfy the assumption of the generalized version of Lemma 3.1 (Appendix C.4).
In additionwe need an assumption on the hard-to-classify margin points which says that main-task labels andadversarial concept label are correlated on margin point of clean main-task classifier.
Assumption 3.4 (Label Correlation of Margin Point) .
For the margin points of clean classifier for themain task, the adversarial-probing label y p and the main task labels y m are correlated, i.e y im =
y ipfor all margins points of clean main-task classifier.
Theorem 3.3.
Let the latent representation Z satisfy Assumption 3.1, h 2 ( z ) be a linear transformation s.t.
h 2 : Z → ζ , main classifier be c m ( ζ ) =
w c m · ζ and the adversarial classifier bec p ( ζ ) =
w c p · ζ .
Let all the assumptions of Lemma 3.1 be satisfied for main-classifier c m ( · ) when using Z directly as input and Assumption 3.2 be satisfied on Z w.r.t.
adversarial-task.
Let h ∗ 2 ( z ) be thedesired encoder which is successful in removing z p from ζ .
Then there exists an undesired/incorrectencoder h 2 ( z ) such that ζ =
h 2 ( z ) =
f ( z m , z p ) for some function “ f ” and the main classifierc m ( h 2 ( z )) has bigger margin than c m ( h ∗ 2 ( z )) and,
1. has Accuracy ( c p ( h 2 ( z )) , y p ) =
Accuracy ( c p ( h ∗ 2 ( z )) , y p ) when adversarial probing classifier
c p ( · ) is trained using any learning objective.
Thus the undesired encoder h 2 ( z ) is indistinguishable
from desired encoder h ∗ 2 ( x ) in terms of adversarial task prediction accuracy but better for main -prediction
task in terms
of max -margin objective. has L ( c p ( h 2 ( z )) , y p ) >
L ( c p ( h ∗ 2 ( z )) , y p ) under Assumption 3. where L is either max margin or cross-entropy loss used for
training adversarial probing classifier
c
p Thus undesired encoderh 2 ( z ) is preferable over desired
encoder h ∗ 2 ( z ) for both main and adversarial task objective.
Proof Sketch.
First we show that w.l.o.g. ζ ∈ R .
Then the main classifier is same as h 2 ( ζ ) andadversarial classifier is α · h 2 ( ζ ) for some α ∈ R .
Then we can show that given a clean h ∗ 2 such thatζ dont contain sensitive z p , we can always perturb it to get h 2 st.
w p ̸ = 0 .
Then we show that this perturbed encoder has bigger margin for main classifier and equivalent accuracy or better loss foradversarial classifier.
For detailed proof see Appendix E.2.
Failure Mode.
We illustrate the result using the same 2D example.
The goal is to remove the genderfeature from the latent-representation shared by both gender and main-task (profession) classifier.
Assume that the shared representation is a scalar value obtained by projecting the features in somedirection.
The adversarial goal is to find a projection direction such that gender classifier obtains arandom guess accuracy 50% .
As Figure 1c shows, there can be two such projection directions thatyield 50% accuracy on gender.
While both projection directions achieve same main-task accuracy,the slanted projection direction has bigger margin than the vertical direction and is thus preferred bymain-task classifier trained using max-margin objective.
E.g., if the profession (main-task) feature ispredictive of gender upto 60%; then the final representation will have the same accuracy; we cannotbe sure if the gender feature is fully removed or is present in a corrupted form.
3.4 Extension to real-world data: A metric for quantifying degree Of spuriousness
Our theoretical analysis shows that probing-based removal methods fail to make the main taskclassifier invariant to unwanted concept.
However, to verify whether the final classifier is using theconcept or not, the theorem statements require knowledge of the concept’s features z p .
For practicalusage, we propose a metric that quantifies the degree of failure or spuriousness for both the main andprobing classifier.
For simplicity, we define it assuming that both main task and concept are binary.
Let D m,p be the dataset where for every input x i we have both the main task label y m and the conceptlabel y p .
We define 2 × 2 groups, one for each combination of ( y m , y p ) .
Without loss of generality,assume that the main-task label y m = 1 is spuriously correlated with concept label y p
= 1 andsimilarly y m = 0 is correlated with y p = 0 .
Thus, ( y m = 1 , y p = 1) and ( y m = 0 , y p = 0) are themajority group S maj while groups ( y m = 1 , y p = 0) and ( y m = 0 , y p = 1) make up the minoritygroup ( S min ).
We expect the main classifier to exploit this correlation and hence perform badlyon S min where the correlation breaks.
Following [28], we posit that minority group accuracy i.e
Acc ( S min ) can be a good metric to evaluate the degree of spuriousness .
We bound the metric bycomparing it with the accuracy on S min of a “clean” classifier that does not use the concept features.
Definition 3.1 (Spuriousness Score ( ψ )) .
Given a dataset, D m,p = S min ∪ S maj containing binarytask label and binary concept, let Acc f ( S min ) be the accuracy of the given main-task classifier( f ) on the minority group S min and Acc ∗ ( S min ) be the accuracy on the minority group of a cleanclassifier that does not use the unwanted spurious concept.
We define spuriousness score of “ f ” as:
[Equation]
To estimate Acc ∗ ( S min ) , we subsample the dataset such that y p takes a single value in the sampleand train the main classifier on it, as done by [26].
Thus, probing label y p no longer is correlated withthe main-task label y m .
There can be other ways to estimate it, e.g., by reweighting the data or usingthe accuracy on S maj .
However, we found that the former had high variance and the latter requiresan equal-noise assumption such that Acc ( S maj ) =
Acc ( S min ) on a clean main-task classifier.
Experimental Results
Theorems 3.2 and 3.3 show the failure of concept removal methods under a simplified setup andmax-margin loss.
But current deep-learning models are not trained using max-margin objectiveand might not satisfy the required assumptions (Assumption 3.1,3.2,3.3,3.4).
Thus, we now verifythe failure modes on three real-world and one synthetic datasets, without making any restrictiveassumptions.
We use RoBERTa [18]) as default encoder and fine-tune it over each real-world dataset.
For Synthetic-Text dataset we use sum of pre-trained GloVe embeddings ([21]) of words in sentenceas default encoder.
For more details of experimental setup refer Appendix F.
4.1 Datasets: Main task and spurious/sensitive concept
Real-world data.
We use three different real-world datasets: MultiNLI [34], Twitter-PAN
[22] and
Twitter-AAE [5].
In MultiNLI dataset, given two sentences—premise and hypothesis–the main taskis to predict whether hypothesis entails , contradicts or is neutral with respect to premise.
We simplifyto a binary task of predicting whether a hypothesis contradicts the premise or not. Since negation
[Figure]
a i n - T a s k A cc u r a c y ( A ll )
[Figure]
words like nobody,no,never and nothing have been reported to be spuriously correlated with with thecontradiction label [13], We create a ‘negation’ concept denoting the presence of these words.
Thegoal is to remove the negation property from the NLI model’s representation space.
In Twitter-PAN16dataset, the main task is to detect whether a tweet mentions another user or not, as in [8].
The datasetcontains gender label for each tweet, which we consider as the sensitive concept to be removed fromthe main model’s representation.
In Twitter-AAE dataset, again following [8], the main-task is topredict binary sentiment labels from a tweet’s text.
The tweets are associated with race of the author,the sensitive concept that should be removed from the main model’s representation.
Synthetic-Text.
To understand the reasons for failure, we introduce a Synthetic-Text dataset where itis possible to change the input text based on a change in concept (thus implementing Def. 2.1).
Herewe can directly evaluate whether the concept is being used by the main-task classifier by interveningon the concept (adding or removing) and observing the change in prediction scores.
The main-taskis to predict whether a sentence contains a numbered word (e.g., one, fifteen, etc. ).
We introducespurious concept (length) by increasing the length of sentences that contain numbered words.
Predictive correlation.
To assess robustness of removal methods, we create multiple datasets withdifferent predictive correlation between two labels y m and y p .
The predictive correlation ( κ ) measureshow informative one variable is for predicting the other, κ =
Pr ( y m · y p > 0) =
(cid:80) Ni
=1 1 [ y m · y p > 0] N ,where N is the size of dataset and 1 [ · ] is the indicator function which is 1 if the argument is trueotherwise 0 .
Predictive-correlation lies in κ ∈ [0 . 5 , 1] where κ = 0 .
5 means no-correlation and κ = 1means the attributes are completely correlated.
For complete description of all datasets and how wevary the predictive-correlation, refer to Appendix F; and for additional results refer Appendix G.
4.2 Results: Null Space Removal
In general, for any model given as input to INLP, it may be difficult to verify whether the removalmethod removed the correct features.
Hence, we construct a benchmark where the input classifier isclean , i.e., it does not use the concept at all.
We do so by training on a subset of data with the samevalue of the property, as in [26].
Since the input classifier does not have any concept features in itsrepresentation, we should expect that INLP should not have any effect on the main task classifier.
Eventually all task-relevant features are destroyed.
We start with the Synthetic-Text dataset bytraining a clean classifier on the main-task and input it to INLP for removing the sensitive concept.
To keep the conditions favorable for INLP, both the main task and sensitive-concept probing taskcan achieve 100% accuracy using their causally derived features respectively.
In Figure 2a, iteration21-40 shows individual steps of null-space removal.
Since, the given pre-trained classifier was cleani.e no using the sensitive features, null-space removal shouldn’t have any effect on it.
Colored linesshow different datasets used by INLP with different level of predictive correlation κ .
We observe thatfor all value of κ , the main-task classifier’s accuracy eventually goes to 50%
random guess accuracy
[Figure]
a i n
[Figure]
u r i
[Figure]
u s n e ss
[Figure] c o r e [Figure] implying that the main-task related attribute has been removed by the removal process, as predictedby Theorem 3.2.
Higher the value of correlation κ , faster the removal of main-task attribute happens.
We obtain a similar pattern over the real-world datasets, MultiNLI, Twitter-PAN16 and Twitter-AAE.
Figure 2d,2e and 2f show a decrease in the main-task accuracy to random-guess even when the inputclassifier for each dataset is ensured to be clean , thus demonstrating the failure of INLP method.
Early stopping increases the reliance on spurious features.
To avoid the full collapse, a stoppingcriteria of INLP is to stop when the main-task classifier’s performance drops [23].
In Figure 2b wemeasure the sensitivity of Synthetic-Text main-task classifier w.r.t.
to sensitive concept by changingthe feature in input sentences corresponding to sensitive concept and measuring the change in maintask classifier prediction probability ∆ Prob.
At lower iterations of INLP, the change in main-taskoutput due to change in sensitive concept’s value, ∆ Prob, is higher than that of the input classifier.
For example, for κ = 0 . 8 , the main-task classifier’s performance drops at iteration 27, but it has ahigh ∆ Prob ≈ 25% as shown in Figure 2b.
Hence it is possible that stopping prematurely will lead usto a classifier which is more unfair or reliant to sensitive concept than it was before, consistent withthe first statement in Theorem 3.2 stating that INLP will lead to mixing of features in latent space.
Failure of causally-inspired probing like Amnesic probing.
Amnesic Probing [9] declares that asensitive concept is being used by the model if, after removal of the concept from from the latentrepresentation using INLP, there is a drop in the main-task performance.
But Figure 2a, 2d, 2e and 2fshow that even when the input classifier for their corresponding main-task is clean, i.e., does not usethe sensitive concept, INLP leads to drop in performance of the main-classifier.
Hence, removal-basedmethods like Amnesic probing will falsely claim that the sensitive concept is being used.
4.3 Results: Adversarial Removal Method
We demonstrate the failure of adversarial-removal method (AR) in removing the spurious/unwantedconcept from the main classifier.
Here we compare the standard ERM training of the main classifierwith AR method over the same number of epochs (20).
We follow the training procedure of [8], andconduct a hyper-parameter sweep on the adversarial training strength to select the value which ismost effective in removing the unwanted property.
For more details, refer to Appendix F.
Cannot remove the spurious concept fully.
For MultiNLI, Figure 3b shows the spuriousness score(Definition 3.1) of ERM and AR classifiers as we vary the predictive correlation ( κ ) between themain-task label and sensitive concept label in the training dataset.
While the spuriousness score forclassifier trained using AR (blue curve) is lower than that of ERM for all values of κ , it is substantiallyaway from zero.
Thus, the AR method fails to completely remove the sensitive/spurious conceptcompletely from the latent representation.
Figure 3c shows the possible reason using the sensitiveconcept probing classifier accuracy for ERM and AR.
The probe accuracy after adversarial trainingdoesn’t decrease to 50% but stops at accuracy proportional to the predictive correlation κ .
This is expected since even if the AR would have been successful in removing the sensitive feature,the main-task feature would still be predictive of sensitive concept label by κ due to the spuriouscorrelation between them.
However, the converse is not true: an accuracy of κ does not imply thatsensitive concept is fully removed.
The results substantiate the first statement of Theorem 3.3: giventwo representations where one ( desired ) does not have concept features while the other ( undesired )contains the sensitive concept features, the undesired one may be better for the main task loss even asboth may have the same probing accuracy.
Figure 3d, 3e and 3f show the spuriousness score of ARin comparison to classifier trained with ERM on Twitter-PAN16, Twitter-AAE and Synthetic-Textdatasets respectively.
The failure of AR is worse here: there is no significant reduction in spuriousnessscore for AR in comparison to ERM.
In case of Synthetic-Text dataset, ERM has zero spuriousnessscore but AR have non-zero score.
We expand more on this observation in Appendix G.3.
Comparison with previous work.
Our experimental results extend the observations from previouswork on adversarial removal failure [8].
If post-removal the latent representation used by the maintask classifier is still predictive of the removed concept, [8] claimed it as a failure of the removalmethod.
However, this claim may not be correct since a feature could be present in the latent spaceand yet not used by model [26].Therefore, we propose the spuriousness score for the main classifier asa metric to verify if a feature is being used.
We also confirm the observation from [8] that adversarialremoval can fail even if there is no correlation between concept and task label ( κ = 0 . 5 ), and extendtheir work by showing failure modes on multiple values of κ .
For more details on verifying thisobservation with other metrics and other ablations, see Appendix G.3.
Ablations.
In Suppl. , we report results on using BERT instead of RoBERTa as the input encoder(Suppl.
G.2, G.3), the effect of using different modeling choices like loss-function, regularization,e.t.c.
(G.4), and the behavior of probing classifiers when concept is not there in latent space (G.1).
Related work
Concept removal methods.
When the removal of a concept can be simulated in input space (e.g., intabular data or simpler concepts), approaches to remove a concept include data augmentation [16],gradient regularization [27].
However, when change in a concept cannot be propagated via change ininput tokens, it becomes a non-trivial problem.
Apart from the null space and adversarial removal,past work has used explanations for the classifier’s prediction [17].
Methods like [24, 25] have triedto restrict the adversarial function to be projection operation like [23] and derive close form solutionfor concept removal for various objective.
But all of them could be clubbed under same umbrellaof adversarial removal.
Given the difficulty of building a good estimator for the features causallyderived from the concept, our work highlights the failure modes of popular removal methods.
Limitations of using a probing classifier for model interpretability.
We also contribute to thegrowing literature on the limitations of a probing classifier’s accuracy in capturing whether the mainclassifier is using the concept [3], It is known that probing classifiers capture not just the property,but any other features that may be correlated with it [28, 2, 16, 33].
As a result, many improvementshave been proposed to better estimate whether a property is being used, including the use of controllabels or datasets [14, 26].
Parallely, new causality-inspired probing methods [9] have been proposedthat compare the main task accuracy on a representation without the property, constructed using thenull space removal method.
The hope is that such improvements can make probing more robust.
Ourresults question this direction.
To demonstrate the unreliability of a probing classifier, we construct asetup that is most favorable for learning only the property’s features and still find that learned probingclassifier will include a non-zero weight for the other features too.
As we showed in 3.2 and 4.2,causally-inspired probing methods [9] have the same limitation because the removal method dependson the output of an auxiliary classifier that is equivalent to a probing classifier.
Conclusion and Discussion
Using both theoretical analysis and empirical results over real-world datasets, we presented a critiqueof recent work that claims to remove spurious or sensitive concepts from a text classifier.
Given thepractical relevance of the removal problem, we also provided an spuriousness metric to evaluate anyclassifier on its use of unwanted concepts.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 3 and
(b) Did you describe the limitations of your work?
[Yes] See Appendix B
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix B
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] Our work show failure modes of existing method.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C,D and E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix F.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix F.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4 and Appendix F.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix F.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4 and
Appendix F.
(b) Did you mention the license of the assets?
[Yes] See Appendix F.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
See Appendix F.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[Yes] See Appendix F.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes] See Appendix F.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]