L EARNING D E - BIASED R EPRESENTATIONS WITH B IASED R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many machine learning algorithms are trained and evaluated by splitting data from a single source into training and test sets.
While such focus on in-distribution learning scenarios has led interesting advances, it has not been able to tell if models are relying on dataset biases as shortcuts for successful prediction (e.g., using snow cues for recognising snowmobiles).
Such biased models fail to generalise when the bias shifts to a different class.
The cross-bias generalisation problem has been addressed by de-biasing training data through augmentation or re-sampling, which are often prohibitive due to the data collection cost (e.g., collecting images of a snowmobile on a desert) and the difﬁculty of quantifying or expressing biases in the ﬁrst place.
In this work, we propose a novel framework to train a de-biased representation by encouraging it to be different from a set of representations that are biased by design.
This tactic is feasible in many scenarios where it is much easier to deﬁne a set of biased representations than to deﬁne and quantify bias.
Our experiments and analyses show that our method discourages models from taking bias shortcuts, resulting in improved performances on de-biased test data.
I NTRODUCTION
Most machine learning algorithms are trained and evaluated by randomly splitting a single source of data into training and test sets.
Although this is a standard protocol, it is blind to a critical problem: the existence of dataset bias (Torralba & Efros, 2011).
For instance, many frog images are taken in swamp scenes, but swamp itself is not a frog.
Nonetheless, a neural network will exploit this bias (i.e., take “shortcuts”) if it yields correct predictions for the majority of training examples.
If bias is sufﬁcient to achieve high accuracy, there is little motivation for models to learn the complexity of the intended task, despite its full capacity to do so.
Consequently, a model that relies on bias will achieve high in-distribution accuracy, yet fail to generalise when the bias shifts.
We tackle this “cross-bias generalisation” problem where a model does not exploit its full capacity due to the “sufﬁciency” of bias cues for prediction of the target label in training data.
For example, language models make predictions based on the presence of certain words (e.g., “not” for “contradiction”)
(Gururangan et al., 2018) without much reasoning on the actual meaning of sentences, even if they are in principle capable of sophisticated reasoning.
Similarly, convolutional neural networks (CNNs) achieve high accuracies on image classiﬁcation by using local texture cues as shortcut, as opposed to more reliable global shape cues (Geirhos et al., 2019; Brendel & Bethge, 2019).
Existing methods attempt to remove a model’s dependency on bias by de-biasing the training data through data augmentation (Geirhos et al., 2019) or re-sampling tactics (Li & Vasconcelos, 2019).
Others have introduced a pre-deﬁned set of biases that a model is trained to be independent against (Wang et al., 2019).
These prior works assume that bias can easily be deﬁned or quantiﬁed, but real-world biases often do not (e.g., texture bias above).
To address this limitation, we propose a novel framework to train a de-biased representation by encouraging it to be “different” from a set of representations that are biased by design.
Our insight is that biased representations can easily be obtained by utilising models of smaller capacity (e.g., bag of words for word bias and CNNs of small receptive ﬁelds for texture bias).
Experiments show that our method is effective in reducing a model’s dependency on “shortcuts” in training data, as evidenced by improved accuracies in test data where the bias is either shifted or removed.
[Figure]
s i g n a l
[Figure]
P ROBLEM D EFINITION
We provide a rigorous deﬁnition of our over-arching goal: overcoming the bias in models trained on biased data.
We show that the problem we tackle is novel and realistic.
2. C ROSS - BIAS GENERALISATION
We ﬁrst deﬁne random variables, signal S and bias B as cues for the recognition of an input X as certain target variable Y .
Signals S are the cues essential for the recognition of X as Y ; examples include the shape and skin patterns of frogs for frog image classiﬁcation.
Biases B ’s, on the other hand, are cues not essential for the recognition but correlated with the target Y ; many frog images are taken in swamp scenes, so swamp scenes can be considered as B .
A key property of B is that intervening on B should not change Y ; moving a frog from swamp to a dessert scene does not change the “frogness”.
We assume that the true predictive distribution p ( Y | X ) factorises as (cid:82) p ( Y | S, B ) p ( S, B | X ) , signifying the sufﬁciency of p ( S, B | X ) for recognition.
Under this framework, three learning scenarios are identiﬁed depending on the change of relationship p
( S, B, Y ) across training and test distributions,
p ( S tr , B tr , Y tr ) and p ( S te , B te , Y te ) , respectively: in-distribution, cross-domain, and cross-bias generalisation.
See Figure 1 for a summary.
In-distribution.
p ( S tr , B tr , Y tr ) = p ( S te , B te , Y te ) .
This is the standard learning setup utilised in many benchmarks by splitting data from a single source into training and test data at random.
Cross-domain.
p ( S tr , B tr , Y tr ) (cid:54) = p ( S te , B te , Y te ) and furthermore p ( B tr ) (cid:54) =
p ( B te ) .
B in this case is often referred to as “domain”.
For example, training data consist of images with ( Y tr =frog,
B tr =wilderness) and ( Y tr =bird, B tr =wilderness), while test data contain ( Y te =frog, B te =indoors) and ( Y te =bird, B te =indoors).
This scenario is typically simulated by training and testing on different datasets (Ben-David et al., 2007).
Cross-bias.
p ( B tr ) (cid:54)⊥⊥ p ( Y tr ) 1 and the dependency changes across training and test distributions: p ( B tr , Y tr ) (cid:54) = p ( B te , Y te ) .
We further assume that p ( B tr ) =
p ( B te ) , to clearly distinguish the scenario from the cross-domain generalisation.
For example, training data only contain images of two types ( Y tr =frog, B tr =swamp) and ( Y tr =bird, B tr =sky), but test data contain unusual class-bias combinations ( Y te =frog, B te =sky) and ( Y te =bird, B te =swamp).
Our work addresses this scenario.
2. E XISTING CROSS - BIAS GENERALISATION METHODS AND THEIR ASSUMPTIONS
Under cross-bias generalisation scenarios, the dependency p ( B tr ) (cid:54)⊥⊥ p ( Y tr ) makes bias B a viable cue for recognition.
The model trained on such data becomes susceptible to interventions on B , limiting its generalisabililty when the bias is changed or removed in the test data.
There exist prior approaches to this problem, but with different types and amounts of assumptions on B .
We brieﬂy recap the approaches based on the assumptions they require.
In the next part § 2.3, we will deﬁne our novel problem setting that requires an assumption distinct from the ones in prior approaches.
When an algorithm to disentangle bias B and signal S exists.
Being able to disentangle B and
S lets one collapse the feature space corresponding to B in both training and test data.
A model trained on such normalised data then becomes free of biases.
As ideal as it is, building a model to perfectly disentangle B and S is often unrealistic.
When a generative algorithm or data collection procedure for p ( X | B ) exists.
When additional examples can be supplied through p ( X | B ) , the training dataset itself can be de-biased, i.e., B ⊥⊥ Y .
For example, one can either collect or synthesise unusual images like frogs in sky and birds in swamp to balance out the bias.
Such a data augmentation strategy is indeed a valid solution adopted by many prior studies (Panda et al., 2018; Geirhos et al., 2019;
Shetty et al., 2019).
However, collecting unusual inputs can be expensive (Peyre et al., 2017), and building a generative model with pre-deﬁned bias types (Geirhos et al., 2019) may suffer from bias mis-speciﬁcation and the lack of realism.
When a predictive algorithm or ground truth for p ( B | X ) exists.
Conversely, when one can tell the bias B for every input X , two approaches are feasible.
The ﬁrst is a data re-weighting solution: we give greater weights on frogs in sky than frogs in swamps to even out the correlation in p ( B, Y )
(Li et al., 2018; Li & Vasconcelos, 2019).
The second approach removes the dependency between the model predictions f ( X ) and the bias B .
Many existing approaches for fairness in machine learning have proposed independence-based regularisers to encourage f ( X ) ⊥⊥ B (Zemel et al., 2013) or the conditional independence f ( X ) ⊥⊥ B | Y (called the “separation” constraint,
Hardt et al.
Other approaches have proposed to remove predictability of p ( B | X ) based on f ( X ) through domain adversarial losses (Li & Vasconcelos, 2019; Wang et al., 2019) or projection (Wang et al., 2019; Quadrianto et al., 2019).
The knowledge on p ( B | X ) is provided in many realistic scenarios.
For example, when the aim is to remove gender biases B in a job application process p ( Y | X ) , applicants’ genders p ( B | X ) are supplied as ground truths.
However, there exist cases when B is difﬁcult to even be deﬁned or quantiﬁed but can only be indirectly speciﬁed.
We tackle such a scenario in the next part.
2. O UR SCENARIO : C APTURING BIAS WITH A PARTICULAR SET OF MODELS
Under the cross-bias generalisation scenario, certain types of biases are not easily addressed by the above methods.
Take texture bias as an example ( § 1, Geirhos et al.
(1) texture B and shape
S cannot easily be disentangled, (2) building a generative model p ( X | B ) or collecting unusual images is expensive, (3) building the predictive model p ( B | X ) for texture requires enumeration (classiﬁer) or embedding (regression) of all possible textures, which is not feasible.
However, slightly modifying the third assumption results in a problem setting that allows interesting application scenarios.
Instead of assuming explicit knowledge on p ( B | X ) , we approximate B by deﬁning a set of models G that are biased towards B by design.
For texture biases, for example, we deﬁne G to be the set of convolutional neural network (CNN) architectures with ≤ 5 × 5 overall receptive ﬁelds.
Then, any learned model g ∈ G can by design make predictions g ( x ) based on the patterns that can only be captured with small receptive ﬁelds (i.e., textures).
More precisely, we deﬁne G to be a bias-characterising model class for the bias-signal pair ( B, S ) if for every possible joint distribution p ( B, X ) there exists a g ∈ G such that p ( B | X )
≈ g
( X ) ( recall condition ) and every g ∈ G satisﬁes g ( X ) ⊥⊥ S | B
( precision condition ).
In practice, G may not necessarily include all biases and may also capture important signals (i.e., imperfect recall and precision).
With this in mind, our framework is formulated so that f ( X ) does not ignore signals captured by G – we do not require G to be perfect. Further detail is provided in § 2.3.
There exist many scenarios when such G can be deﬁned, as we are given several evidence for the type of bias.
For instance, action recognition models have been reported to rely heavily on static cues without learning temporal cues (Li et al., 2018; Li & Vasconcelos, 2019); though the actual bias may not precisely be any static cue , we can still regularise the 3D convolutional networks towards better generalisation across static cue biases by deﬁning G to be the set of 2D convolutional architectures.
It has been argued that visual question answering (VQA) models, too, rely overly on language biases rather than the visual cues (e.g. without looking at the image, one knows the answer to what colour is the banana is yellow) (Agrawal et al., 2018). We can deﬁne G as the set of models looking at the language modality only. Entailment models are biased towards the presence of certain words (e.g. when there are many nots, the sentence is contradictory), rather than really understanding the underlying meaning of sentences (McCoy et al., 2019; Niven & Kao, 2019).
We can design G to be the set of bag-of-words classiﬁers (He et al., 2019; Clark et al., 2019).
Generally, these scenarios exemplify situations when the added architectural capacity is not fully utilised due to the sufﬁciency of simpler cues for solving the task in the given training set.
P ROPOSED M ETHOD
We present a solution for the cross-bias generalisation when the bias-characterising model class G is known (see § 2.3); the method is referred to as REBI’S 2 .
The solution consists of training a model f for the task p
( Y | X ) with a regularisation term encouraging the independence between the prediction f ( X ) and the set of all possible biased predictions { g ( X ) | g ∈ G } .
We will introduce the precise deﬁnition of the regularisation term and discuss why and how it leads to the unbiased model.
3. REBI’S : R EMOVING BIAS WITH BIAS
If p ( B | X ) is fully known, we can directly encourage f ( X ) ⊥⊥ B .
Since we only have access to the set of biased models G ( § 2.3), we seek to promote f ( X ) ⊥⊥ g ( X ) for every g ∈ G .
Simply put, we de-bias a representation f ∈ F by designing a set of biased models G and letting f run away from G .
This leads to the independence from bias cues B while leaving signal cues S as valid recognition cues; see § 2.3.
We will specify REBI’S learning objective after introducing our independence criterion, HSIC.
Hilbert-Schmidt Independence Criterion (HSIC)
Since we need to measure degree of independence between continuous random variables f ( X ) and g ( X ) in high dimensional spaces, it is infeasible to resort to histogram-based measures; we use HSIC (Gretton et al., 2005).
For two random variables U and V and kernels k and l , HSIC is deﬁned as HSIC k,l ( U, V ) :
=
|| C k,lUV || 2 HS where C k,l is the cross-covariance operator in the Reproducing Kernel Hilbert Spaces (RKHS) of k and l (Gretton et al., 2005), an RKHS analogue of covariance matrices.
|| · || HS is the Hilbert- Schmidt norm, a Hilbert-space analogue of the Frobenius norm.
It is known that for two random variables U and V and radial basis function (RBF) kernels k and l , HSIC k,l ( U, V )
=
if and only if U ⊥⊥ V .
A ﬁnite-sample estimate HSIC k,l 0 ( U, V ) has been used in practice for statistical testing (Gretton et al., 2005; 2008), feature similarity measurement (Kornblith et al., 2019), and model regularisation (Quadrianto et al., 2019; Zhang et al., 2018).
HSIC k,l 0 ( U, V ) with m samples is deﬁned as ( m − 1) − 2 tr ( (cid:101) U (cid:101) V T ) where (cid:101)
U is a mean-subtracted matrix of pairwise kernel similarities (cid:101) U ij =
k
( u i , u j ) − m − 1 (cid:80) j (cid:48) k
( u i , u j (cid:48) ) among samples { u i } ∼ U .
(cid:101) V is deﬁned similarly.
Minimax optimisation for bias removal.
In our case, we compute
[Equation]
1) with an RBF kernel k for the degree of independence between representation f ∈ F and the biased representations G .
We write HSIC 0 ( f, G ) and HSIC 0 ( f, g ) as shorthands.
Since the problem min f HSIC 0 ( f, G ) allows trivial solutions f
= const, we use the canonical kernel alignment (CKA) (Shawe-Taylor & Cristianini, 2004; Kornblith et al., 2019) criterion deﬁned by[-CKA 
-]
CKA 0 ( f, g ) =
HSIC 0 ( f, g ) /
( HSIC 0 ( f, f ) 12 HSIC 0 ( g, g ) 12 ) .
The learning objective for f is then deﬁned as
[Equation]
where L ( f, X, Y ) is the loss for the main task p
( Y | X ) and λ > 0 .
We consider replacing the inner optimisation with L 2 minimisation min g ∈ G || f − g
||
2 , while retaining the CKA regularisation for the outer optimisation for f .
Intuitively, L 2 minimisation poses a stronger similarity condition between f and g (in fact identity) than does CKA , leading to better de-biasing performances ( § 4.2.3).
Independence versus separation.
The CKA regularisation in equation 2 encourages f ( X ) ⊥ ⊥ g ( X ) .
This may lead to less stable optimisation as removing bias often increases the main task loss L .
Furthermore, g ( X ) may also capture signals along with bias (imperfect precision), and suppressing important signals is not desirable.
To avoid such cases, we formulate our objective as conditional independence f ( X ) ⊥⊥ g ( X ) | Y by computing the separation CKA , SCKA 0 ( f, g ) =
|Y| − 1 (cid:80)
y ∈Y CKA
( f ( X | Y =
y ) , g ( X | Y = y )) , where the term “separation” (or “equalised odds”) comes from the fairness literature (Hardt et al., 2016). Unlike independence that requires f ( X ) to ignore g ( X ) altogether, conditional independence allows f ( X ) to utilise cues captured by g ( X ) (i.e., whether it is bias or signal) if it highly correlates with Y. In other words, REBI’S encourages f ( X ) to learn features beyond those already captured by g ( X ) , so that it can generalise well to test sets with different biases (i.e., cross-bias generalisation).
The ﬁnal learning objective for REBI’S is then
[Equation]
3. W HY AND HOW DOES IT WORK ?
Independence describes relationships between random variables, but we use it for function pairs.
Which functional relationship does statistical independence translate to?
In this part, we argue with proofs and observations that the answer to the above question is the dissimilarity of invariance types learned by a pair of models .
Linear case: Equivalence between independence and orthogonality.
We study the set of function pairs ( f, g ) satisfying f ( X ) ⊥⊥ g
( X ) for suitable random variable X ∼ p ( X ) .
Assuming linearity of involved functions and the normality of X , we obtain the equivalence between statistical independence and functional orthogonality.
Lemma 1.
Assume that f and g are afﬁne mappings f
( x ) =
Ax + a and g ( x ) =
Bx + b where
A ∈ R m × n and B ∈ R l × n .
Assume further that X is a normal distribution with mean µ and covariance matrix Σ .
Then, f ( X ) ⊥⊥ g ( X ) if and only if ker( A ) ⊥ ⊥ Σ ker( B ) ⊥ .
For a positive semi-deﬁnite matrix Σ , we deﬁne (cid:104) r, s (
cid:105) Σ
= (cid:104) r, Σ s (cid:105) , and the set orthogonality ⊥ Σ likewise.
Proof in § A.
In particular, when f and g have 1-dimensional outputs, the independence condition is translated to the orthogonality of their weight vectors and decision boundaries.
From a machine learning point of view, f and g are models with orthogonal invariance types.
Non-linear case: HSIC as a metric learning objective.
We lack theories to fully characterise general, possibly non-linear, function pairs ( f, g ) achieving f ( X ) ⊥⊥ g
( X ) ; it is an interesting open question.
For now, we make a set of observations in this general case, using the ﬁnite-sample independence criterion HSIC 0 ( f, g ) :
= ( m − 1) − 2 tr ( (cid:101) f
(cid:101) g T )
= 0 , where (cid:101) f is the mean-subtracted kernel matrix (cid:101) f ij =
k ( f ( x i ) , f ( x j ))
− m − 1 (cid:80)
k k ( f ( x i ) , f ( x k )) and likewise for (cid:101) g (see § 3.1).
Note that tr ( (cid:101) f (cid:101) g T ) is an inner product between ﬂattened matrices (cid:101) f and (cid:101) g .
We consider the inner product minimising solution for f on an input pair x 0 (cid:54) =
x 1 given a ﬁxed g .
The problem can be written as min f ( x 0 ) ,f ( x 1 ) tr ( (cid:101) f (cid:101) g T ) , which is equivalent to min f ( x 0 ) ,f ( x 1 ) (cid:101) f 01 · (cid:101) g 10 .
Suppose (cid:101) g 10 > 0 .
It indicates a relative invariance of g on ( x 1 , x 0 ) , since k ( g ( x 1 ) , g ( x 0 ))
> m − 1 (cid:80) i k
( g ( x 1 ) , g ( x i )) .
Then, the above problem boils down to min f ( x 0 ) ,f ( x 1 ) (cid:101) f 01 , signifying the relative variance of f on ( x 0 , x 1 ) .
Following a similar argument, we obtain the converse statement: if g is relatively variant on a pair of inputs, invariance of f on the pair minimises the objective.
We conclude that min f HSIC 0 ( f, g ) against a ﬁxed g is a metric-learning objective for the embedding f , where ground truth pairwise matches and mismatches are relative mismatches and matches for g , respectively.
As a result, f and g learn different sorts of invariances.
Effect of HSIC regularisation on toy data.
We have established that HSIC regularisation encourages the difference in model invariances.
To see how it helps to de-bias a model, we have prepared synthetic two-dimensional training data following the cross-domain generalisation case in Figure 1:
X =
( B, S )
∈ R 2 and Y ∈ { red, yellow, green } .
Since the training data is perfectly biased, a multi-layer perceptron (MLP) trained on the data only shows 55% accuracy on de-biased test data (see decision boundary ﬁgure in Appendix § B).
To overcome the bias, we have trained another MLP with equation 3 where the bias-characterising class G is deﬁned as the set of MLPs that take only the bias dimension as input.
This model exhibits de-biased decision boundaries (Appendix § B) with improved accuracy of 89% on the de-biased test data.
E XPERIMENTS
In the previous section, REBI’S has been introduced and theoretically justiﬁed.
In this section, we present experimental results of REBI’S .
We ﬁrst introduce the setup, including the biases tackled in the experiments, difﬁculties inherent to the cross-bias evaluation, and the implementation details ( § 4.1).
Results on Biased MNIST ( § 4.2) and ImageNet ( § 4.3) are shown afterwards.
4. E XPERIMENTAL SETUP
Which biases do we tackle?
There is a broad spectrum of bias types to be addressed under the cross-bias generalisation setting.
Our work is targeting the biases that arise due to the existence of shortcut cues that are sufﬁcient for recognition in training data.
In the experiments, we tackle a representative bias of such type: “local pattern” biases for image classiﬁcation.
Even if a CNN image classiﬁer has wide receptive ﬁelds, empirical evidence indicates that they heavily rely on local patterns (i.e., color and texture) as opposed to global shape cues (Geirhos et al., 2019).
While it is difﬁcult to precisely deﬁne and quantify all local pattern biases, it is easy to capture it through a class of CNN architectures: those with smaller receptive ﬁelds.
This is precisely the setting where we beneﬁt from REBI’S .
Evaluating cross-bias generalisation is difﬁcult.
To measure the performance of a model across real-world biases, one requires an unbiased dataset or one where the types and degrees of biases can be controlled.
Unfortunately, data in real world arise with biases.
To de-bias a frog and bird image dataset with swamp and sky backgrounds (see § 2.1), either rare data samples must be collected (search for photos of a frog on sky) or one must intervene with the data generation process (throw a frog into the sky and take a photo).
Either way, it is an expensive procedure (Peyre et al., 2017).
Preparing an unbiased data is feasible in some cases when the bias type is simple (e.g., collecting natural language corpus of unbiased gender pronouns, Webster et al.
However, we are addressing biases that are expressed in terms of a class of representations but perhaps are difﬁcult to precisely express in language, such as texture bias of image classiﬁers ( § 2.3).
We thus evaluate our method along two axes: (1) Biased MNIST and (2) ImageNet.
Biased MNIST contains synthetic biases (colour and texture) which we freely control in training and test data for indepth analysis of REBI’S .
In particular, we can measure its performance on perfectly unbiased test data.
On ImageNet, we evaluate our method against realistic biases.
Due to the difﬁculty of deﬁning and obtaining bias labels on real images, we use proxy ground truths for the local pattern bias to measure the cross-bias generalisability.
MNIST and ImageNet experiments complement each other in terms of experimental control and realism.
Implementation of REBI’S .
We describe the speciﬁc design choices in REBI’S implementation (equation 3) in our experiments.
We will open source the code and data.
To train a model that overcomes local pattern biases, we ﬁrst deﬁne biased model architecture families G such that they precisely and sufﬁciently encode biased representations: CNNs with relatively small receptive ﬁelds (RFs).
The biased models in G will by design learn to predict the target class of an image through only local cues.
On the other hand, we deﬁne a larger search space F with larger RFs for our unbiased representations.
In our work, all networks f and g are fully convolutional networks.
f ( x ) and g ( x ) denote the ﬁnal convolutional layer outputs (feature maps), on which we compute the independence measures like
HSIC, CKA, and SCKA ( § 3.1).
We perform global average pooling and a learnable linear classiﬁer on f ( x ) , trained along with the outer optimisation, to compute the cross-entropy loss in equation 3.
For Biased MNIST, F is the LeNet F (LeCun et al., 1998) architecture with RF 28.
It has two 5convs 3 , after each of which max-pooling layers of 2 × 2 kernels are applied, followed by three linear layers.
G has the same number of layers but either with all convolutional layers of 1 × 1 kernels and without max-pooling operations, called BlindNet1 , or with one 3-conv and one 1-conv, called
BlindNet3 , each with a RFs of 1 and 3, respectively.
On ImageNet, we use ResNet (He et al., 2016) architecture for F (either ResNet18 with RF of 435 or ResNet50 with RF of 427).
G is deﬁned as BagNet (Brendel & Bethge, 2019) architectures with the same depth as ResNet ’s
(either BagNet18 with RF of 43 or BagNet50 with RF of 91).
More implementation details are provided in § C.
4. B IASED MNIST
We ﬁrst verify our model on a dataset where we have full control over the type and amount of bias during training and evaluation.
We describe the dataset and present the experimental results.
4.2. D ATASET AND EVALUATION
We construct a new dataset called Biased
MNIST designed to measure the extent to which models generalise to bias shift.
We modify MNIST (LeCun et al., 1998) by introducing two types of bias B – colour and texture – that highly correlate with the label Y during training.
With B alone, a
CNN can achieve high accuracy without having to learn inherent signals for digit recognition S , such as shape, providing little motivation for the model to learn beyond these superﬁcial cues.
We inject colour and texture biases by adding colour or texture patterns on training image backgrounds (see Figure 2).
We pre-select 10 distinct colour or 3 × 3 texture patterns b ( y ) for each digit y ∈ { 0 , · · · , 9 } .
Then, for each image of digit y , we assign the pre-deﬁned pattern b ( y ) with proba- bility ρ ∈ [0 , 1] and any other pattern (pre-deﬁned for other digits) with probability (1 − ρ ) .
ρ then controls the bias-target correlation in the training data: ρ = 1 .
leads to complete bias and ρ = 0 .leads to an unbiased dataset.
We consider two datasets: Single-bias MNIST with only colour bias and Multi-bias MNIST with both colour and texture biases ( ρ = 0 .
99 in all the experiments).
We evaluate the models generalisability to bias shift by evaluating under the following criterion:
Biased.
p ( S te , B te , Y te ) =
p ( S tr , B tr , Y tr ) (an in-distribution case in § 2.1).
Whatever bias the training set contains, it is replicated in the test set.
This measures the ability of de-biased models to maintain high in-distribution performance while generalising to unbiased settings.
Unbiased.
B te ⊥⊥ Y te .
We assign biases on test images independently of the labels.
Bias is no longer predictive of Y and a model needs to utilise actual signals S to yield correct predictions.
We have additional ﬁne-grained measures on Multi-bias MNIST: removed colour bias ( colour ; texture bias remains) and removed texture bias ( texture ; colour bias remains) cases.
Colour and texture biases are marginalised out in the test set, respectively, to factorise generalisability across different types of bias.
4.2. R ESULTS
Results on Single- and Multi-bias MNIST are shown in Table 1.
REBI’S lets a model overcome bias.
We observe that vanilla LeNet F achieves 100% accuracy under the “biased” metric (the same bias between training and test data) in Single- and Multi-bias
MNIST.
This is how most machine learning tasks are evaluated, yet this does not show the extent [Table] to which the model depends on bias for prediction.
When the bias cues are randomly assigned to the label at evaluation, vanilla LeNet accuracy collapses to 27.5% and 29.6% under the “unbiased” metric on Single- and Multi-bias MNIST, respectively.
The intentionally biased BlindNet models
G result in an even lower accuracy of 10.8% on Single-bias MNIST, close to the random chance 10%.
This reveals that the seemingly high-performing model has in fact overﬁtted to bias and has not learned beyond this fallible strategy.
REBI’S , on the other hand, achieves robust generalisation across all settings by learning to be different from BlindNet representations G .
REBI’S achieves a +47.4 pp (Single) and +59.0 pp (Multi) higher performances than the vanilla model under the cross-bias generalisation setup (the unbiased metric), with a slight degradation in original accuracies (-3.
pp
and -5.3 pp, respectively).
Comparison against HEX .
Previously, HEX (Wang et al., 2019) has attempted to reduce the dependency of a model on “superﬁcial statistics”, or high-frequency textural information.
HEX measures texture via neural grey-level co-occurrence matrices (NGLCM) and projects out the NGLCM feature from the output of the model of interest.
We observe that F HEX , where HEX is applied on F , is effective in removing texture biases (85.8% to 91.1% cross-texture accuracy), but still vulnerable to colour biases (accuracy drops from 45.1% to 31.1%).
Hand-crafting texture features as done by
HEX has resulted in its limited applicability beyond the hand-crafted bias type.
By designing the model family architecture, instead of a speciﬁc feature extractor, REBI’S achieved a representation free of broader types of biases.
4.2. F ACTOR ANALYSIS
Some design choices exist, leading to our ﬁnal model ( § 3.1).
We examine how the factors contribute to the ﬁnal performance.
See Table 2 for ablative studies on the Single-bias MNIST.
[Table]
Impact of the independence criterion.
Three independence measures, HSIC, CKA, and SCKA, have been considered.
SCKA, the separation CKA, used in REBI’S , shows a superior de-biasing performance (74.9%) against baseline choices.
It improves upon HSIC by avoiding the trivial solution (a constant function) [Table] and upon CKA via more stable optimisation due to the milder conditional independence f ( X ) ⊥⊥ B | Y that does not contradict the classiﬁcation objective.
Impact of the L 2 objective in the inner optimisation.
We then study the effect of our choice to replace the inner SCKA optimisation with the L 2 objective.
L 2 is considered, as it poses a stronger convergence condition than SCKA does.
We conﬁrm indeed that the L 2 objective results in a better de-biasing performance.
Impact of updating g ∈ G .
The advantage of specifying a class of models G instead of a single, ﬁxed model g 0 is that HSIC ( f, G ) can be computed more precisely ( § 3.1).
We quantify this beneﬁt.
Fixing the biased representation to g 0 results in sub-optimal de-biasing performance, 44.7%.
By including multiple ﬁxed biased models { g 0 , · · · , g 7 } ⊂ G , de-biasing improves to 58.1%, but is not as good as the updated g case, 74.9%.
It is thus important to precisely compute the representationto-set independence.
More detailed analysis around the receptive ﬁeld sizes of models in G is in
Appendix § E.
[Table]
4. I MAGE N ET
In ImageNet experiments, we further validate the applicability of REBI’S on the local pattern bias in realistic images (i.e,, objects in natural scenes).
The local pattern bias often lets a model achieve good in-distribution performances by exploiting the local cue shortcuts (e.g, determining a turtle class by not seeing its shape but the background texture).
4.3. D ATASET AND EVALUATION
We construct 9-Class ImageNet , a subset of ImageNet (Russakovsky et al., 2015) containingsuper-classes as done in Ilyas et al.
(2019), since full-scale analysis on ImageNet is not scalable.
We additionally balance the ratios of sub-class images for each super-class to solely focus on the effect of the local pattern bias.
Since it is difﬁcult to evaluate cross-bias generalisability on realistic unbiased data ( § 4.1), we settle for alternative evaluations:
Biased.
p ( S te , B te , Y te ) =
p ( S tr , B tr , Y tr ) Accuracy is measured on the in-distribution validation set Though
widely
-used, this metric is blind to a model’s generalisability to unseen bias-target combinations
Unbiased.
B te ⊥⊥ Y
te As a proxy to the perfectly de-biased test data, which is difﬁcult to collect ( § 4.1), we use texture clusters IDs c ∈ { 1 , · · · , K } as the ground truth labels for local pattern bias.
For full details of texture clustering algorithm, see Appendix § F.
For an unbiased accuracy measurement, we compute accuracies for every set of images corresponding to a target-texture combination ( c, y ) .
The combination-wise accuracy A c,y is computed by Corr ( c, y ) /
Pop ( c, y ) , where Corr ( c, y ) is the number of correctly predicted samples in ( c, y ) and Pop ( c, y ) is the total number of samples in ( c, y ) , called the population at ( c, y ) .
The unbiased accuracy is then the mean accuracy over all A c,y where the population is non-zero Pop ( c, y ) (cid:54) = 0 .
This measure gives more weights on samples of unusual texture-class combinations (smaller Pop ( c, y ) ) that are less represented in the usual biased accuracies.
Under this unbiased metric, a biased model basing its recognition on textures is likely to show sub-optimal results on unusual combinations, leading to a drop in the unbiased accuracy.
ImageNet-A. ImageNet-A (Hendrycks et al., 2019) contains failure cases of ImageNet
pre
-trained
ResNet50 among web images.
The images consist of many failure modes of networks when “frequently appearing background elements” become erroneous cues for recognition (e.g. a bee image feeding on hummingbird feeder is recognised as a hummingbird).
Improved performance on
ImageNet-A is an indirect signal that the model learns beyond the bias shortcuts.
4.3. R ESULTS
We measure performances of ResNet18 and ResNet50 , each trained to be different from
BagNet18 and BagNet50 respectively using REBI’S , under the metrics in the previous part.
Results are shown in Table 3.
Vanilla models are biased.
Both ResNet18 and ResNet50 show good performances on the biased accuracy (93.3% and 91.7%, respectively), but dropped performances on the texture-unbiased accuracies (85.8% and 78.3%, respectively).
BagNet18 and BagNet50 perform worse than the vanilla ResNets as they are heavily biased towards texture by design (i.e., small receptive ﬁeld size).
The drop signiﬁes the biases of vanilla models towards texture cues; by basing their predictions on texture cues they obtain generally better accuracies on texture-class pairs ( c, y ) that are represented more.
The drop also shows the limitation of current evaluation schemes where cross-bias generalisation is not measured.
REBI’S leads to less biased models.
When REBI’S is applied on ResNet18 and ResNet to encourage them to unlearn cues learnable by BagNet18 and BagNet50 , respectively, we observe general boost in unbiased accuracies.
ResNet18 improves from 85.8% to 88.4%; ResNet50 from 78.3% to 89.2%.
Our method thus robustly generalises to less represented texturetarget combinations at test time.
We observe that our method also shows improvement on the challenging ImageNet-A subset (e.g. from 29.5% to 31.3% for ResNet50 ), which further shows our superiority on generalisability to unusual texture-class combinations.
Similarly, data augmentation via Stylised ImageNet (Geirhos et al., 2019) attempts to reduce a model’s dependency on texture by augmenting data with texturised images.
While it shows improvements in reducing texture bias (87.6% for unbiased accuracy), it does not increase the generalisability to the challenging natural adversarial examples (29.7% for ImageNet-A).
More detailed analysis on per-texture and per-class accuracies are included in Appendix § G; learning curves for the baseline and REBI’S are in Appendix § H.
Qualitative analysis.
We qualitatively present the cases where our method successfully de-biases a texture-target dependency.
Figure 3 shows examples of common and uncommon texture-target combinations for “grass” and “close-up” texture clusters.
The shown uncommon instances are the ones ResNet18 has incorrectly predicted the class.
For example, crab in the grass has been predicted as turtle, presumably because turtles co-occur a lot with grass backgrounds in training data.
On the other hand, REBI’S (ours) robustly generalises to unusual texture-class combinations.
C ONCLUSION
We have identiﬁed a practical problem faced by many machine learning algorithms that the learned models exploit bias shortcuts to recognise the target (cross-bias generalisation problem in § 2).
In particular, models tend to under-utilise its capacity to extract non-bias signals (e g. global shapes for object recognition) when bias shortcuts provide sufﬁcient cues for recognition in the training data (e g. local patterns and background cues for object recognition)
(Geirhos et al., 2019).
We have addressed this problem with the REBI’S framework which does not rely on expensive, if not infeasible, training data debiasing schemes. Given an identiﬁed set of models G that encodes the bias to be removed, REBI’S encourages a model f to be statistically independent of G ( § 3). We have provided theoretical justiﬁcations for the use of statistical independence in § 3. and have validated the superiority of REBI’S in removing biases from models through experiments on modiﬁed MNIST and ImageNet ( § 4) A S TATISTICAL I NDEPENDENCE IS E QUIVALENT TO F UNCTIONAL O RTHOGONALITY FOR L INEAR M APS .
We provide a proof for the following lemma in § 3.2.
Lemma 1.
Assume that f and g are afﬁne mappings f
( x ) =
Ax + a and g ( x ) =
Bx + b where
A ∈ R m × n and B ∈ R l × n .
Assume further that X is a normal distribution with mean µ and covariance matrix Σ .
Then, f ( X ) ⊥⊥ g ( X ) if and only if ker( A ) ⊥ ⊥ Σ ker( B ) ⊥ .
For a positive semi-deﬁnite matrix Σ , we deﬁne (cid:104) r, s (
cid:105) Σ
= (cid:104) r, Σ s (cid:105) , and the set orthogonality ⊥ Σ likewise.
Proof.
Due to linearity and normality, the independence f ( X ) ⊥⊥ g ( X ) is equivalent to the covariance condition Cov ( f ( X ) , g ( X ))
= 0 .
Covariance is computed as:
[Equation]
Note that
[Equation]
B D ECISION B OUNDARY V ISUALISATION FOR T OY E XPERIMENT
We show the decision boundaries of the toy experiment in § 3.2.
See Figure 4.
GIF animations of decision boundary changes over training can be found at anonymous link.
[Figure]
C I MPLEMENTATION D ETAILS
We solve the minimax problem in equation 3 through alternating stochastic gradient descents (ADAM Kingma & Ba (2014)), where we alternate between 5 epochs for the outer problem and 5 epochs for the inner one.
The regularisation parameter λ is set to 7 , 000 for CKA and 100 for
L 2 minimisation on MNIST and 0 .
for ImageNet.
Note that we use a large λ for MNIST as the degree of synthetic bias is excessively large (bias-target correlation set to ρ = 0 . 99 ) compared to the degree of bias in realistic settings.
We have used batch sizes 256 ( 32 for ResNet50 ) and learning rates 0 .
001 with linear decay.
We train all models for 80 epochs for Biased MNIST. For 9-class ImageNet, we train ResNet18 for 200 epochs and ResNet50 for 100 epochs. BagNet18 and BagNet50 have 18 and 50 convolutional layers which are the same structure (Basic or Bottleneck blocks) as ResNet18 and ResNet50 , respectively.
The internal kernel sizes of BagNets are set to 1 × 1 following the original paper’s philosophy (Brendel & Bethge (2019)).
D P ERFORMANCE BY BIAS AND LABEL ON B IASED MNIST.
In the Biased MNIST experiments ( § 4.2), we have shown either biased or unbiased set statistics.
In this section, we provide more detailed results where model accuracies are computed per bias class
B = b and per target label Y =
y .
We visualise the case-wise accuracies of the baseline LeNet and the REBI’S trained LeNet ⊥⊥ BlindNet in Figure 5.
The diagonals in each matrix indicate the pre-deﬁned bias-target pair ( § 4.2.1).
Thus, the biased accuracies can be computed by taking the mean of diagonal entries in each matrix and the unbiased accuracies through the mean of all entries in each matrix.
The vanilla LeNet ’s tendency to have higher accuracies on diagonal entries and near-zero performances on many other off-diagonal entries indicate the fact that LeNet is relying a lot on the colour (bias) cues.
REBI’S successfully resolves this tendency in the vanilla model, exhibiting more uniform performances across all bias-target pairs ( b, y ) .
Note that accuracies below the main diagonal are relatively high as they are the classes that pre-deﬁned patterns are assigned to at probability (1 − ρ ) .
Figure 6 demonstrates that our method is successful across different degrees of bias in a given training data.
[Figure]
E I MPACT OF R ECEPTIVE F IELDS OF G FOR B IASED MNIST
It is conceptually important to design a the set of biased models G that encode the bias B as precisely as possible.
See precision and recall conditional for G in § 2.3).
To see if this is empirically true, we have measured the performance of REBI’S with F = LeNet and G = BlindNet , where the
BlindNet receptive ﬁelds are controlled by replacing convolutional layers with 1 × 1 convolutions, resulting in receptive ﬁelds { 1 , 5 , 7 , 28 } .
The 28 × 28 receptive ﬁeld indicate the case when LeNet is used as G .
We measure the biased and unbiased set performances on Single-bias MNIST ( § 4.2.1).
[Figure] [Figure]
Results are shown in Figure 7.
We observe relatively stable biased set performances and decreasing unbiased set performances.
The decrease in de-biasing ability is attributed to the violation of the precision condition for G ( § 2.3): 1 × 1 receptive ﬁelds are sufﬁcient to capture any colour bias variations, and de-biasing against models of larger receptive ﬁelds will further make f not see the meaningful signal S (cues beyond 1 × 1 -expressible colours).
In the extreme case, when F = LeNet is trained to be independent against itself
G = LeNet , the de-biasing performance drops signiﬁcantly.
F T EXTURE C LUSTERING
In our ImageNet experiments ( § 4.3), we obtained proxy ground truths for the local pattern bias using texture clustering.
We extract texture information from images by clustering the gram matrices of low-layer feature maps as done in standard texturisation methods (Gatys et al., 2015; Johnson et al., 2016); we use feature maps from layer relu1 2 of a pre-trained VGG16 (Simonyan & Zisserman, 2014).
As we intend to evaluate whether a given model is biased towards local pattern cues, we only utilise features from the lower layer encoding lower layer features like edges and colours rather than high-level semantics.
Figure 8 shows that each cluster effectively captures similar local patterns across different classes.
For each cluster, we visualise its top-3 correlated classes.
We can see that certain classes share a common texture: cat, monkey, and dog share a “face”-like texture.
If a certain class is biased towards a particular texture during training, a model can take the shortcut to utilise texture cues for recognising the target class, leading to a sub-optimal cross-bias generalisation to unusual class-texture combinations (e.g. crab on grass).
[Figure]
D og
F a ce
M a m m a l Sp o tt e d
Sh e ll
W a t e r
G r a ss
V i v i d
C l
[Figure]
s
[Figure]
H
[Figure]
n
[Figure]
Crab
Bird
Turtle
Fish
Insect
I m ag e N e t
C l a ss   -   - - -  -    - -  - -  - - - - -       - - - -       - -       - - -     - - - -     - - - - -
[Figure]
N o r m a li s e d
C l u s t e r S
i ze
[Figure]
G B IAS IN DATA AND MODELS
We study the bias in the 9-class ImageNet data and the models trained on them ( § 4.3).
Figureshows the statistics of texture biases in data and the model biases that result from them.
To measure the dataset bias, we empirically observe correlations between texture and target classes by counting the number of samples for each texture-class pair ( c, y ) (denoted “population” Pop ( c, y ) in the main paper).
We observe indeed that there does exist a strong correlation between texture clusters and class labels.
We say that a class has a dominant texture cluster if the largest cluster for the class contains more than half of the class samples.
6 out of 9 classes considered has the dominant texture cluster: (“Dog”, “Dog”), (“Cat”, “Face”), (“Monkey”, “Mammal”), (“Frog”, “Spotted”), (“Crab”, “Shell”) and (“Insect”, “Close-up”).
Figure 10 further shows the accuracies of the baseline ResNet18 and REBI’S to indicate the presence of bias in the models, and how REBI’S overcomes the bias despite the bias in data itself.
We measure the average of accuracies in classes with dominant texture clusters (biased classes) and the average in less biased classes.
We observe that ResNet18 shows higher accuracy on biased classes ( 89 . 4% ) than on less biased classes ( 83 . 9 ), signifying its bias towards texture.
On the other hand, REBI’S achieves similar accuracies 88 .
7% (biased classes) and 88 .
2% (unbiased classes).
We stress that REBI’S overcomes the bias even if the training data itself is biased.
H L EARNING C URVES ON I MAGE N ET
We visualise the learning curves for the baseline vanilla ResNet50 and REBI’S trained
ResNet50 against BagNet50 in Figure 11.
We observe that REBI’S gradually de-biases a representation beyond the baseline model.