W HICH S HORTCUT C UES W ILL DNN S C HOOSE ? A S TUDY FROM THE P ARAMETER -S PACE P ERSPECTIVE
A BSTRACT
Deep neural networks (DNNs) often rely on easy–to–learn discriminatory features, or cues , that are not necessarily essential to the problem at hand.
For example, ducks in an image may be recognized based on their typical background scenery, such as lakes or streams.
This phenomenon, also known as shortcut learning , is emerging as a key limitation of the current generation of machine learning models.
In this work, we introduce a set of experiments to deepen our understanding of shortcut learning and its implications.
We design a training setup with several shortcut cues, named WCST-ML , where each cue is equally conducive to the visual recognition problem at hand.
Even under equal opportunities, we observe that (1) certain cues are preferred to others, (2) solutions biased to the easy–to–learn cues tend to converge to relatively ﬂat minima on the loss surface and (3) the solutions focusing on those preferred cues are far more abundant in the parameter space.
We explain the abundance of certain cues via their Kolmogorov (descriptional) complexity: solutions corresponding to Kolmogorov-simple cues are abundant in the parameter space and are thus preferred by DNNs.
Our studies are based on the synthetic dataset DSprites and the face dataset UTKFace.
In our WCST-ML , we observe that the bias of models leans toward simple cues, such as color and ethnicity.
Our ﬁndings emphasize the importance of active human intervention to remove the model biases that may cause negative societal impacts
I NTRODUCTION
Emerging studies on the inner mechanisms of deep neural networks (DNNs) have revealed that many models have shortcut biases (Cadene et al., 2019;
Weinzaepfel & Rogez, 2021; Bahng et al., 2020; Geirhos et al., 2020).
DNNs often pick up simple, non-essential cues, which are nonetheless effective within a particular dataset.
For example, a DNN trained for the task of animal recognition may recognize ducks while attending on water backgrounds, given the strong correlations between such background cues and the target label (Choe et al., 2020).
These shortcut biases often result in a striking qualitative difference between human and machine recognition systems; for example, convolutional neural networks (CNNs) trained on ImageNet extensively rely on texture features, while humans would preferentially look at the global shape of objects (Geirhos et al., 2019).
In other cases, the shortcut bias arises in models that suppress certain streams of inputs: visual question answering (VQA) models often neglect the entire image cues, for one does not require images to answer questions like “what color is the banana in the image?”
(Cadene et al., 2019)
Since DNNs have successfully outperformed humans on many tasks (Silver et al., 2016; Rajpurkar et al., 2017), such a phenomenon may look benign at face value.
However, the shortcut biases become problematic when it comes to the generalization to more challenging test-time conditions, where the shortcuts are no longer valid (Cadene et al., 2019; Weinzaepfel & Rogez, 2021; Bahng et al., 2020; Geirhos et al., 2020).
These biases also cause ethical concerns when the shortcut features adopted by a model are sensitive like gender or skin color (Wang et al., 2019; Xu et al., 2020).
Instead of proposing a method or solution, this work focuses on deepening our understanding of the shortcut bias phenomena.
In particular, we design a dataset where multiple cues are equally valid
Predictions on oﬀ-diagonal cells
Scale
S h a p e
Color
Scale
S h a p e
Color
Diagonal Training Data
Unbiased Testing Data
X
Y
f   scale shape color X lets us infer the cue adopted by f .
Train f on the diagonal dataset.
pred adopted cue ( ( ( ( ( ( ( ( = ( for solving a particular task and observe which cues tend to be preferentially adopted by DNNs. The experimental setup is inspired by the Wisconsin Card Sorting Test (WCST, Banno et al. in Cognitive Neuroscience 1 .
See Figure 1 for an illustration of the setup.
It consists of a training set with multiple highly correlated cues ( e.g. color, shape, and scale) that offer equally plausible pathways to the successful target prediction ( Y ∈ { 1 , 2 , 3 } ).
We call this a diagonal training set, highlighting the spatial arrangement of such samples in the product space of all combinations.
A model f trained on such a dataset will adopt or neglect certain cues.
We analyse the cues adopted by a model by observing its predictions on off-diagonal samples.
With regards to Figure 1, for example, consider the f ’s prediction for a small, blue triangle as an off-diagonal sample.
The prediction values f ( ) ∈ { 1 , 2 , 3 } tell us which cue the model is biased towards to, e.g. if f ( ) = 1 , then f is biased towards scale; if f ( ) = 2 , towards shape; and if f ( ) = 3 towards color.
All three scenarios are plausible and only testing on off-diagonal samples will reveal model bias.
We make important observations on the nature of shortcut bias under WCST-ML .
We discover that, despite the equal amounts of correlations with the target label, there tends to be a preferential ordering of cues.
The preference is largely shared across different DNN architectures, such as feedforward networks, ResNets
(He et al., 2015), Vision Transformers (Dosovitskiy et al., 2021), and multiple initial parameters.
From the parameter-space perspective, we further observe that the set of solutions Θ p biased to the preferred cues takes a far greater volume than those corresponding to the averted cues Θ a .
The loss landscape also tends to be ﬂatter around Θ p than around Θ a .
Why are certain cues preferred to others by general DNNs?
We provide an explanation based on the
Kolmogorov complexity of cues, which measures the minimal description length for representing cues (Kolmogorov, 1963).
Prior studies have shown that in the parameter space of generic DNNs, there are exponentially more Kolmogorov-simple functions than Kolmogorov-complex ones (VallePerez et al., 2019; De Palma et al., 2019).
Based on these theoretical results, we argue that DNNs are naturally drawn to Kolmogorov-simple cues.
We empirically verify that the preferences for cues correlate well with their Kolmogorov complexity estimates.
What are the consequences of the inborn preference for simple cues?
Firstly, this may hinder the generalization of DNNs to challenging test scenarios where the simple shortcut cues are no longer valid (Geirhos et al., 2020).
Secondly, we expose the possibility that certain protected attributes correspond to the simple shortcut cue for the task at hand, endangering the fairness of DNNs (Barocas et al., 2017).
In such a case, human intervention on the learning procedure may be necessary to enforce fairness, for the dataset and DNNs can be naturally drawn to exploit protected attributes.
The primary goal of this manuscript is to shed light on the nature of shortcut biases and the underlying mechanisms behind the scenes.
Our contributions are summarized as follows: an experimental setup for studying the shortcut bias in-depth ( WCST-ML ) (§2); novel observations on the nature of shortcut biases, such as the existence of preferential ordering of cues and its connections to the geometry of the loss landscape in the parameter space (§3); an explanation based on the descriptional complexity of cues (§4); and a discussion on the implications on generalization and fairness (§5), such as the preferential use of ethnical features for face recognition on the UTKFace dataset.
S ETUP
We introduce the setup that will provide the basis for the analysis in this paper.
We describe the procedure for building a dataset with multiple equally valid cues for recognition (§2.1).
The procedure is applied to DSprites and UTKFace datasets in §2.2.
In §2.3, we introduce terminologies for the analysis of the parameter space and make theoretical connections with our data framework.
2. DATA FRAMEWORK : WCST-ML
Many factors affect the preference of models to certain cues.
The existence of dominant classes is an example; it encourages models to favor cues conducive to a good performance on the dominant classes (Barocas et al., 2017; Hashimoto et al., 2018).
In other cases, some cues have higher degrees of correlation with the target label (Geirhos et al., 2020).
In this work, we test whether bias is still present under fair conditions, i.e.: when a training dataset contains a set of valid cues, each of which equally correlates with the targets, will DNNs still have a preference for certain cues?
If so, why?
To study this, we introduce a data construction framework called Wisconsin Card Sorting Test for
Machine Learners ( WCST-ML ), named after a clinical test in cognitive neuroscience (Banno et al., 2012).
See Figure 1 for an overview.
As a running example, we assume a dataset where each image can be described by varying two latent variables, object shape and object color .
Let X and
Y denote image and label, respectively.
We write X ij for the image with color i and shape j , where i, j ∈ { 1 , · · · , L } .
When we want to consider K > 2 varying factors, we may write X i 1 , ··· ,i K for the image random variable with k th factor chosen to be i k ∈ { 1 , · · · , L } .
Importantly, we ﬁx the number of categories for each factor to L to enforce similar conditions for all cues.
Similar learning setups have appeared in prior papers: “Cross-bias generalisation” (Bahng et al., 2020), “What if multiple features are predictive?”
(Hermann & Lampinen, 2020), and “Zero generalization opportunities” (Eulig et al., 2021).
While we fully acknowledge the conceptual similarities, we stress that our work presents the ﬁrst dedicated study into the cue selection problem and the underlying mechanisms.
The same set of images { X ij | 1 ≤ i, j ≤ L } admits two possible tasks: color and shape classiﬁcation.
The task is determined by the labels Y .
Denoting Y ij as the label for image X ij , setting
Y ij =
i leads to the color classiﬁcation, and setting Y ij =
j leads to the shape classiﬁcation tasks.
We may then build the data distribution for the task at hand via
[Equation]
for color and shape recognition tasks, respectively.
More generally, we may write
[Equation]
for the data distribution where the task is to recognize the k th cue.
We deﬁne the union of random variables as the balanced mixture: (cid:83) Li =
1 Z i :=
Z I where I ∼ Unif { 1 , · · · , L } .
We now introduce the notion of a diagonal dataset , where every cue ( e.g. color and shape) contains all the needed information to predict the true label Y .
That is, a perfect prediction for either color or shape attribute leads to a 100% accuracy for the task at hand.
This can be done by letting the factors always vary together i =
j in the dataset (and thus the name).
We write
[Equation]
Y ii = i ) .
[Equation]
Such a dataset completely leaves it to the model to choose the cue for recognition.
Given a model f trained on D diag , we analyse the recognition cue adopted by f by measuring its unbiased accuracy on all the cells (See Figure 1).
There are K different unbiased accuracies for each task, depending on how the off-diagonal cells are labelled: e.g. D color and D shape in equation 1.
For a general setting with K cues, the unbiased accuracy for k th cue is deﬁned as
Y= Y=
Y= Y=
Y= Y=
Y =
I n d i a n
Y =
W h
i t e
Y =
A s i a n
Y =
B l a c k
Y=infant
Y=children
Y=young adult Y=senior
[Equation]
Proposition 1.
For k ∈ { 1 , · · · , K } , acc k ( f ) = 1 if and only if f ( X i 1 , ··· ,i K ) =
i k almost surely for all 1 ≤ i 1 , · · · , i K ≤ L .
Moreover, if the condition above holds ( i.e. f is perfectly biased to cue k ), then acc m ( f ) =
1 L for all m (cid:54) =
k .
The proposition implies that the unbiased accuracy is capable of detecting the bias in a model f : acc k ( f ) = 1 implies that f ’s prediction is solely based on the cue k .
It also emphasizes that it is impossible for a model to be perfectly biased to multiple cues.
Finally, we remark that the WCST-ML analysis does not require the cues to be orthogonal or interpretable to humans.
The only requirement is the availability of the labelled samples ( X i 1 , ··· ,i K , Y i 1 , ··· ,i K ) for the cue of interest.
2. DATASETS FOR ANALYSIS
DSprites (Matthey et al., 2017) is an image dataset of symbolic objects like triangles, squares, and ellipses on black background.
It consists of images with all possible combinations of ﬁve factors: shape, scale, orientation, and X-Y position.
The total number of combinations is 3 × 6 × 40 × 32 ×
32 = 737 , 280 .
We augment the dataset with another axis of variation: 4 colors (white, red, green, and blue).
This results in 737 , 280 × 4 = 2 , 949 , 120 images in the augmented dataset.
Each image is of resolution 64 × 64 .
We have identiﬁed degenerate orientation labels due to rotational symmetries for certain shapes.
We process the data further by collapsing the symmetries to a uniﬁed orientation label.
For example, we assign the same orientation label for squares at rotations 0 , π 2 , π , and 3 π 2 .
UTKFace (Zhang et al., 2017) is a face dataset with 33 , 488 images with exhaustive annotations of age, gender, and ethnicity per image.
We utilize the Aligned and Cropped Face version to focus the variation in the facial features only.
Ages range from 0 to 116; gender is either male or female; and ethnicity is one of White, Black, Asian, Indian, and Others (e.g. Hispanic, Latino, Middle Eastern) following Zhang et al.
Images are resized from 224 × 224 to 64 × 64 .
Applying the data framework.
For each analysis, we select a subset of features S and build the training set D diag and test sets D k (§2.1) anew based on S .
Figure 2 shows examples from DSprites with S = { shape , color } and UTKFace with S
= { age , ethnicity } , respectively.
To enforce the same number of classes L per feature, we set l as the minimal number of classes among the selected features S .
For features with # classes >
L , we sub-sample the classes to match # classes =
L .
For continuous features like age and rotation, we build categories based on intervals, deﬁned such that the L classes are balanced.
We randomly vary cues / ∈ S in the constructed datasets D diag and D k .
Figure 2 shows the example datasets of D diag and D k for each dataset.
2. PARAMETER SPACE AND SOLUTIONS
We deﬁne the hypothesis class for DNNs through their weights and biases, collectively written as θ ∈ Θ ( e.g. all possible weight and bias values of the ResNet50 architecture (He et al., 2016)).
We will use a non-negative loss L ≥ 0 to characterize the ﬁt of the model f θ on data D .
0-1 loss (and cross-entropy loss) is such an example:
[Equation]
We deﬁne the solution set as Θ (cid:63) :
= { θ ∈ Θ : L ( θ ; D ) = 0 } ; in practice we deﬁne it with a small threshold L ( θ ; D ) ≤ (cid:15) .
We now make a connection between the datasets and solution sets.
Assume that we have a dataset of form ( X ij , Y ij ) where Y ij =
i (the color task; see §2.1).
The corresponding solution set is deﬁned as Θ Y ij ij :
= { θ : L ( θ ; ( X ij , Y ij = i ))
= 0 } .
Deﬁne it similarly for the shape task ( Y ij = j ).
Proposition 2.
If D = (cid:83) ( i,j ) ∈
I ( X ij , Y ij ) for some index set I , then the corresponding solution set is the intersection of the solutions sets (cid:84) ( i,j ) ∈ I Θ (cid:63)Y ij ij .
Proof.
If L never attains 0, the solution set is always empty and the statement is vacuously true.
Otherwise, we may decompose L ( θ ) into the average 1 | I | (cid:80) ( i,j ) ∈
I
L ij ( θ ) .
Due to the non-negativity of the loss function, L ( θ ) =
if and only if L ij ( θ ) =
for all ( i, j ) ∈ I .
(cid:4)
The proposition leads to a neat summary of the relations among solution sets like Θ color :
= { θ :
L ( θ ; ( X ij , i )) =
0 } and Θ shape deﬁned similarly.
Corollary 3.
Θ color ⊂ Θ diag and Θ shape ⊂ Θ diag .
Moreover, Θ color (cid:84) Θ shape = ∅ .
Proof.
The ﬁrst statement immediately follows from the proposition above and the fact that D diag =
D color (cid:84) D shape .
For the second, observe that there is no function that satisﬁes f θ ( X ij )
=
i and f θ ( X ij ) =
j at the same time for i (cid:54) =
j .
(cid:4) ΘΘΘ 3 Θ diag Θ K
−Θ K
We may easily extend the result to datasets with K cues.
Given the solutions sets Θ k for each cue 1 ≤ k ≤ K , we have Θ k ⊂ Θ diag for all k and { Θ k } k are pairwise disjoint (diagram on the right).
This follows from Proposition 1 that a single model cannot achieve low error for multiple cues ( D k ) at the same time.
O BSERVATIONS
Based on the proposed WCST-ML , we study the natural tendencies of deep neural networks (DNNs) to favor certain cues over the others.
We also present a parameter-space view that hints to the underlying mechanisms for the cue preference.
3. PREFERENTIAL ORDERING OF CUES
In the ﬁrst set of experiments, we observe which cues are preferentially used by the network when training on a dataset composed of an equally represented set of cues: the diagonal dataset D diag in §2.1.
We train three qualitatively different types of DNNs: (1) Feed Forward neural network (FFnet), (2) ResNet with depth 20 (ResNet20), and (3) Vision Transformer (ViT).
Details of the architectures and training setups can be found in Appendix
§A.
The results are summarized in Figure 3.
Models adopt cues with uneven likelihood.
Figure 3 shows the unbiased accuracies of the models when trained on the diagonal sets for each dataset (DSprites and UTKFace) with the involved cues denoted as S .
We report the diagonal accuracy and the K unbiased accuracies for each cue in S on the held-out test set.
The number of labels L is 3 for DSprites and 2 for UTKFace.
On DSprites, we observe that for all architectures, color is by far the most preferred cue (unbiased accuracy near 100%).
For FFnet and ViT, the extreme color bias forces the accuracy on other cues to be
L = 33 .
3% as predicted by Proposition 1.
ResNet20 behaves less extremely and picks up scale, shape, and orientation, in the order of preference.
On UTKFace, there is no extreme dominant cue, but there is a general preference to use ethnicity cues for making predictions, followed by gender and age cues.
The trend is clear for all models, and while the FFnet model shows a slightly more variable performance across runs, the ranking of cues is preserved throughout.
The preference of
[Figure]
For each cue, we show 3-D plots of the same loss surface (side and top views). X and Y axes are ethnicity echoes the DNN’s preference of color as a strong cue in DSprites; DNNs are drawn to utilizing skin color .
Another curious phenomenon is that none of the three cues sufﬁciently explains the 100% diagonal accuracy, suggesting the existence of an unlabeled shortcut cue other than the three considered.
Finally, when the most dominant cue is left out, other cues activate in its place: scale for DSprites and gender for UTKFace (Appendix §B).
Consistency of preferences.
We have observed a consistent preferential ordering of cues shown by qualitatively different types of architectures.
The results suggest the existence of a common denominator that depends solely on the nature of the cues, rather than the architectures.
We ﬁnally remark that the experiments were run 10 times with different random initialization; error bars in
Figure 3 show ± σ standard deviation.
The small variability observed across most training shows the consistency of the preferential ranking with respect to initialization.
For a pair of cues, we refer to the one that is more likely to be chosen by a model trained on D diag as the preferred cue and the other as the averted cue .
3. THE ABUNDANCE OF PREFERRED - CUE SOLUTIONS
Why are certain cues preferred to others?
We look for the answers in the parameter space.
More speciﬁcally, we study the geometric properties of the loss function and the corresponding solution sets in the parameter space.
The section is based on §2.3.
We write the solution set corresponding to the preferred cues as preferred solutions Θ p and averted cues as averted solutions Θ a .
How to ﬁnd the averted solutions θ ∈ Θ a ?
Naively training a DNN on D diag will most likely result in a preferred solution Θ p .
We thus use the full dataset D k corresponding to the averted cue k . (§2.1) as the training set to ﬁnd an averted solution θ ∈ Θ a .
To ensure a fair comparison in terms of the amount of training data, we also use the full dataset D k for the preferred cues.
Qualitative view on the loss surface.
We examine the geometric properties of the loss surface for
ResNet20 around different solutions types (preferred and averted).
We use the 2D visualization tool for loss landscape by Li et al.
The X-Y grid is spanned by two random vectors in R D where
D is the number of parameters.
In Figure 4, we observe that the preferred solutions ( e.g. Θ color ) are characterized with a ﬂatter and wider surface in the vicinity.
In contrast, local surface around averted ( e.g. Θ orientation ) solutions exhibit relatively sharper and narrower valleys.
A similar trend is observed for the UTKFace cues at a milder degree: Θ ethnicity is a slightly ﬂatter solution than Θ age .
Θ p Θ a basin for basin for Θ p Θ a Size of the basin of attraction.
We compare the size of the preferred Θ p and averted Θ a solution sets using the notion of the basin of attraction (BA) for a cue k : a subset N in the parameter space such that initializing the training with a point θ 0 ∈ N leads to a k -biased solution θ (cid:63) ∈
Θ k .
To measure the size of the BA for Θ k , we start by obtaining a solution θ k ∈ Θ k biased to the cue k .
We then check when the ﬁnal models trained from perturbed points θ k + v (where (cid:107) v (cid:107) 2 is small) stops being biased to the cue k .
We treat the norm of the bias-shifting critical perturbation (cid:107) v (cid:107) 2 as the size of the BA for Θ k .
Preferred solutions have wider BA.
Figure 5 shows the size of BA measured according to the method above.
The upper row shows initialization around solutions biased to cues in DSprites.
For the color cue, we observe that initializing far away from the color-biased solution still results in a color-biased solution.
For shape, scale, and orientation cues, we observe that initializing away from the respective solutions gradually results in color-biased solutions with a reduction in the bias towards the respective biases in the unperturbed initial parameters.
The sizes of basins of attraction are ordered according to color > shape > scale > orientation, largely following the preferential ordering observed in §3.1.
The lower row of Figure 5 shows the results for the three cues in UTKFace.
As opposed to DSprites, the solutions for ethnicity, age, and gender do not re-converge to a clear cue after retraining for only 50 epochs.
The sizes of basins of attraction are ordered as ethnicity > gender > age, following the preferential ordering of cues observed in §3.1.
[Figure] [Figure]
Path connectivity.
Recent works have proposed methods to ﬁnd near zeroloss paths between two solutions (Garipov et al., 2018; Draxler et al., 2018;
Benton et al., 2021).
We extend the technique to ﬁnd the zero-loss (w.r
[Figure] t. D diag ) [Figure] C
[Figure] l o r t o [Figure] n
[Figure]
E t hn i c it
y t
o g e nd e r
[Figure]
path between preferred and averted solutions. We locate the ﬂipping moment for the bias along the path; there must be a bias-shift boundary where the bias to the preferred cue shifts to the averted cue along the path (by an intermediate-value-theorem-like argument).
By comparing the length of the segment up to this boundary, we may estimate the relative abundance of the preferred and averted solutions in the parameter space.
Paths connecting preferred and averted solutions and the bias-shift boundary.
Figure 6 visualizes the paths between the preferred and averted solutions as well as the corresponding shifts in biases along the path.
We observe that the paths generally show a single shift in bias.
The shift is also usually quick; for the most part of the path, the solutions tend to be heavily biased towards either of the cues.
For the path from Θ color to Θ orientation , we observe a tendency for Θ color to dominate the path until near Θ orientation , suggesting the relative abundance of color-biased solutions Θ color in the set of diagonal solutions Θ diag compared to the orientation Θ orientation .
The path from Θ ethnicity to Θ gender is composed of a longer segment for the Θ ethnicity part.
The relative abundance of cues in the diagonal solutions Θ diag resonates the preferential ordering of cues seen in §3.1.
E XPLANATIONS
We have observed intriguing properties of cues and shortcut biases.
Even under the equal opportunities for the cues to be adopted, models have a preference for a few cues over the others.
The preference is stable across architectures and weight initialization.
We have also found a strong correlation between the preference for a cue and the abundance of solutions biased to that cue.
In this section, we explain the observations based on the complexity of cues .
Kolmogorov complexity of cues (KCC).
We formally deﬁne cue as the condition distribution p Y | X for a given input distribution p X ; in a sense, it concurs with the deﬁnition of task .
Kolmogorov complexity (KC, Kolmogorov (1963)) measures complexity of binary strings based on the minimal length among programs that generate the string.
Achille et al.
(2018) have deﬁned the Kolmogorov complexity of a task (or a cue ) p Y | X as:
[Equation]
for some sufﬁciently small scalar δ > 0 .
f denotes a discriminative model mapping X to Y and L is a suitable loss function.
K ( f ) is the KC for model f .
Intuitively, K ( p Y | X ) measures the minimal complexity of the function f required to memorize the labeling p Y | X on the training set ( i.e. L < δ ).
Measuring KCC.
We estimate the KC of four cues (color, scale, shape, and orientation) considered in DSprites based on equation 6.
Since it is impractical to optimise K ( f ) over an open-world of all possible models f , we constrain the search to a homogeneous family of models f ∈ F .
We use ResNet20 with variable channel sizes (Zagoruyko & Komodakis, 2016) for F .
Given that KC is uncomputable, we use the number of parameters in f as an approximation to K ( f ) in practice
(Valle-Perez et al., 2019).
We set the memorization criterion L ( f ; X, Y ) <
δ as “training-set error for f is < 1% ”.
Under this setup, we have veriﬁed that the complexities of cues are 1.2K parameters for color, 4.6K for scale, 17.6K for shape, and > 273 K for orientation.
We were not able to fully memorize the orientation labels under the network scales we considered.
The complexity ranking strongly concurs with the preferential ordering found in §3.1.
The simplicity bias in DNN parameter space.
How are the cue complexity and shortcut bias connected?
We explain the underlying mechanism based on a related theory in a different context: explaining the generalizability of DNNs with the simplicity bias (Valle-Perez et al., 2019; Shah et al., 2020).
It has been argued that, despite a large number of parameters, DNNs still generalize due to the inborn tendency to encode simple functions.
This phenomenon has ﬁlled the theoretical gap left by the classical learning theory.
The argument is that a (uniform) random selection of parameters for a DNN is likely to result in a function f
that is simple in the sense of KC.
More precisely, the likelihood of a function f being encoded by a DNN is governed by the inequality p ( f ) (cid:46) 2 − a (cid:101) K ( f )+ b , where K ( f ) is the KC of f and a > 0 , b are constants independent of f (Valle-Perez et al., 2019).
This inequality stems from the Algorithmic Information Theory and has been validated to hold for a large variety of systems that arise naturally, such as RNA sequences and solutions to a large class of differential equations (Dingle et al., 2018).
The argument concludes that the natural abundance of simple functions in the parameter space makes it more likely for the solutions to be simpler (Wu et al., 2017), leading to the simplicity bias and thus helping DNNs generalize well.
The simplicity bias leads to shortcuts.
We have seen that cues have different KCs, necessitating
DNNs with different complexities to fully represent them.
The simplicity bias in the parameter space explains the abundance of solutions biased to simple cues like color Θ color rather than complex cues like orientation Θ orientation (§3.2).
The difference in the abundance in turn makes it more likely for a model trained on the diagonal dataset D diag to be biased towards simple cues (§3.1).
D ISCUSSION AND C ONCLUSION
We have delved into the shortcut learning phenomenon in deep neural networks (DNNs).
We have devised a training setup WCST-ML where multiple cues are equally valid for solving the task at hand and have veriﬁed that, despite the equal validity, DNNs tend to choose cues in a certain order ( e.g. color is preferred to orientation).
We have shown that the simple cues (in the sense of Kolmogorov) are far more frequently represented in the parameter space and thus are far more likely to be adopted by DNNs.
We conclude the paper with two ﬁnal remarks on the implications of our studies.
Simplicity bias and generalization.
The simplicity bias theory (Valle-Perez et al., 2019) has been developed to explain the unreasonably good generalizability of DNNs.
Our paper shows that it may interfere with the generalization.
This apparent paradox stems from the difference in the setup.
The simplicity bias is often safe to be exploited in the standard iid setup, albeit with caveats argued by
Shah et al.
On the other hand, it may lead to less effective generalization across distribution shifts as in our learning setup (Geirhos et al., 2020), if the simple cues are no longer generalizable.
Thus, for the generalization across distribution shifts, machine learners may need to be given additional guidance to overcome the inborn tendencies to pick up simple cues for recognition.
Societal implications.
An issue with the simplicity bias and shortcut learning is that relying on simple cues is sometimes unethical.
We have seen in the UTKFace experiments that ethnicity tends to be a more convenient shortcut than age , for example, if they equally correlate with the target label at hand.
This is an alarming phenomenon.
It emphasizes the importance and inevitability of active human intervention to undo certain naturally-arising biases to socially harmful cues in learned models.
It also conﬁrms that we are on the right track to actively regularize the models to behave according to the social norms (Barocas et al., 2017; Mehrabi et al., 2021).
Toolbox contribution.
Along the way, we contribute interesting novel tools that future research can utilise, such as the computation of Kolmogorov complexity (KC) for generic cues (equation 6) and the method for ﬁnding a heterogeneous zero-loss curve connecting two solutions biased to different cues (§3.2; Garipov et al.
REPRODUCIBILITY
We understand and resonate with the importance of reproducible research practices in machine learning.
The reproducibility of this paper hinges on our use of open-source data, models, training techniques, and deep learning frameworks.
We have speciﬁed a detailed experimental setup in §2.2 and §A. For the consistency of experimental observations, we have run the experiments multiple times with different random seeds, whenever applicable.
For those cases, we have reported the mean and standard deviation for each data point.
ETHICS STATEMENT
We delve into the potential ethical issues with DNNs.
Our work excavates the potential ethical harms caused by deep neural networks’ natural tendencies to prefer simple cues, where using the simple cues may not conform well to the social norm ( e g. ethnicity cue).
We have thus advocated an active human involvement to regularize such DNNs to adopt more socially acceptable cues.
As for the experiments, part of our studies is based on a face dataset (UTKFace).
We conﬁrm that the UTKFace dataset has been released with an open-source license with the non-commercial clause 2 .
The copyright for the images still belongs to their respective owners.
Inheriting the policy of UTKFace, we will erase the face samples in Figure 2 upon request by the owners of the respective images.
A M ODEL A RCHITECTURES
FFnet.
The FFnet architecture tested is a fully connected feedforward neural network with 5 hidden layers with respectively 4096, 2048, 600, 300 and 100 hidden units.
ReLU activations are used throughout.
We train it using the Adam optimizer with the default parameters given by PyTorch 1.8.0.
ResNet20.
The ResNet20 architecture is adapted from He et al.
(2016), with a depth of 20.
We train it using the Adam optimizer with the default parameters given by PyTorch 1.8.0.
ViT. The Visual Transformer architecture is adapted from (Touvron et al., 2021).
Given the dimensions of the images in ColorDSprites and consequently UTKFace, we implement a ViT model with patch size 8, embedding dimension 192, depth 12, and 3 attention heads.
We train the model with the SGD optimizer with learning rate 5 × 10 − 3 , momentum 0 .
9 , and weight decay 1 × 10 − 4 .
B P REFERENTIAL O RDERING AFTER R EMOVING THE D OMINANT C UE
For both DSprites and UTKFace, removing the dominant cue (color and ethnicity respectively) from S does not change the ranking of the remaining cues; e.g. scale and gender then become the dominant features in respective datasets.
See Figure B.1 that extends the main paper Figure 3.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
ce
S =
[Figure]
nd
[Figure]
ViT
C L OSS C URVATURE AROUND S OLUTIONS
We supplement the visualization of loss landscape around solutions in Figure 4 of main paper with quantitative estimation of the respective curvatures.
The mean curvature of a surface deﬁned by function f is deﬁned as:
[Equation]
To make it computationally tractable for large D ( e.g. DNN parameters), we estimate the curvature with a Monte-Carlo estimation as done in Izmailov et al.
For a point x ∈ R D , a sufﬁciently small scalar (cid:15) > 0 , and a unit sphere S D − 1 ⊂ R D , the curvature is computed by:
[Equation]
where r ∼ Unif ( S D − 1 ) .
[Equation]
How does this approximate the mean curvature?
We provide the following proposition.
Proposition 4.
For a point x ∈ R D , a scalar (cid:15) > 0 , a twice-differentiable function f , and a unit sphere S D − 1 ⊂ R D , the following quantity
[Equation]
where r ∼ Unif ( S D )
[Equation]
[Equation]
approximates the Laplacian ∆ f
up to a constant multiplier.
Proof.
We begin with a Taylor expansion on f ( x + (cid:15)r ) :
[Equation]
where h ( (cid:15),r ) (cid:15)→ 0 as (cid:15) ↓ 0 for any r .
Thus,
[Equation]
by E r = 0 and the Dominated Convergence Theorem for Lebesgue integrals (also noting that r is deﬁned on a compact set S D − 1 ).
We now apply a change of basis such that the Hessian ∇ 2 f ( x ) is diagonalised: ∇ 2 f ( x ) =
Q T Λ Q .
This is possible because the Hessian is a real, symmetric matrix.
Since the change of basis do not alter the distribution for r , we compute
[Equation]
using the isotropicity of r and the preservation of the trace against changes of basis.
We thus conclude
[Equation]
where C is a constant independent of f and x
[Figure]
[Equation]
Results.
Figure C.1 shows the approximated curvature at varying radius values (cid:15) around different solutions for ResNet20.
We have used 100 Monte-Carlo samples.
We observe that for DSprites, the curvature of the orientation loss surface tends to overshoot quickly as one walks away from the solution point.
Surfaces for other cues tend to be smoother.
On UTKFace, gender tends to provide a steeper surface than the other cues like age and ethnicity
[Figure]