Original: Signal Strength and Noise Drive Feature Preference in CNN Image Classiﬁers
Revised: Signal Strength and Noise Drive Feature Preference in CNN Image Classiﬁers

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Feature preference in Convolutional Neural Network (CNN) image classiﬁers isintegral to their decision making process, and while the topic has been well studied,it is still not understood at a fundamental level.
Revised: Feature preference in Convolutional Neural Network (CNN) image classiﬁers is integral to their decision making process, and while the topic has been well studied, it is still not understood at a fundamental level.

Original: We test a range of task relevantfeature attributes (including shape, texture, and color) with varying degrees ofsignal and noise in highly controlled CNN image classiﬁcation experiments usingsynthetic datasets to determine feature preferences.
Revised: We test a range of task relevant feature attributes (including shape, texture, and color) with varying degrees of signal and noise in highly controlled CNN image classiﬁcation experiments using synthetic datasets to determine feature preferences.

Original: We ﬁnd that CNNs will preferfeatures with stronger signal strength and lower noise irrespective of whetherthe feature is texture, shape, or color.
Revised: We ﬁnd that CNNs will prefer features with stronger signal strength and lower noise irrespective of whether the feature is texture, shape, or color.

Original: This provides guidance for a predictivemodel for task relevant feature preferences, demonstrates pathways for bias inmachine models that can be avoided with careful controls on experimental setup,and suggests that comparisons between how humans and machines prefer taskrelevant features in vision classiﬁcation tasks should be revisited.
Revised: This provides guidance for a predictive model for task relevant feature preferences, demonstrates pathways for bias in machine models that can be avoided with careful controls on experimental setup, and suggests that comparisons between how humans and machines prefer task relevant features in vision classiﬁcation tasks should be revisited.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks (DNNs) can be used for a wide range of tasks, yet we do not yet have afundamental understanding of how DNNs actually perform many of these tasks.
Revised: Deep neural networks (DNNs) can be used for a wide range of tasks, yet we do not yet have a fundamental understanding of how DNNs actually perform many of these tasks.

Original: In this paper wefocus on image classiﬁcation and seek to explain why image classiﬁers select certain features of theinput space over others.
Revised: In this paper we focus on image classiﬁcation and seek to explain why image classiﬁers select certain features of the input space over others.

Original: Feature preference in CNNs has been explored through prior research and the results often suggestthat machines classify images very differently from humans.
Revised: Feature preference in CNNs has been explored through prior research and the results often suggest that machines classify images very differently from humans.

Original: Adversarial example research hasshown that CNN classiﬁers can be easily fooled by small and imperceptible (at least to humans)manipulations to inputs.
Revised: Adversarial example research has shown that CNN classiﬁers can be easily fooled by small and imperceptible (at least to humans) manipulations to inputs.

Original: Ilyas et al.
Revised: Ilyas et al.

Original: (2019) suggest that CNN classiﬁers key off of widespreadnon-robust brittle features that are present in the dataset but imperceptible to humans leading to amisalignment with human expectation.
Revised: (2019) suggest that CNN classiﬁers key off of widespread non-robust brittle features that are present in the dataset but imperceptible to humans leading to a misalignment with human expectation.

Original: Jacobsen et al.
Revised: Jacobsen et al.

Original: (2019) suggest that one type of adversarialvulnerability is a result of narrow learning and is caused by an overreliance on a few highly predictivefeatures in their decisions rendering the models excessively invariant.
Revised: (2019) suggest that one type of adversarial vulnerability is a result of narrow learning and is caused by an overreliance on a few highly predictive features in their decisions rendering the models excessively invariant.

Original: They suggest this is a resultof cross-entropy: maximizing the bound of mutual information between the labels and the features.
Revised: They suggest this is a result of cross-entropy: maximizing the bound of mutual information between the labels and the features.

Original: However they  models lock in on some highly predictive features and others
Revised: However, they do not offer an explanation for why models lock in on some highly predictive features and ignore others.

Original:  (2019) developed a benchmark to test the robustness of
Revised: Hendrycks & Dietterich (2019) developed a benchmark to test the robustness of

Original: CNNs to perturbations and corruptions.
Revised: CNNs to perturbations and corruptions.

Original: Rusak et al.
Revised: Rusak et al.

Original: (2020) pointed out that the human visual systemis generally robust to a wide range of image noise but that machine models strongly degrade withvarious types of unseen corruptions.
Revised: (2020) pointed out that the human visual system is generally robust to a wide range of image noise but that machine models strongly degrade with various types of unseen corruptions.

Original: Bias in machine models is a very important topic that is being actively investigated.
Revised: Bias in machine models is a very important topic that is being actively investigated.

Original: Geirhos et al.(2019) designed a set of cue conﬂict experiments to compare how machines and humans classify
Revised: Geirhos et al. (2019) designed a set of cue conﬂict experiments to compare how machines and humans classify

Original: ImageNet objects.
Revised: ImageNet objects.

Original: When ImageNet-trained CNNs were fed images with conﬂicting shape and texturefeatures, the results showed that while humans preferred to classify these cue conﬂicts according  shape, the machines preferred to classify the images according to texture, which was described astexture bias.
Revised: When ImageNet-trained CNNs were fed images with conﬂicting shape and texture features, the results showed that while humans preferred to classify these cue conﬂicts according to shape, the machines preferred to classify the images according to texture, which was described as texture bias.

Original: Subsequent research by Hermann et al.
Revised: Subsequent research by Hermann et al.

Original: (2019) showed that by augmenting the trainingdata of an ImageNet CNN, they were able to increase shape bias, and concluded that texture bias isnot an inductive bias of the underlying model.
Revised: (2019) showed that by augmenting the training data of an ImageNet CNN, they were able to increase shape bias, and concluded that texture bias is not an inductive bias of the underlying model.

Original: Another machine bias was identiﬁed by Shah et al.(2020)–simplicity bias–which they described as the tendency for neural network classiﬁers trainedwith gradient descent to form the “simple” decision boundary before a more robust “complex” one.
Revised: Another machine bias was identiﬁed by Shah et al. (2020)–simplicity bias–which they described as the tendency for neural network classiﬁers trained with gradient descent to form the “simple” decision boundary before a more robust “complex” one.

Original: They suggest that simplicity bias can limit the robustness and generalization of machine models.
Revised: They suggest that simplicity bias can limit the robustness and generalization of machine models.

Original: Geirhos et al.
Revised: Geirhos et al.

Original: (2020) outlines various types of DNN biases and unintended failure examples whichthey describe as shortcut learning.
Revised: (2020) outlines various types of DNN biases and unintended failure examples which they describe as shortcut learning.

Original: This occurs when solutions are found to tasks that are not theresult of learning the intended human-like solution.
Revised: This occurs when solutions are found to tasks that are not the result of learning the intended human-like solution.

Original: Hermann & Lampinen (2020) used syntheticimage datasets and found that the “easiness” and “predictivity” (how often the feature predicts aclass) were positively correlated with a CNN’s preference for that feature.
Revised: Hermann & Lampinen (2020) used synthetic image datasets and found that the “easiness” and “predictivity” (how often the feature predicts a class) were positively correlated with a CNN’s preference for that feature.

Original: Most of the previous work on feature preference in CNNs has shown that machine models willprefer “easy” or “simple” features over more “complex” or “difﬁcult” features, and this can lead toerrors, biases, and misalignment between machine and human vision.
Revised: Most of the previous work on feature preference in CNNs has shown that machine models will prefer “easy” or “simple” features over more “complex” or “difﬁcult” features, and this can lead to errors, biases, and misalignment between machine and human vision.

Original: In this work, we present afoundation for what actually makes task relevant features “harder” or “easier” for CNNs to identify (and ultimately use) in classiﬁcation tasks
Revised: In this work, we present a foundation for what actually makes task relevant features “harder” or “easier” for CNNs to identify (and ultimately use) in classiﬁcation tasks.

Original: Contributions
Revised: Contributions

Original: • CNNs will prefer task relevant features that are represented with larger signal—larger numberof pixels—over task relevant features that are represented by smaller signal—smaller numberof pixels.
Revised: • CNNs will prefer task relevant features that are represented with larger signal—larger number of pixels—over task relevant features that are represented by smaller signal—smaller number of pixels.

Original: • CNNs will prefer task relevant features that are represented with lower noise.
Revised: • CNNs will prefer task relevant features that are represented with lower noise.

Original: We identifyseveral feature attributes that lower preference including: deviation, overlap, and predictivity.• CNNs show no strong preference between color, shape, or texture—feature equivalency—when signal and noise are carefully controlled.
Revised: We identify several feature attributes that lower preference including: deviation, overlap, and predictivity. • CNNs show no strong preference between color, shape, or texture—feature equivalency— when signal and noise are carefully controlled.

Original:  Methods
Revised:  Methods

Original: To understand what features will be preferred by a CNN image classiﬁer in a controlled environment,we construct pairs matrix datasets for ten different synthetic features.
Revised: To understand what features will be preferred by a CNN image classiﬁer in a controlled environment, we construct pairs matrix datasets for ten different synthetic features.

Original: Speciﬁcally, we create ninedifferent sets of ﬁve classes, with each class representing one of the 45 combinations of the tensynthetic features.
Revised: Speciﬁcally, we create nine different sets of ﬁve classes, with each class representing one of the 45 combinations of the ten synthetic features.

Original: Within each class, two of the ten features are randomly placed against a blackbackground.
Revised: Within each class, two of the ten features are randomly placed against a black background.

Original: We construct the sets such that every feature gets placed within a class with every otherfeature only once, and every feature occurs only once within each set.
Revised: We construct the sets such that every feature gets placed within a class with every other feature only once, and every feature occurs only once within each set.

Original: We then train a ResNet-18 Heet{+ResNet-
 He et+} al.
Revised: We then train a {+ResNet-
 He et+} al.

Original: (2015) on each set using a modiﬁed version of the torchvision ImageNet training script.
Revised: (2015) on each set using a modiﬁed version of the torchvision ImageNet training script.

Original: For each pair matrix, we construct a cue conﬂict test set where every combination of two of features(45 in total) are rendered together.
Revised: For each pair matrix, we construct a cue conﬂict test set where every combination of two of features (45 in total) are rendered together.

Original: After a ResNet-18 is trained, it is fed these cue conﬂict testimages to determine the feature preference between all features in the pairs matrix
Revised: After a ResNet-18 is trained, it is fed these cue conﬂict test images to determine the feature preference between all features in the pairs matrix

Original: and then the overall preference scores for each feature is aggregated to generate feature preference rankings.
Revised: and then the overall preference scores for each feature is aggregated to generate feature preference rankings.

Original: Inthese experiments, we measure preference by dividing the total number of times a feature’s class ispredicted in cue conﬂict images containing the feature by the total number of images containing thefeature in the cue conﬂict test set.
Revised: In these experiments, we measure preference by dividing the total number of times a feature’s class is predicted in cue conﬂict images containing the feature by the total number of images containing the feature in the cue conﬂict test set.

Original: We render 300 images per class for training, 100 images per class for validation, 100 images perclass for testing, and 100 images per feature combination during cue conﬂict testing.
Revised: We render 300 images per class for training, 100 images per class for validation, 100 images per class for testing, and 100 images per feature combination during cue conﬂict testing.

Original: We create onedataset per set, and average preference results across ﬁve training runs.
Revised: We create one dataset per set, and average preference results across ﬁve training runs.

Original: All models are trained for90 epochs with SGD using learning rate 0.1, which gets decayed by 0.1 at epochs 30 and 60, andwith weight decay 0.0001.
Revised: All models are trained for 90 epochs with SGD using learning rate 0.1, which gets decayed by 0.1 at epochs 30 and 60, and with weight decay 0.0001.

Original: Images are normalized by ImageNet per-channel means and standarddeviations.
Revised: Images are normalized by ImageNet per-channel means and standard deviations.

Original: Features in training images are placed within a 192x192 box, padded with 32 pixels oneach side, and the 256x256 result is randomly cropped into a 224x224 image.
Revised: Features in training images are placed within a 192x192 box, padded with 32 pixels on each side, and the 256x256 result is randomly cropped into a 224x224 image.

Original: This procedure is usedfor experiments detailed in Section 3.
Revised: This procedure is used for experiments detailed in Section 3.

Original:  Factors That Inﬂuence Feature Preference
Revised:  Factors That Inﬂuence Feature Preference

Original: In this section we present the results from our pairs matrix experiments and explore the factors thateither increase or decrease feature preference.
Revised: In this section we present the results from our pairs matrix experiments and explore the factors that either increase or decrease feature preference.

Original: Pixel Count
Revised: Pixel Count

Original: We ﬁnd that when variables are controlled, there is a high correlation between the number of pixelsused to represent a feature and that feature’s preference.
Revised: We ﬁnd that when variables are controlled, there is a high correlation between the number of pixels used to represent a feature and that feature’s preference.

Original: Speciﬁcally, we construct a pairs matrix thatcontains three elementary shapes (triangle, square, and plus), four different colors contained within acircle (red, green, blue, and yellow), and three different textures (blocky, wavy, and banded).
Revised: Speciﬁcally, we construct a pairs matrix that contains three elementary shapes (triangle, square, and plus), four different colors contained within a circle (red, green, blue, and yellow), and three different textures (blocky, wavy, and banded).

Original: We varythe pixel count for each of the ten features and test for preference.
Revised: We vary the pixel count for each of the ten features and test for preference.

Original: Within these ten features, we create three feature groups, where each group contains features withapproximately the  number of pixels: one group contains one shape, color, and texture with a largenumber of pixels; one group contains one shape, color and texture with a small number of pixels; andone group contains one shape, color, and texture with a medium number of pixels.
Revised: Within these ten features, we create three feature groups, where each group contains features with approximately the same number of pixels: one group contains one shape, color, and texture with a large number of pixels; one group contains one shape, color and texture with a small number of pixels; and one group contains one shape, color, and texture with a medium number of pixels.

Original: We have one sparecolor that is inserted into the medium pixel group.
Revised: We have one spare color that is inserted into the medium pixel group.

Original: We observe a strong correlation between the number of pixels that deﬁne a feature and the averagepreference given to that feature, which can be seen in Figure 3.
Revised: We observe a strong correlation between the number of pixels that deﬁne a feature and the average preference given to that feature, which can be seen in Figure 3.

Original: Large and small feature groups arereversed in Figure 3 (a) and (b), but the correlation between the number of pixels and preferenceremains.
Revised: Large and small feature groups are reversed in Figure 3 (a) and (b), but the correlation between the number of pixels and preference remains.

Original: Moreover, this correlation holds across various feature types including shapes, colors, andtextures.
Revised: Moreover, this correlation holds across various feature types including shapes, colors, and textures.

Original: This shows that for CNN classiﬁcation the number of pixels that represent a task relevantfeature deﬁnes signal strength, which in turn drives feature preference.
Revised: This shows that for CNN classiﬁcation the number of pixels that represent a task relevant feature deﬁnes signal strength, which in turn drives feature preference.

Original: Importantly, when signalstrength is normalized, the CNN shows feature equivalency; no preference for color, shape, or texturefeatures was observed.
Revised: Importantly, when signal strength is normalized, the CNN shows feature equivalency; no preference for color, shape, or texture features was observed.

Original: Deviation
Revised: Deviation

Original: Deviation on task relevant features is a common characteristic of popular classiﬁcation datasets suchas MNIST and ImageNet, and increasing deviation will make a classiﬁcation task more difﬁcult.
Revised: Deviation on task relevant features is a common characteristic of popular classiﬁcation datasets such as MNIST and ImageNet, and increasing deviation will make a classiﬁcation task more difﬁcult.

Original: For example, a handwritten seven might exist in two variations: with a horizontal line through thecenter and without; a handwritten digit classiﬁcation model would have to learn both.
Revised: For example, a handwritten seven might exist in two variations: with a horizontal line through the center and without; a handwritten digit classiﬁcation model would have to learn both.

Original: Thus, it followsthat adding deviation to a task relevant feature will likely make the feature less preferred by a CNN,as it increases the difﬁculty for a model to capture that feature.
Revised: Thus, it follows that adding deviation to a task relevant feature will likely make the feature less preferred by a CNN, as it increases the difﬁculty for a model to capture that feature.

Original: Once we established controls forsignal strength above, we moved to the second set of experiments which focused on quantifying theeffects of different types of deviation on task relevant feature preference.
Revised: Once we established controls for signal strength above, we moved to the second set of experiments which focused on quantifying the effects of different types of deviation on task relevant feature preference.

Original: For these experiments,we conducted pair matrix experiments similar to what was used for signal strength, but we addeddeviation to the hue of a color circle during training.
Revised: For these experiments, we conducted pair matrix experiments similar to what was used for signal strength, but we added deviation to the hue of a color circle during training.

Original: As displayed in Figure 4, the amount of huedeviation added to a color during training is linearly correlated with the preference of that feature.
Revised: As displayed in Figure 4, the amount of hue deviation added to a color during training is linearly correlated with the preference of that feature.

Original: Overlap
Revised: Overlap

Original: We also hypothesize that inter-class feature overlap has a negative effect on feature preference.
Revised: We also hypothesize that inter-class feature overlap has a negative effect on feature preference.

Original: In thisexperiment, we linearly interpolate the blue color circle to the green color circle, and keep their pixelcounts the same.
Revised: In this experiment, we linearly interpolate the blue color circle to the green color circle, and keep their pixel counts the same.

Original: All aspects of every feature in Pairs Matrix 1 are kept constant, except for the bluecircle feature which we bring down to the medium pixel feature group.
Revised: All aspects of every feature in Pairs Matrix 1 are kept constant, except for the blue circle feature which we bring down to the medium pixel feature group.

Original: Like deviation, interpolatingtwo features together linearly decreases the preference of the CNN for both features, as can be seen
Revised: Like deviation, interpolating two features together linearly decreases the preference of the CNN for both features, as can be seen

Original: [Figure]
Revised: [Figure]

Original: r e f e r e n c e
Revised: r e f e r e n c e

Original: R
Revised: R

Original: [Figure]
Revised: [Figure]

Original: bluegreenplusredsquarebandedblockywavytriangleyellow
Revised: bluegreenplusredsquarebandedblockywavytriangleyellow

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in Figure 5.
Revised: in Figure 5.

Original: If class relevant features are interpolated together, then each feature loses the predictivepower they hold for their respective classes.
Revised: If class relevant features are interpolated together, then each feature loses the predictive power they hold for their respective classes.

Original: Predictivity
Revised: Predictivity

Original: Drawing from experiments described by Hermann & Lampinen (2020), we conducted experimentswhere we varied the frequency that a feature is present in its given class for each set in a pairs matrixexperiment.
Revised: Drawing from experiments described by Hermann & Lampinen (2020), we conducted experiments where we varied the frequency that a feature is present in its given class for each set in a pairs matrix experiment.

Original: For example, if the predictivity of a feature is set to 50%, the feature is present in only50% of training instances for that class.
Revised: For example, if the predictivity of a feature is set to 50%, the feature is present in only 50% of training instances for that class.

Original: As seen in Figure 6, decreasing a feature’s predictivity willcause a decrease in the preference in a pairs matrix experiment.
Revised: As seen in Figure 6, decreasing a feature’s predictivity will cause a decrease in the preference in a pairs matrix experiment.

Original: Like inter-class overlap betweenfeatures, decreasing a feature’s predictivity decreases that feature’s predictive power for its respectiveclass, which will result in a lower feature preference relative to other predictive feature options.
Revised: Like inter-class overlap between features, decreasing a feature’s predictivity decreases that feature’s predictive power for its respective class, which will result in a lower feature preference relative to other predictive feature options.

Original: [Figure]
Revised: [Figure]

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: The results of these experiments show that CNNs exhibit signal preference.
Revised: The results of these experiments show that CNNs exhibit signal preference.

Original: For vision recognition,the base signal of a feature may be described as the number of pixels used to represent that feature.
Revised: For vision recognition, the base signal of a feature may be described as the number of pixels used to represent that feature.

Original: Increasing the signal will increase the preference for that feature over other task relevant featureswith lower signal assuming all other variables are controlled as shown in Figure 3.
Revised: Increasing the signal will increase the preference for that feature over other task relevant features with lower signal assuming all other variables are controlled as shown in Figure 3.

Original: The results alsoshow that increasing noise factors which make the feature more difﬁcult to capture or decrease thepredictive power of the feature will lower feature preference as shown in Figures 4, 5, and 6.
Revised: The results also show that increasing noise factors which make the feature more difﬁcult to capture or decrease the predictive power of the feature will lower feature preference as shown in Figures 4, 5, and 6.

Original: Forvision recognition, noise includes 1) deviation between feature instances within a class 2) inter-classoverlap between a task relevant feature and another task relevant feature, and 3) predictivity (presenceor absence of the feature on some class instances in the dataset).
Revised: For vision recognition, noise includes 1) deviation between feature instances within a class 2) inter-class overlap between a task relevant feature and another task relevant feature, and 3) predictivity (presence or absence of the feature on some class instances in the dataset).

Original: While performance of CNN image classiﬁers has surpassed the capability of humans, we still lack afundamental, formal understanding of how CNNs perform vision tasks.
Revised: While performance of CNN image classiﬁers has surpassed the capability of humans, we still lack a fundamental, formal understanding of how CNNs perform vision tasks.

Original: Gaining this understandingwill not only inform the development of safe and interpretable vision models, but also has the potentialto provide insight into how human vision functions as well.
Revised: Gaining this understanding will not only inform the development of safe and interpretable vision models, but also has the potential to provide insight into how human vision functions as well.

Original: Essential to understanding visual processing systems is a predictive model for such a system’spreference for features in its input space.
Revised: Essential to understanding visual processing systems is a predictive model for such a system’s preference for features in its input space.

Original: Based on the results of this paper, we propose a model forfeature preference generally expressed as:
Revised: Based on the results of this paper, we propose a model for feature preference generally expressed as:

Original: F(pr) =
Revised: [Equation]

Original: Signal (pixel count) - Noise (Deviation + Overlap + Predictivity)

Original: Having a predictive model for CNN feature preference has the potential to help in a range of researchtopics including interpretability, new data augmentation strategies and training objectives to expandthe range of task relevant features that are included thereby potentially improving generalizability,robustness, and accuracy for some tasks and datasets.
Revised: Having a predictive model for CNN feature preference has the potential to help in a range of research topics including interpretability, new data augmentation strategies and training objectives to expand the range of task relevant features that are included thereby potentially improving generalizability, robustness, and accuracy for some tasks and datasets.

Original: While working towards the predictive modelwe also found a pathway to ascribe biases to machine models that might be an artifact of someexperimental setups.
Revised: While working towards the predictive model we also found a pathway to ascribe biases to machine models that might be an artifact of some experimental setups.

Original: In particular, if Signal and Noise are not carefully controlled for, it is possible toﬁnd or mask feature preferences based on the test set that is used without needing to make changes toa trained model or dataset.
Revised: In particular, if Signal and Noise are not carefully controlled for, it is possible to ﬁnd or mask feature preferences based on the test set that is used without needing to make changes to a trained model or dataset.

Original: For example, in our synthetic dataset, we can easily show preference forcolors, shapes, or texture features by simply making the desired feature be deﬁned by more pixels inthe test set.
Revised: For example, in our synthetic dataset, we can easily show preference for colors, shapes, or texture features by simply making the desired feature be deﬁned by more pixels in the test set.

Original: We can also shift the feature preference by adding or removing deviation, overlap, orpredictivity.
Revised: We can also shift the feature preference by adding or removing deviation, overlap, or predictivity.

Original: We also consider the impact that these experiments might have on the comparisons that have beenmade between machine and human vision.
Revised: We also consider the impact that these experiments might have on the comparisons that have been made between machine and human vision.

Original: When tasks and datasets are carefully controlled for Signaland Noise, we expect that feature preferences of machines and humans move closer in alignment,but this must be tested experimentally, and should be explored in future work.
Revised: When tasks and datasets are carefully controlled for Signal and Noise, we expect that feature preferences of machines and humans move closer in alignment, but this must be tested experimentally, and should be explored in future work.

Original: Future work shouldalso test for the extensibility of these results across other tasks (including unsupervised objectives),datasets, data augmentation strategies, and architectures.
Revised: Future work should also test for the extensibility of these results across other tasks (including unsupervised objectives), datasets, data augmentation strategies, and architectures.

Original: The results may also inform DNN learningtheory.
Revised: The results may also inform DNN learning theory.

Original: A Revisiting Texture Bias
Revised: A Revisiting Texture Bias

Original: [Figure]
Revised: [Figure]

Original: M ODIFICATION

Original: [Table]
Revised: [Table]

Original: EXTURE

Original: [Table]

Original: A.1 Methods
Revised: A.1 Methods

Original: To measure the texture bias of ImageNet-trained CNNs, we follow the procedure outlined by Geirhoset al.
Revised: To measure the texture bias of ImageNet-trained CNNs, we follow the procedure outlined by Geirhos et al.

Original: We use the style transfer shape-texture cue conﬂict and silhouette stimuli open-sourcedby the authors.
Revised: We use the style transfer shape-texture cue conﬂict and silhouette stimuli open-sourced by the authors.

Original: We keep the texture bias measurement procedure exactly the same, and modify onlythe test images.
Revised: We keep the texture bias measurement procedure exactly the same, and modify only the test images.

Original: Details and results of these experiments are contained in Section A.2.
Revised: Details and results of these experiments are contained in Section A.2.

Original: All texturebias measurements were recorded using a torchvision ResNet-50.
Revised: All texture bias measurements were recorded using a torchvision ResNet-50.

Original: A.2 Results
Revised: A.2 Results

Original: Geirhos et al.
Revised: Geirhos et al.

Original: (2019) showed that ImageNet-trained CNNs will classify an object according to itstexture rather than shape in what they described as texture bias.
Revised: (2019) showed that ImageNet-trained CNNs will classify an object according to its texture rather than shape in what they described as texture bias.

Original: The result is intriguing and importantsince it suggests that ImageNet trained models seem to classify objects quite differently from humans.
Revised: The result is intriguing and important since it suggests that ImageNet trained models seem to classify objects quite differently from humans.

Original: When we reviewed the cue conﬂict experiment of Geirhos et al.
Revised: When we reviewed the cue conﬂict experiment of Geirhos et al.

Original: the test images appeared tocontain a disproportionate amount of texture signal compared to shape signal which could potentiallyskew the results towards texture bias.
Revised: the test images appeared to contain a disproportionate amount of texture signal compared to shape signal which could potentially skew the results towards texture bias.

Original: The reason for this is 1) during the style-transfer process, atexture will get mapped over the entire image, while the shape remains ﬁxed in a portion of the imageresulting in a large pixel count imbalance between texture and shape (in favor of texture) and 2)the stye-transfer process often distorts shape information.
Revised: The reason for this is 1) during the style-transfer process, a texture will get mapped over the entire image, while the shape remains ﬁxed in a portion of the image resulting in a large pixel count imbalance between texture and shape (in favor of texture) and 2) the stye-transfer process often distorts shape information.

Original: We revisit the experiment to see if ourhypothesis on signal and deviation can be used to gain increased control over the cue conﬂict test setand then study whether the conclusion on texture bias still holds.
Revised: We revisit the experiment to see if our hypothesis on signal and deviation can be used to gain increased control over the cue conﬂict test set and then study whether the conclusion on texture bias still holds.

Original: In the experiments presented in thissection, we modify the texture-shape cue conﬂict images used in Geirhos et al.
Revised: In the experiments presented in this section, we modify the texture-shape cue conﬂict images used in Geirhos et al.

Original:  so that thetexture and shape signals (number of pixels) in each feature are varied in a controlled manner relativeto each other and the cue conﬂict preference is measured.
Revised: (2019) so that the texture and shape signals (number of pixels) in each feature are varied in a controlled manner relative to each other and the cue conﬂict preference is measured.

Original: Masking
Revised: Masking

Original: To mitigate the effect of the texture-shape signal imbalance in the cue conﬂict experiment, we maskthe background around the shape in each image.
Revised: To mitigate the effect of the texture-shape signal imbalance in the cue conﬂict experiment, we mask the background around the shape in each image.

Original: This eliminates texture signal outside of the object,and increases shape signal by increasing the contrast between the background and silhouette of theshape.
Revised: This eliminates texture signal outside of the object, and increases shape signal by increasing the contrast between the background and silhouette of the shape.

Original: We ﬁnd that after performing this masking operation, ImageNet-trained CNNs show a preferencefor shape (66% shape bias, 34% texture bias) following the testing process that Geirhos et al.
Revised: We ﬁnd that after performing this masking operation, ImageNet-trained CNNs show a preference for shape (66% shape bias, 34% texture bias) following the testing process that Geirhos et al.

Original: (2019)described.
Revised: (2019) described.

Original: While Geirhos et al.
Revised: While Geirhos et al.

Original: and Baker et al.
Revised: and Baker et al.

Original: (2018) conducted a similar experiment thatstill resulted in a texture bias, there was a key difference that we believe led to a different result fromour experiment.
Revised: (2018) conducted a similar experiment that still resulted in a texture bias, there was a key difference that we believe led to a different result from our experiment.

Original: In experiments performed in Geirhos et al.
Revised: In experiments performed in Geirhos et al.

Original: and Baker et al.
Revised: and Baker et al.

Original: (2018), a texturewas mapped onto the silhouette of a shape whereas in our experiment the texture was mapped ontothe object using style-transfer.
Revised: (2018), a texture was mapped onto the silhouette of a shape whereas in our experiment the texture was mapped onto the object using style-transfer.

Original: Silhouettes do not contain all the shape signal, as there is some shapeinformation contained within an object that gets removed in a silhouette representation.
Revised: Silhouettes do not contain all the shape signal, as there is some shape information contained within an object that gets removed in a silhouette representation.

Original: In contrast,the style-transfer process preserves these important shape signals, so we believe that our experimentscreated a more accurate comparison between shape and texture preference in ImageNet-trained
Revised: In contrast, the style-transfer process preserves these important shape signals, so we believe that our experiments created a more accurate comparison between shape and texture preference in ImageNet-trained

Original: CNNs.
Revised: CNNs.

Original: 1.0 Interpolation 0.
Revised: 1.0 Interpolation 0.

Original: T e x t u r e
Revised: T e x t u r e

Original: P r e f e r e n c e
Revised: P r e f e r e n c e

Original: R=0.
Revised: R=0.

Original: conﬂict texture preference is linearly correlated with the signal strength of the cueconﬂict background.
Revised: conﬂict texture preference is linearly correlated with the signal strength of the cueconﬂict background.

Original: In this experiment, the background of each cue conﬂict image was masked to white, and then interpolated towards the original cue-conﬂict background.
Revised: In this experiment, the background of each cue conﬂict image was masked to white, and then interpolated towards the original cue-conﬂict background.

Original: Higher values of interpolation indicate higher similarity to the original cue-conﬂict background.
Revised: Higher values of interpolation indicate higher similarity to the original cue-conﬂict background.

Original: As the background becomes more similar to the texture cue it increases the overall signal of the texture feature and results in increased texture bias.
Revised: As the background becomes more similar to the texture cue it increases the overall signal of the texture feature and results in increased texture bias.

Revised: [Figure]

Original: 1.0 Fraction of Image Size 0.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: T e x t u r e
Revised: T e x t u r e

Original: P r e f e r e n c e
Revised: P r e f e r e n c e

Original: R=0.
Revised: R=0.

Original: Resizing Resizing
Revised: Resizing Resizing

Original: Since a texture gets mapped over the entire area of an object while shape information is containedin edges, we hypothesized that decreasing the size of the masked version of the texture-shape cueconﬂict test image would further increase shape signal relative to texture signal since the texturesignal will decrease proportional to dimension squared whereas shape will drop linearly as objectsize decreases.
Revised: Since a texture gets mapped over the entire area of an object while shape information is contained in edges, we hypothesized that decreasing the size of the masked version of the texture-shape cue conﬂict test image would further increase shape signal relative to texture signal since the texture signal will decrease proportional to dimension squared whereas shape will drop linearly as object size decreases.

Original: Indeed, there is a strong negative correlation between object size and CNN texturepreference (Figure 9).
Revised: Indeed, there is a strong negative correlation between object size and CNN texture preference (Figure 9).

Original: Using Only Full Shapes
Revised: Using Only Full Shapes

Original: In the cue conﬂict experiments of Geirhos et al.
Revised: In the cue conﬂict experiments of Geirhos et al.

Original: (2019), some of the shape features in the test setimages have incomplete shapes (e.g. the tip of a knife may be missing) whereas texture features aregenerally complete in all test images.
Revised: (2019), some of the shape features in the test set images have incomplete shapes (e.g. the tip of a knife may be missing) whereas texture features are generally complete in all test images.

Original: This should further reduce the amount of shape informationcontained in a test image, and thus result in lower shape preference.
Revised: This should further reduce the amount of shape information contained in a test image, and thus result in lower shape preference.

Original: To test this effect, we removed(manually selected) cue conﬂict test images with incomplete shapes from the cue conﬂict test set,and re-measured texture-shape preference.
Revised: To test this effect, we removed (manually selected) cue conﬂict test images with incomplete shapes from the cue conﬂict test set, and re-measured texture-shape preference.

Original: Interestingly, we saw no signiﬁcant change in texturepreference using the original style-transfer images with incomplete shapes removed, but we didobserve an increase (8.5%) in shape preference after exterior masking.
Revised: Interestingly, we saw no signiﬁcant change in texture preference using the original style-transfer images with incomplete shapes removed, but we did observe an increase (8.5%) in shape preference after exterior masking.

Original: This further illustrates theimpact of background texture signal in the original style-transfer cue conﬂict test images.
Revised: This further illustrates the impact of background texture signal in the original style-transfer cue conﬂict test images.

Original: Discussion
Revised: Discussion

Original: Informed by the results outlined in Section 3, we demonstrated increased control over test imagesin the texture-shape cue conﬂict experiments proposed by Geirhos et al.
Revised: Informed by the results outlined in Section 3, we demonstrated increased control over test images in the texture-shape cue conﬂict experiments proposed by Geirhos et al.

Original: In Hermann et al.(2019), the authors were able to shift the CNN from texture bias to shape bias by augmenting thetraining data of the ImageNet model but we were able to shift from texture bias to shape bias strictly bychanging the test images while using exactly the same ImageNet-trained model.
Revised: In Hermann et al. (2019), the authors were able to shift the CNN from texture bias to shape bias by augmenting the training data of the ImageNet model but we were able to shift from texture bias to shape bias strictly by changing the test images while using exactly the same ImageNet-trained model.

Original: We believe follow-upexperiments with full control over signal, deviation, overlap, and predictivity in the test images acrossall classes are needed to accurately quantify the level of texture bias in ImageNet-trained CNNs.
Revised: We believe follow-up experiments with full control over signal, deviation, overlap, and predictivity in the test images across all classes are needed to accurately quantify the level of texture bias in ImageNet-trained CNNs.

Original: B Revisiting Excessive Invariance 
Revised: B Revisiting Excessive Invariance B.1 Methods

Original: [Figure] B.1 Methods

Original: The procedure for this experiment largely follows experiments done in Jacobsen et al.
Revised: The procedure for this experiment largely follows experiments done in Jacobsen et al.

Original: Weconstruct three different test sets, and one training set.
Revised: We construct three different test sets, and one training set.

Original: For test sets, one is the unmodiﬁed MNISTtest set, one contains MNIST digits with location-based class-conditional pixels, and the last onlylocation-based class-conditional pixels.
Revised: For test sets, one is the unmodiﬁed MNIST test set, one contains MNIST digits with location-based class-conditional pixels, and the last only location-based class-conditional pixels.

Original: The training set is constructed by placing the location-basedclass-conditional pixel next to MNIST digits.
Revised: The training set is constructed by placing the location-based class-conditional pixel next to MNIST digits.

Original: To extract segments of the training set with a given amount of deviation, we trained a ResNet-20classiﬁer on the original MNIST dataset, and created a new training set containing images closest,within a given percentage, to the per-class mean in latent space.
Revised: To extract segments of the training set with a given amount of deviation, we trained a ResNetclassiﬁer on the original MNIST dataset, and created a new training set containing images closest, within a given percentage, to the per-class mean in latent space.

Original: We trained models using a ResNet-20 optimized using SGD with learning rate 0.1 (decayed by 0.1 atepochs 30 and 40) for 50 epochs, and with weight decay 0.0001.
Revised: We trained models using a ResNet-20 optimized using SGD with learning rate 0.1 (decayed by 0.1 at epochs 30 and 40) for 50 epochs, and with weight decay 0.0001.

Original: A random 55,000 images from thetotal 60,000 MNIST pool of training images was used for training and the rest was left for validation.
Revised: A random 55,000 images from the total 60,000 MNIST pool of training images was used for training and the rest was left for validation.

Original: Our model achieved 99.55% clean accuracy on unmodiﬁed MNIST.
Revised: Our model achieved 99.55% clean accuracy on unmodiﬁed MNIST.

Original: After training, a model is testedon all three test sets.
Revised: After training, a model is tested on all three test sets.

Original: We average all results across ﬁve training runs.
Revised: We average all results across ﬁve training runs.

Original: B.2 Results
Revised: B.2 Results

Original: Jacobsen et al.
Revised: Jacobsen et al.

Original: (2019) demonstrated in their Binary shiftMNIST experiment that by adding a singlelocation-based, class-conditional pixel to an MNIST dataset during training but removed at test time,classiﬁcation accuracy dropped from 100% to 13%–just slightly above random guessing.
Revised: (2019) demonstrated in their Binary shiftMNIST experiment that by adding a single location-based, class-conditional pixel to an MNIST dataset during training but removed at test time, classiﬁcation accuracy dropped from 100% to 13%–just slightly above random guessing.

Original: Clearly themodel had developed a strong preference for the pixel feature over the MNIST digit feature whenboth were equally predictive.
Revised: Clearly the model had developed a strong preference for the pixel feature over the MNIST digit feature when both were equally predictive.

Original: Given our results above on feature preference, we wanted to see if we could increase the preferencefor the MNIST features in the Binary shiftMNIST experiment by decreasing the deviation of the
Revised: Given our results above on feature preference, we wanted to see if we could increase the preference for the MNIST features in the Binary shiftMNIST experiment by decreasing the deviation of the

Original: MNIST features.
Revised: MNIST features.

Original: We ﬁrst separated the MNIST dataset into class segments based on the deviationfrom the mean in latent space of a ResNet-20 MNIST classiﬁer.
Revised: We ﬁrst separated the MNIST dataset into class segments based on the deviation from the mean in latent space of a ResNet-20 MNIST classiﬁer.

Original: The deviation segments ranged from1% (all class instances were within 1% of the mean in latent space and then replicated), 5%, 10%,25%, 50%, and 100% (the full MNIST dataset).
Revised: The deviation segments ranged from 1% (all class instances were within 1% of the mean in latent space and then replicated), 5%, 10%, 25%, 50%, and 100% (the full MNIST dataset).

Original: We then trained a ResNet-20 on each of these modiﬁed MNIST datasets, augmented with a singlelocation-based pixel for each class like the Binary shiftMNIST experiment.
Revised: We then trained a ResNet-20 on each of these modiﬁed MNIST datasets, augmented with a single location-based pixel for each class like the Binary shiftMNIST experiment.

Original: Results from thisexperiment, Binary shiftMNIST experiment 1, can be seen in Figure 11.
Revised: Results from this experiment, Binary shiftMNIST experiment 1, can be seen in Figure 11.

Original: We then tested this samemodel by removing the MNIST digits but leaving the location-based pixels in Binary shiftMNISTexperiment 2 (results shown in Figure 12).
Revised: We then tested this same model by removing the MNIST digits but leaving the location-based pixels in Binary shiftMNIST experiment 2 (results shown in Figure 12).

Original: In Binary shiftMNIST experiment 3 we tested the samemodel with both the location-based pixels and MNIST digits present in the test images (results shownin Figure 13).
Revised: In Binary shiftMNIST experiment 3 we tested the same model with both the location-based pixels and MNIST digits present in the test images (results shown in Figure 13).

Original: B.3 Discussion
Revised: B.3 Discussion

Original: When the full MNIST dataset was trained with the location-based pixel features in Binary shiftMNISTexperiment 1, we were able to reproduce the result from Jacobsen et al.
Revised: When the full MNIST dataset was trained with the location-based pixel features in Binary shiftMNIST experiment 1, we were able to reproduce the result from Jacobsen et al.

Original: (2019) where the modelwas unable to accurately classify the MNIST features when the location-based pixel features wereremoved from the test set.
Revised: (2019) where the model was unable to accurately classify the MNIST features when the location-based pixel features were removed from the test set.

Original: However, as we decreased the deviation in the MNIST training digits, wewere able to progressively increase the classiﬁcation accuracy for the MNIST features suggesting
Revised: However, as we decreased the deviation in the MNIST training digits, we were able to progressively increase the classiﬁcation accuracy for the MNIST features suggesting

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A cc u r a c y
Revised: A cc u r a c y

Original: R=-0.
Revised: R=-0.

Original: Full MNIST
Revised: Full MNIST

Original: The training includes a location-based, class-conditional pixel the digits deviation
Revised: The training includes a location-based, class-conditional pixel the digits deviation

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: R=
Revised: R=

Original: [Figure]
Revised: [Figure]

Original: Only
Revised: Only

Original: Pixel
Revised: Pixel

Original: [Figure]
Revised: [Figure]

Original: that the model began including the MNIST features in its feature representations.
Revised: that the model began including the MNIST features in its feature representations.

Original: When the MNISTtraining features were within 5% of mean, the model was able to classify the MNIST test set witha relatively high accuracy.
Revised: When the MNIST training features were within 5% of mean, the model was able to classify the MNIST test set with a relatively high accuracy.

Original: In Binary shiftMNIST experiment 2, we found that the classiﬁcationaccuracy decreased as the deviation in the MNIST training digits decreased showing that the modelbecame invariant to the location-based pixels once the deviation in the MNIST digits was sufﬁcientlylow.
Revised: In Binary shiftMNIST experiment 2, we found that the classiﬁcation accuracy decreased as the deviation in the MNIST training digits decreased showing that the model became invariant to the location-based pixels once the deviation in the MNIST digits was sufﬁciently low.

Original: When the model was tested with both location-based pixels and MNIST digital we found that it wasable to classify at a high accuracy for all the deviation segments.
Revised: When the model was tested with both location-based pixels and MNIST digital we found that it was able to classify at a high accuracy for all the deviation segments.

Original: We believe that this shows that themodel was learning 1) just the location-based pixels when the MNIST digits had high deviation, 2)just the MNIST digits when they had low deviation because of the much larger signal provided by the
Revised: We believe that this shows that the model was learning 1) just the location-based pixels when the MNIST digits had high deviation, 2) just the MNIST digits when they had low deviation because of the much larger signal provided by the

Original: MNIST features and 3) a subset of both the location-based pixels and MNIST digits in an entangledrepresentation for the mid level deviation segments (i.e. not able to accurately classify either featureseparately).
Revised: MNIST features and 3) a subset of both the location-based pixels and MNIST digits in an entangled representation for the mid level deviation segments (i.e. not able to accurately classify either feature separately).

Original: This shows that the model we developed in Section 3 holds predictive potential for otherresults in machine learning.
Revised: This shows that the model we developed in Section 3 holds predictive potential for other results in machine learning.