Assessing The Importance Of Colours For CNNs In Object Recognition
Abstract
Humans rely heavily on shapes as a primary cue for object recognition.
As secondary cues, colours and textures are also beneﬁcial in this regard.
Convolutional neural networks (CNNs), an imitation of biological neural networks, have been shown to exhibit conﬂicting properties.
Some studies indicate that CNNs are biased towards textures whereas, another set of studies suggests shape bias for a classiﬁcation task.
However, they do not discuss the role of colours, implying its possible humble role in the task of object recognition.
In this paper, we empirically investigate the importance of colours in object recognition for CNNs.
We are able to demonstrate that CNNs often rely heavily on colour information while making a prediction.
Our results show that the degree of dependency on colours tend to vary from one dataset to another.
Moreover, networks tend to rely more on colours if trained from scratch.
Pre-training can allow the model to be less colour dependent.
To facilitate these ﬁndings, we follow the framework often deployed in understanding role of colours in object recognition for humans.
We evaluate a model trained with congruent images (images in original colours eg.
red strawberries) on congruent, greyscale, and incongruent images (images in unnatural colours eg.
blue strawberries).
We measure and analyse network’s predictive performance (top-1 accuracy) under these different stylisations.
We utilise standard datasets of supervised image classiﬁcation and ﬁne-grained image classiﬁcation in our experiments.
Introduction
Colours play a vital role in our day to day life.
We utilise colours for visual identiﬁcation [1], search [2], gaze guidance in natural scenes [3] etc.
As an example, importance of colour can be understood whilst identifying ripe fruits in a background of foliage [1].
Initially, it was widely believed that only shapes and not colours inﬂuence object recognition in humans [4, 5].
However, many studies [6, 7] indicate that colours do assist object recognition in humans.
The ﬁndings by Tanaka and Presnell [6] show that colours facilitate recognition of high colour diagnostic objects (natural objects like fruits) but have little effect on low colour diagnostic objects (man-made objects like airplanes).
Their experiments were based on a variation of Stroop effect [8].
Human participants were asked to name objects in different colour schemes.
They observed that naming of objects with congruent colours was much faster than naming incongruently coloured objects.
Greyscale images served as a neutral medium and the response time for them were in between congruent and incongruent images.
In a similar study conducted by Hagen et al.
[9] for investigating the role of colour in expert object recognition, similar ﬁndings were reported.
Neural networks are models of machine learning designed to mimic the neurological working of a human brain [12].
Today, they are employed in many different ﬁelds solving numerous tasks [13–15].
Considering the widespread adoption and blackbox nature of neural networks, considerable studies have also been performed to understand their inner working
[Figure]
Zeiler and Fergus [16] illustrated the hierarchical nature of learnt features.
Engilberge et al.
[20] investigated the colour sensitivity of neural networks and observed that at the shallow end the neurons are more sensitive to colour information in an image.
A number of existing studies highlight the nature of representations learnt by the network however, do so with conﬂicting results.
One set of results shows that neural networks rely predominantly on shapes [21–25].
On the other hand, many more approaches oppose the theory of shape bias in CNNs [26, 10, 27–29].
Rather than primarily relying on shapes, neural network’s predictions are guided by the texture information of an image.
Texture is referred to as “a function of the spatial variation in pixel intensities” [30, 31].
Gatys et al.
[32] showed that a linear classiﬁer based on texture representations of a neural network performs comparable to the original model.
Similarly, Geirhos et al.
[11] demonstrated that ImageNet [33] trained model is biased towards texture.
The objective of our paper is to help bridging the gap between human perception and artiﬁcial intelligence, providing empirical experiments based on classical neuroscience framework to exhaustively investigate the dependency of CNNs on colour information.
We believe the majority of existing approaches do highlight representation bias of CNNs but fail to address the role of colours.
Bahng et al.
[34] tend to the issue of colour bias in their experiments, however, do so with the motivation to learn unbiased representations.
Moreover, by either focusing on small number of test images [21, 29] or a single dataset [11, 26] we are unable to observe a bigger picture.
We believe a fair approach of highlighting the importance of colour is by utilising the framework as used by the psychophysical experiments of [1, 6, 9, 35].
We can easily observe the relevance of colours by comparing performances on congruent and greyscale images.
Additionally, by comparing the performances on greyscale and incongruent images we will be able to observe the effect of incorrect colour information (see Figure 4 for sample images).
In this paper, we evaluate the importance of colours for numerous datasets under 2 settings:
Local Information (Section 4.1): Evaluating the importance of colours when a CNN can only tend to small patches in a global shape agnostic manner.
Global Information (Section 4.2):
For this setting, no such restriction is applied on the network and corresponds to standard approach of training a CNN.
Apart from these 2 modes of experimentation, we also evaluate a model under 4 different training schemes to replicate a typical training scenario for a classiﬁcation task.
We train a network (i) from scratch, (ii) with ﬁne-tuning on pre-trained weights, (iii) with colour based augmentations (random hue, saturation, brightness, etc.), and (iv) with incongruent images to reduce colour dependency.
We conclude in Section 5.
Data
In our study we have employed datasets from image classiﬁcation and ﬁne-grained visual classiﬁcation.
The datasets used are: • CIFAR-100 [36] • STL-10 [37] • Tiny ImageNet [38] • CUB-200- [39] • Oxford-Flowers [40] • Oxford-IIIT Pets [41].
Table 5 in appendix provides some common statistics of the datasets used.
Method
A dataset D = { ( x i , y i ) , i = 1 , . . .
, N } is composed of images x i ∈ R C
× H × W and their corresponding labels y i .
D train , D test denotes the split of the dataset into train and test sets respectively.
We convert D into different colour schemes as described below:
• Congruent Images ( D C ):
These are the images in original colours.
All the subsequent transformations described
below
are applied to D C .
• Greyscale Images ( D G ): The congruent images converted into greyscale(luminance) images, G = 0 .
29 ∗ r + 0 .
587 ∗ g + 0 .
114 ∗ b .
We copy the single channel greyscale image into 3 channels.
• Incongruent Images ( D I ): The channels of the congruent images are switched to generate unnaturally coloured images.
Formally, the default correspondence for colour channels is as C [0] = red, C [1] = green, C [2] = blue .
We switch the channels such that the new ordering represents C [0] = green, C [1] = blue, C [2] = red .
The advantage of it is that ﬁrstly, it preserves the texture and secondly, the changes in distribution are more contained than approaches deployed by [10, 11].
For example, the Jensen-Shannon divergence [42] JS ( D trainC , D trainG ) = 0 .
06 and JS ( D trainC , D trainI )
= 0 .
1 , whereas same for negative images (as in [10]) and D train C is 0 .
3 for STL-10 dataset.
More comparison is provided in appendix A.2.
Figure 4 shows an example of these stylisations.
As humans, we learn from our surroundings which we perceive in congruent colours.
Since we are following the framework used in identifying role of colours in humans, we train CNNs only on congruent images ( D trainC ) while evaluating the topaccuracy (represented as Acc ) on the test sets of different stylisations described above.
Experiments 4.1 Access to local information
Baker et al.
[29] report that CNNs can represent local shapes however, fail to utilise it in a global context.
Similarly, to highlight absence of shape bias in a CNN, Geirhos et al.
[11] utilise BagNets[28] to compare model performance under artistic stylisations.
BagNets only have access to small patches within an image due to its design and utilises bag-of-features based framework for making a prediction.
It does not make use of spatial ordering of the local features hence making it suitable to compare the relevance of colours to local shape and texture information.
We use BagNet-9 1 which has a 9 × 9 receptive ﬁeld over the image and is built upon the Resnetarchitecture.
We report the mean accuracy and standard deviation over 3 runs.
The network is trained from scratch and the data augmentations are limited to random horizontal ﬂips, random rotation and random cropping.
The details on hyper-parameters to train the network are provided in the supplementary document.
4.1.1 Results
Figure 2 lists the relative accuracies w.r.t D C for different stylisations( D G D C and D I D C ).
When comparing
Acc( D C ) with corresponding D I and D G , we observe signiﬁcant drop in performance.
We can also see the varying nature of the gaps in performance across the datasets.
This shows that the colours do inﬂuence a network but in varying degree.
Moreover, for STL-10 and Oxofrd-IIIT Pets, if we compare Acc( D C ) to Acc( D G ) we observe that the drop in accuracy is there but comparatively less than the other datasets.
This suggests that the network also relies on non-colour features (such as local shapes, textures) to make a decision.
Additionally, by comparing Acc( D G ) and Acc( D I ) we can notice the consistently lower performance for the latter.
This suggests that incorrect colour information does indeed harm the prediction accuracy.
These results indicate that even though a CNN can represent local shape [29] or is biased towards texture[11], colours plays an important part at a local setting.
4.2 Access to global information
In the previous experiment, we limited the view of the network to only attend to 9 × 9 patches of the image in a global shape agnostic fashion.
Here, we use ResNet-18 which has no such restriction on its receptive ﬁeld.
It can thus tend to global shape information in the image.
To assess the importance of colours we train a network in the following 4 ways:
Vanilla training : Similar to the setting in Section 4.1,
we train the network from
scratch.
The data augmentations used are random rotation, random cropping and random horizontal ﬂips.
Fine-tuning :
In practice, often an ImageNet [33] pre-trained network is ﬁne tuned on the dataset at hand [43].
In this experiment, we follow this protocol keeping the augmentation strategy identical to vanilla training.
Fine-tuning with Augmentations : • Colour augmentation :
Often colour based augmentations are used in training [44, 45] allowing the network to be colour invariant.
All the training settings are identical to ﬁne-tune except for the fact that we also add random colour jitter (hue, saturation, contrast, and brightness) to an image while training.
• Channel switch pre-training :
Many of the previous works studying the shape and texture bias have imposed learning limitations by ﬁrst training the network on stylised data [10, 11] and then ﬁne-tuning on original images.
This has been shown to improve predictive performance of the model.
Following this approach, we ﬁrst start with an ImageNet pre-trained model (on congruent images).
Then we ﬁne-tune the network following ‘colour augmentation’ protocol with randomised channel switching as an additional data augmentation method.
We do this in order to emulate stylised pretraining[11].
After ﬁne-tuning the model, we disable the colour augmentations (‘colour jitter’ and ‘randomised channel switching’) and ﬁne-tune the network further only on congruent images.
All the hyper-parameter details are provided in the appendix (see Appendix A.4).
We also provide vanilla training results for MobileNet-v2 and DenseNet-121 alongside ResNet-18 and BagNet-9 (see Appendix 4.3).
4.2.1 Results & Observations
The results are shown in ﬁgure 3.
Detailed results are available in the appendix (see appendix A.3).
We can draw the following observations from the results.
For vanilla and ﬁne-tuned networks, Acc ( D testC ) >
Acc ( D testG ) >
Acc ( D testI ) .
This trend is similar to what existing studies report for object recognition by humans[9, 35].
However, the difference in CNN accuracies across stylisations are signiﬁcantly larger.
Apart from scoring human participants solely based on accuracy, their response time is also taken into account.
For a CNN, there is no variation in the inference time as the architecture remains constant.
However, it can be an interesting extension to understand the differences arising over the predicted estimate.
For instance, many approaches utilise predicted value for the winning category as a network’s conﬁdence in its prediction [46].
The aim of the study will be then to observe the potential impact of colours on its conﬁdence estimate.
Fine-tuning a pre-trained model is widely known to improve the learnt representations of a model and subsequently its accuracy.
We observe the additional beneﬁt of ﬁne- tuning which leads to better performance for greyscale and incongruent images indicating lower dependency on colours.
Incorporating colour augmentations and channel-switching into training can enforce a model to further rely less on colours.
But, it does not improve the network’s accuracy for congruent images.
The variability for cross-style performance is high across datasets.
For example, in the vanilla training setting CUB-200 shows a signiﬁcantly low performance for greyscale when compared to Oxford-IIIT Pets.
We make a similar observation when comparing STLwith CIFAR-100.
One common property of STL-10 and Pets is that they consist of relatively smaller number of classes ( 10 and 37 respectively) when compared to CIFAR-100 and
CUB-200 ( 100 and 200 respectively).
A direction for future work can be to investigate the relationship between number of categories in the dataset and colour dependency of a
CNN.
Apart from exploring the dependency over the number of categories we can also investigate if this variation is dependent on categories in the dataset.
As humans, we rely more on colours for recognising natural objects than man-made objects [6].
This property is referred to as colour diagnosticity.
A similar observation if it exists for CNNs can be worth exploring.
[Figure]
4.3 Vanilla Performance Across Architectures
In this experiment we include MobileNet-v2 [47] and DenseNet-121 [48] along with ResNetand BagNet-9 to compare their performance across different datasets.
This way we can examine if architectural differences play a role in colour bias.
We report the results on different CNN architectures trained under the vanilla setting in Figure 4.
The results show that different architectures display similar behaviour for colour importance across datasets.
On the congruent images, the networks perform the best whereas the performance is worst for incongruent images.
This shows that the underlying the architecture plays a less signiﬁcant role in driving the bias of a network towards colour.
The importance of colour is more dependent on the task at hand.
Conclusion
We believe ours is the ﬁrst work to recognise unattributed impact of colours to the shape/texture driven research for understanding bias in CNNs.
By adopting the psychophysical experiment for
CNNs, we have provided empirical evidence to highlight high impact of colours.
We showed that a variety of different CNNs show high colour dependency for the classiﬁcation task.
This dependency appears to be tied to the dataset than the underlying architecture.
By default, the networks are highly colour dependent and this dependency can be reduced by utilizing pre-trained weights and employing various augmentations in training as showed in our work.
A Appendix A.1 Data
[Table]
A.2 Jensen-Shannon Measure
To reiterate the notations used, a dataset D = { ( x i , y i ) , i = 1 , . .
, N } is composed of images x i ∈ R C
× H × W and their corresponding labels y i .
D train , D test denotes the split of the dataset into train and test sets respectively.
Jensen Shannon divergence between two stylisations of the same dataset is deﬁned as:
[Equation]
where, T i is the transformation corresponding to D i and indexing [ j ] returns the normalised intensity histogram for the j th channel.
The corresponding results in Table 2 show that switching channels is a gentler transformation than composing negatives in preserving original shape and texture.
Greyscale is consistently with the least amount of JS divergence and can suggest as to why consistently Acc ( D G ) >
Acc ( D I ) .
[Table]
[Equation]
[Table]
A.3 Detailed Test results A.4 Training details
We utilised Pytorch framework for all of our experiments.
We list the detailed hyper-parameters below.
Missing key-values in table i can be found via.
recursive search in table i − 1 .
All the ﬁne-grained datasets utilise similar training hyper-parameters and hence we have provided only the details for CUB-200.
[Table] [Table]
[Table] [Table]
[Table] [Table]