P LANCKIAN JITTER : ENHANCING THE COLOR QUALITY OF SELF - SUPERVISED VISUAL REPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation.
The set of used data augmentations is of crucial importance for the quality of the learned feature representation.
We analyze how the traditionally used color jitter negatively impacts the quality of the color features in the learned feature representation.
To address this problem, we replace this module with physics-based color augmentation, called Planckian jitter, which creates realistic variations in chromaticity, producing a model robust to llumination changes that can be commonly observed in real life, while maintaining the ability to discriminate the image content based on color information.
We further improve the performance by introducing a latent space combination of color-sensitive and non-color-sensitive features.
These are found to be complementary and the combination leads to large absolute performance gains over the default data augmentation on color classiﬁcation tasks, including on Flowers-102 (+15%), Cub200 (+11%), VegFru (+15%), and T1K+ (+12%).
Finally, we present a color sensitivity analysis to document the impact of different training methods on the model neurons and we show that the performance of the learned features is robust with respect to illuminant variations.
I NTRODUCTION
Self-supervised learning enables the learning of visual representation without the need for any labeled data (Doersch et al., 2015; Dosovitskiy et al., 2014).
Several recent works learn representations that are invariant with respect to a set of data augmentations, and have obtained spectacular results (Grill et al., 2020; Chen & He, 2021; Caron et al., 2020), signiﬁcantly narrowing the gap with supervised learned representations.
These works vary in their architecture, learning objective, and optimization strategy, however, they are similar in applying a common set of data augmentations to generate the various image views.
The algorithms, while learning to map these different views to the same latent representation, learn complex semantic representation for visual data.
The set of transformations (data augmentation) that are considered induce a set of invariances that characterizes the learned visual representation.
Before deep learning revolutionized the way visual representations were computed, separate features were hand-designed to represent its various properties, leading to research on shape (Lowe, 2004), texture (Manjunath & Ma, 1996), and color features (Finlayson & Schaefer, 2001; Geusebroek et al., 2001).
Color features were typically designed to be invariant with respect to a set of scene accidental events, such as shadows, shading, illuminant and viewpoint changes.
With the rise of deep learning, feature representations that exploit simultaneously color, shape and texture are learned implicitly and the invariances are a byproduct of the end-to-end training (Krizhevsky et al., 2009).
As discussed above, the current set of self-supervised learning methods explicitly deﬁne a set of invariances (related to the applied data augmentations) that are to be learned.
In this work, we focus on the current de-facto choice for color augmentations.
We argue that they seriously cripple the color quality of these representations and we propose an alternative color augmentation.
Figure 1 (left) illustrates the currently applied color transformation for a sample image, depicted in the middle of the left-most grid.
It is clear that the applied color transformation signiﬁcantly alters the colors of the original image, both in terms of hue and saturation.
One of the justiﬁcations in literature for such strong color augmentations is that without large color changes, mapping images to the same latent representation can be purely done based on color and no complex shape features are learned, therefore the best results when using only two transformations are obtained when applying image cropping with a color augmentation (Chen et al., 2020a).
However, looking at the reported example it is evident that a representation that maps these images to the same latent representation cannot rely on the object color and as a results the quality of the color representation learned with such algorithms is expected to be inferior.
Therefore, in this paper we propose another set of color augmentations, shown on the right side of
Figure 1.
In addition to introducing more natural variations in the image chromaticity, the proposed color augmentation also affects neutral regions such as the petals shown in the ﬁgure, which are left unvaried with the original color transformations.
We draw on existing color imaging literature, that aimed to design features that were invariant with respect to illuminant changes that were commonly encountered in real-world scenes (Finlayson & Schaefer, 2001).
Our augmentation, called Planckian jitter applies a physically realistic illuminant variation to the images.
We consider the illuminants that are described by Planck’s Law for black body radiation and that are known to be similar to illuminants encountered in real-life (Tominaga et al., 1999).
In the experimental section, we show that self-supervised features learned with Planckian jitter yield superior features, leading to gains of over 5% on several downstream color classiﬁcation tasks.
However, since our color augmentation is less extreme, the learned shape features are of lower quality than with the original color jitter.
A simple combination of both feature representations leads to huge performance gains with respect to default color jitter of between 10-15% on several color downstream tasks.
In addition, we show that our augmentation method can be applied to several state-of-the-art self-supervised learning methods.
Finally, we analyze the color sensitivity of the learned color representations.
B ACKGROUND AND RELATED WORKS 2. S ELF - SUPERVISED LEARNING AND CONTRASTIVE LEARNING
Recent improvements in self-supervision learn a semantically rich feature representation without the need of any labels.
In SimCLR (Chen et al., 2020a) similar samples are created by augmenting an input image, while dissimilar are chosen by random.
To make contrastive training more efﬁcient
MoCo method (He et al., 2020) and its improved version (Chen et al., 2020b) use a memory bank for learned embeddings which helps for efﬁcient sampling.
This memory is kept in sync with the rest of the network during training time by using a momentum encoder.
Several methods do not have any explicit contrastive pairs.
BYOL (Grill et al., 2020) propose an asymmetric network by introducing an additional MLP predictor between the outputs of the two branches.
One of the branches is kept ”ofﬂine” - updated by a momentum encoder.
SimSiam (Chen & He, 2021) goes even further and presents a simpliﬁed solution without a momentum encoder.
Moreover, it obtains similar high-quality results and does not require a large mini-batch size.
We will use the SimSiam method to verify our proposed color augmentation (we also apply our approach to SimCLR (Chen et al., 2020a) and Barlow Twins (Zbontar et al., 2021) in the experiments).
The main part is a CNN-based encoder, learned end-to-end in an asymmetric siamese architecture,
[Figure]
t
[Figure] r ad [Figure] xzzpz where one branch has an additional predictor (Multi-Layer Perceptron, or MLP) whose output aims to be as close as possible to the other branch (see Figure 2).
The second branch is not updated during backward propagation.
A negative-cosine loss function is used, deﬁned as:
[Equation]
where z 1 , z 2 are encoded values respectively for two different augmented versions x 1 and x 2 of the same image x . Note that in Eq. 1 they are alternated between the two branches, but it is always only the ﬁrst branch that uses a Multilayer Perceptron, producing either p 1 or p 2 .
Additionally, no contrastive term is present: only the similarity is enforced during learning.
2. D ATA AUGMENTATION
Data augmentation plays an important part in the learning process of self-supervised methods.
In the works by Chen et al.
(2020a) and Zbontar et al.
(2021), authors discussed the importance of the different data augmentations.
A set of well-deﬁned transformations was proposed within the SimCLR method.
This set is commonly used in later works in the self-supervision ﬁeld.
The augmentations include: rotation, cut, ﬂip, color jitter, blur and gray scale.
These operations are randomly applied to an image to generate different views ( x 1 , x 2 ).
Applied to the same image contrastive-like selfsupervised methods learn representation invariant for such distortions.
The multiple view creation is a task-related procedure (Tian et al., 2020).
However, keeping in mind usefulness of learned representations for downstream tasks, color jittering is one of the most important ones (Chen et al., 2020a;
Zbontar et al., 2021), operating on hue, saturation, brightness and contrast.
However, color jitter is expected to induce a certain level of color-invariance (invariance to hue, saturation, brightnesss and contrast) which are consequently transferred to the downstream task as well.
As a consequence, we expect these learned features to underperform on downstream tasks for which color is crucial.
Color imaging literature has long researched color features that were invariant with respect to scene accidental events, such as shading, shadows, and illuminant changes (Geusebroek et al., 2001;
Finlayson & Schaefer, 2001).
Having features with invariance with respect to these events was found to be beneﬁcial for object recognition.
Having invariance with respect to hue and saturation changes (which are induced by the currently used color jitter operation) is detrimental for object recognition, especially for those objects were these characteristics are fundamental.
Therefore, in this work, we aim to revisit early theory on illuminant invariance (Finlayson & Schaefer, 2001) to design an improved color augmentation function that induces invariances common in the real-world and that does not deteriorate the color quality of the learned features.
P LANCKIAN J ITTER
The range of image transformations introduced by traditional color jitter creates a variability in training data that indiscriminately explores all hues at various levels of saturation.
The resulting invariance can be useful for downstream tasks where chromatic variations are indeed irrelevant (such as car body color in vehicle recognition), but will be detrimental for downstream applications where the color information is known to be critical (such as ﬂowers, birds, vegetables classiﬁcation).
On the other hand, completely removing color invariance risks producing a model with little generalization capability, unable to handle the common variations in illumination conditions due to various sources of indoor and outdoor lighting.
3. P HYSICS - BASED P LANCKIAN JITTER
Here we describe an alternative data augmentation procedure, called Planckian jitter , that exploits the physical description of a black-body radiator to re-illuminate the training images within a realistic distribution (Finlayson & Schaefer, 2001; Tominaga et al., 1999).
The resulting augmentations are more realistic than those of the default color jitter (see Fig. 1).
The resulting learned selfsupervised feature representation is thus expected to be robust to illumination changes that can be commonly observed in real-world images, while simultaneously maintaining the ability to discriminate the image content based on color information.
Given an input RGB training image I , our physics-based Planckian jitter procedure applies a chromatic adaptation transform that simulates realistic variations in the illumination conditions:
We sample a new illuminant spectrum σ T ( λ ) from the distribution of a black body radiator.
We transform the sampled spectrum σ T ( λ ) into its sRGB representation ρ T ∈ R 3 . 3.
We create a jittered image I (cid:48) by reilluminating I with the sampled illuminant ρ T . 4.
We introduce brightness and contrast variation, producing a Planckian-jittered image
I (cid:48)(cid:48) .
A radiating black body at temperature T can be synthesized using Planck’s Law (Andrews, 2010): [Equation]
[Equation]
) where c = 2 . [-
-]99792458 × 10 8 m/s is the speed of light, h = 6 .
× 10 − 34 Js is Planck’s constant, and k = 1 .
380662 × 10 − 23 J/K is Boltzmann’s constant.
For our experiments, we sampled T in the interval between 3000 K and 15000 K which is known to result in a set of illuminants that can be encountered in real life (Tominaga et al., 1999).
Then, we discretized wavelength λ in 10nm steps ( ∆ λ ) in the interval between 400nm and 700nm.
The resulting spectra are visualized in Figure(left) in Appendix A.1.
The conversion from spectrum into sRGB is obtained through a series of intermediate steps (Wyszecki & Stiles, 1982):
We ﬁrst map the spectrum into the corresponding XYZ stimuli, using the 1931 CIE standard observer color matching functions c { X,Y,Z } ( λ ) , in order to bring the illuminant into a standard color space that represents a person with average eyesight.
We normalize this tristimulus by its Y component, convert it into CIE 1976 L*a*b color space, and ﬁx its L component to 50 in a 0-to-100 scale.
This allows us to constrain the intensity of the represented illuminant in a controlled manner as a separate task.
We then convert the resulting values in sRGB, obtaining ρ T
=
{ R, G, B } .
The resulting distribution of illuminants is visualized with the Angle-Retaining Chromaticity diagram (Buzzelli et al., 2020) in Figure 4 (right) in Appendix A.1.
All color space conversions assume a D65 reference white, which means that a neutral surface illuminated by average daylight conditions would appear achromatic.
Once the new illuminant has been converted in sRGB, it is applied to the input image I by resorting to a Von-Kries-like transform (von Kries, 1902), given by the following channel-wise scalar multiplication:
[Equation] where we assume the original scene illuminant to be white (1,1,1).
Finally, brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination:
[Equation]
where c B = 0 .
8 and c C = 0 .
8 represent, respectively, brightness and contrast coefﬁcients, and µ is a spatial average function.
3. L ATENT S PACE C OMBINATION
The self-supervised learning paradigm involves a pretraining phase that relies on data augmentation to produce a set of features with certain invariance properties.
These features are then used as the representation for a second phase, where we learn a given supervised downstream task.
Default color jitter conﬁgurations generate features that are strongly invariant to color information, thus relying mainly on shape and texture for the downstream task.
Constraining the color jitter consequently reduces the role of shape in favor of color in the decision process of the second phase.
As stated by (Chen et al., 2020a), in fact, completely removing color jitter creates features that focus mainly on the color, and ignore the more difﬁcult problem of learning a good shape representation.
Our Planckian jitter is designed to operate in a middle ground, constraining the type of color transformation that should be given invariance to, and thus leaving space for shape information.
Nonetheless, the relative role of color and non-color in the feature representation can be better tuned by creating a concatenation of features learned in different setups.
For this reason, we propose a conﬁguration where two pretraining phases are run in parallel: one with default color augmentation, which is expected to exploit mostly shape information, and one with Planckian jitter, which is expected to exploit color information.
The two resulting features are then concatenated and used as the representation to learn downstream tasks (in our experiments each representation hasdimensions and the latent space combination 1024).
E XPERIMENTAL RESULTS
In this section we verify the superiority of the proposed color data augmentation method against the default color jitter and we ablate its components.
We evaluate the impact on downstream classiﬁcation tasks, and analyze the color sensitivity of the learned backbone networks.
We have additional results on computational time of the proposed Planckian augmentation in Appendix A.
4. T RAINING AND EVALUATION SETUP
We train a {+ResNet-
+} (He et al., 2016) backbone encoder in the SimSiam framework and then evaluate it on a classiﬁcation downstream task with a linear classiﬁer.
To analyze the impact of the augmentation operations on the learned features, we train the encoder with different data augmentations before the evaluation.
See Figure 2 for a complete overview of the training procedure.
We perform the unsupervised training with two different image resolutions: 32 × 32 and 64 ×pixels 1 .
To train the model at the lowest resolution, we adopted the C IFAR -100 dataset (Krizhevsky et al., 2009), which is composed of 50000 32 × 32 images, divided into 100 classes.
Since the training has been done with images of size 32 × 32 , the ResNet18 architecture has been slightly modiﬁed: the ﬁrst convolutional layer’s kernel size has been modiﬁed from 7 × 7 to 3 × 3 and the ﬁrst Max
Pooling layer has been removed.
Adopting the same modiﬁcation to the architecture, we also trained a higher resolution version of the model, using the T INY -I MAGE N ET dataset (Le & Yang, 2015), which contains 100,000 images of 200 classes (500 for each class) at resolution 64 × 64 pixels.
The linear classiﬁer training at resolution 32 × 32 has been performed with two different datasets: the C IFAR -100 and the F LOWERS -102 datasets (Nilsback & Zisserman, 2008).
The C IFARhas been used in order to deﬁne a baseline for the classiﬁcation task in generic conditions, since it has been used also in the original work on SimCLR.
The F LOWERS -102 dataset, which contains 8189 photos of ﬂowers, divided in 102 classes, has been selected to assess the quality of the features extracted in scenarios where color information covers an important role.
The F LOWERS -102 dataset has been resized to dimension 32 × 32 to ﬁt the pre-trained model input dimensions.
For the training of the linear classiﬁer at resolution 64 × 64 , we used ﬁve different datasets: T INY -I MAGE N ET ,
F LOWERS -102 dataset, the V EG F RU dataset by Saihui Hou & Wang (2017), the CUB-200 dataset by Welinder et al.
and the T1K+ dataset by Cusano et al.
More details about these datasets are provided in Appendix A.2.
In the case of CUB-200, each image has been cropped using the bounding boxes given in the annotations of the dataset.
These ﬁve datasets have been resized at resolution 64 × 64 and used to train the linear classiﬁer.
For the T1K+, dataset we adopted theclass labeling to train the linear classiﬁer and to test our models.
The model for supervised and unsupervised training has been written in Pytorch v1.7.0 and run on an
NVIDIA Titan V 12 GB GPU.
The SimSiam training has been performed using Stochastic Gradient
[Table]
Descent, with a starting learning rate 0 .
03 , cosine annealing learning rate scheduler, and mini-batch size 512 (like in original work Chen & He (2021)).
The linear classiﬁer training for C IFAR -100 is done with exactly the same settings, except a staring learning rate set to 0 .
1 and a train process to 500 epoch.
For other datasets we used Adam optimizer with initial learning rate set to 0 .
4. E XPERIMENTAL RESULTS - P LANCKIAN J ITTER A UGMENTATION 4.2. A BLATION S TUDY
To assess the impact of color data augmentations we deﬁne six different conﬁgurations:
None : no color jittering operations (Random Color Jitter and Random Grayscale) are used.
• Default Color Jitter: the default conﬁguration, as used in SimSiam and SimCLR, uses both Random Color Jitter and Random Grayscale operations.
• Default Color Jitter w/o Random GrayScale : same as Default without the Random Grayscale operation.
• Planckian Jitter : uses the complete proposed Planckian Jitter operation, operating on chromaticy and brightness and contrast aspects of the images.
No Random Grayscale is applied.
• Planckian Jitter (chromaticity only) : applies Planckian jitter without modifying brightness and contrast values in the image.
• Planckian Jitter (brightness/contrast only) : this conﬁguration only modiﬁes images brightness and contrast values of the input images.
In all experiments these augmentations are combined with the other default augmentations, namely crop, horizontal and vertical ﬂip and blur.
Given these six conﬁgurations, six different models have been trained and have been ﬁne tuned for image classiﬁcation.
For resolution 32 × 32 the model has been ﬁne tuned with the datasets C IFAR -100 and F LOWERS -102.
The results in terms of accuracy are reported in Table 1.
Looking at the results, we can identify basically two different trends for the two datasets.
Considering C IFAR -100, as can be observed, removing the augmentations related to the colors makes the model less powerful, due to the loss of color invariance in the features extracted by the encoder, trained with contrastive learning approach.
This behaviour is coherent to what was reported by Chen et al.
(2020a).
As can be seen in Table 1, if color augmentations (i.e. brightness/contrast and Random GrayScale) are removed completely ( None conﬁguration), the accuracy drops of 18% .
Furthermore, on the F LOWERS -102 the behaviour is the opposite with respect to C IFAR -100: removing the color augmentations helps the model to better classify images, obtaining an improvement of 6 , 93% of accuracy.
This behaviour conﬁrms how the color invariance, learned in contrastive learning environment, negatively impacts downstream tasks where the color information has a more relevant role.
For these scenarios a solution that takes into consideration the color information is preferred.
Considering the introduction of realistic color transformations, we can observe how this kind of augmentation operations positively impacts the contrastive training, producing models that achieve even better results with respect to the conﬁguration without any kind of image color manipulation.
As can be seen from Table 1, by simply reducing the jittering operation to inﬂuence brightness and contrast, leaving hue and saturation unchanged, yields another boost in accuracy of 5 .
When we start modifying chromaticity values using a realistic approach, moving to Planckian Jitter augmentation, the ﬁnal result is a boost of 6 .
28% in accuracy with respect to the None conﬁguration, for a total boost of 12 .
75% in accuracy with respect to the Default conﬁguration.
Also on C IFAR100 dataset an improvement of 5 .
38% is given by the use of Planckian Jitter with respect to the conﬁguration that does not use any color augmentation operation.
Despite this improvement, in this scenario the contrastive training with the realistic augmentation does not give better results with respect to the Default conﬁguration.
Given the results obtained using the data augmentations reported in Table 1, and given the considerations made in Section 3.2, we ﬁnally evaluate the impact of combining latent spaces from different backbones.
The combination analyzed is obtained by concatenating the features extracted by the models trained with different conﬁgurations (e.g. Default with Planckian Jitter ). We tested three different Latent Space Combinations (LSC): the None conﬁguration with the Default Color Jitter w/o Random GrayScale, the Default conﬁguration with the Default Color Jitter w/o Random GrayScale and the Default Color Jitter with the proposed Planckian Jitter.
The results have been reported in Table 2.
We can see the beneﬁt coming from the use of a realistic augmentation instead of using the features coming from a non realistic ones. On both datasets, the Latent space combination with Default and Planckian Jitter conﬁguration achieves the best results.
On the original task, represented by the C IFAR -100 dataset, this combination achieves a total accuracy of 63 .
54% , a 3 . 61% more with respect to the Default conﬁguration and a 16 .
23% more with respect to the
Planckian Jitter conﬁguration.
Comparing to the LSC using the Default ColorJitter w/o Random
Grayscale, the version with the proposed Planckian Jitter achieves a small improvement of 1 . 27% in classiﬁcation accuracy.
On the downstream task, represented by the F LOWERS -102 dataset, the Latent space combination reaches an accuracy value of 51 .
66% : an improvement of 21 .
66% and 8 .
91% in accuracy respectively for the two original conﬁgurations.
Comparing here with the LSC using the Default ColorJitter w/o Random Grayscale, the combination with Planckian Jitter achieve an higher result, with a bigger gap in terms of accuracy with respect to the C IFAR -100 scenario.
Here the use of Planckian Jitter brings in an improvement of 4 .
01% , conﬁrming the impact of using realistic augmentation in color related classiﬁcation tasks.
In both of the cases the LSC using the
None conﬁguration with the Default Color Jitter w/o Random GrayScale achieve the worst results out of the three LSC versions.
4.2. E VALUATION ON DOWNSTREAM TASKS
Given the results obtained from the ablation study, we performed the analysis of the proposed conﬁgurations on other downstream task, using the backbone trained on higher resolution images ( 64 ×pixels).
We reported here a subset of the previously tested conﬁgurations: None , Default Color Jitter , Default Color Jitter without Random GrayScale , Planckian Jitter (brightness/contrast only) and Planckian Jitter . We also reported the Latent space combination of the Default Color Jitter combined with the proposed Planckian Jitter.
Since the backbone for resolution 64 × 64 has been trained on T INY -I MAGE N ET dataset, we ﬁrst ﬁne-tuned and tested the behaviour of the different models on this dataset, then on the other ones representing the downstream tasks.
As can be seen from the results in Table 3, the behaviour of the three models on the T INY -I MAGE N ET is the same we have seen on the C IFAR -100.
This is expected since the tasks represented by these two datasets are similar in terms of relation between the color information and class discrimination.
Looking at the other datsets, we can see that the introduction of the Planckian Jitter augmentation improves the accuracy on the test sets, and an even higher accuracy can be achieved by applying Latent space combination . FRAMEWORK DATA
[Table] [Table]
These experiments on different downstream tasks related to color and at higher resolution prove the effectiveness and the generalization capability of the proposed data augmentation and latent spaces combination.
On the different tasks, the proposed data augmentation offers a reliable way to train a backbone feature extractor more effective with respect to the original set of augmentations.
Moreover, with the Latent space combination we showed how the combination of the different feature spaces helps in the training of a more general model for image classiﬁcation, since we obtain improvements not only on color related datasets, but also on the original general content datasets.
4.2. G ENERALITY OF P LANCKIAN J ITTER
To show that the method is generally applicable to self-supervised methods which exploit color jitter augmentation, we performed experiments on two other popular self-supervised models: the
SimCLR and the Barlow Twins models.
This comparison is reported in Table 2.
As can be seen, for all of the three models tested, the application of the Planckian Jitter improves the accuracy on the downstream task.
Independently from the model used, the Default Color Jitter conﬁguration of data augmentation gives the worst results on the F LOWERS -102 dataset.
This difference from the Planckian Jitter conﬁguration is mainly related to the model sensitivity to color information: since the default conﬁguration makes the different models invariant to color information, the classiﬁcation on the downstream color-related task suffer, independently from the contrastive learning model adopted.
An analysis on the color sensitivity of the different models is reported in section 4.3.
The Latent space combination (Default Color Jitter with Planckian Jitter) conﬁguration consistently achieves better results on both of the datasets.
On F LOWERS -102 the ﬁnal results improve considerably by over 20% for all three models.
4. C OLOR SENSITIVITY ANALYSIS
In order to better understand the ﬁnal models obtained by training the backbone architectures in the contrastive learning framework, we perform a robustness analysis on the F LOWERS -102 dataset with realistic illuminant variations and analyze the models sensitivity to color information.
Given the different backbones trained on C IFAR -100, we train the linear classiﬁer on C IFARand F LOWERS -102 datasets, and test the models with different versions of these two datasets.
Assuming as reference point the D65 illuminant, which for the purpose of this test is considered the default illuminant condition of each image in the datasets used for training and ﬁne tuning,
we
D e f a u l t C o l o r J i tt e r
D e f a u l t C o l o r J i tt e r w / o R a n d o m G r a y S c a l e
P l a n c k i a n J i tt e r ( b r i g h t n e ss / c o n t r a s t o n l y )
P l a n c k i a n J i tt e r ( c h r o m a t i c i t y o n l y )
P l a n c k i a n J i tt e r
(-inf, 0) [0, 0.1) [0.1, 0.2) [0.2, 0.3) [0.2, 0.4) [0.4, 0.5) [0.5, 0.6) [0.6, 0.7) [0.7, 0.8) [0.8, 1)
# o f n e u r o n s change the illuminant of the two considered datasets.
Using the 25 illuminants derived from the
Planckian Jitter, we create different versions of each dataset, one for every illuminant.
We are now interested in evaluating the robustness of the learned representations with respect to these illuminant changes.
The results on Flowers-102 can be seen in Figure 3.
Another plot containing also the accuracies on C IFAR -100 is provided in Figure 7 from Appendix A.3.
The Planckian Jitter obtains stable performance from around 4000-14000K.
On the other hand, the Default Color Jitter is more sensitive to the illumination color and the classiﬁcation accuracy decreases when the scene illuminant is moving away from the white illuminant.
Finally, we conducted a color sensitivity analysis to inspect the impact of the color information for each neuron in the trained models.
In order to perform the analysis we adopted the color selectivity index presented by Rafegas & Vanrell (2018).
This index is deﬁned as the property of a neuron to highly activate when color is present in the input images, and to not activate when color is absent.
We computed the index for the last layer of the different backbones.
More details on the color selectivity index can be found in Appendix A.3.
The results are shown in Figure 3 and represent the amount of color sensitive neurons for each of the considered models.
As can be seen, a major difference can be observed between the Default Color Jitter and the other proposed solutions: this difference in the sensitivity of the models is mainly related to the presence of the Random GrayScale augmentation in the Default Color Jitter conﬁguration.
This result conﬁrms the hypothesis that models trained in this way are color invariant, a property that negatively affects the model in scenarios where color information has an important role, as demonstrated by our experiments.
We can see also how the Planckian Jitter (chromaticity only) conﬁguration is slightly more color sensitive with respect to the other conﬁgurations, due to the augmentation related only to realistic color variations in the scene.
C ONCLUSION
Existing research on self-supervised learning mainly focuses on tasks where color is not a decisive component, and involuntarily exploits data augmentation procedures that negatively affect colorsensitive tasks, as we showed in this paper.
We presented an alternative data augmentation technique based on the physical properties of light, called Planckian Jitter, and showed its positive effects on a wide variety of tasks where the intrinsic color of the objects (related to their reﬂectance) is crucial for discrimination, while the illumination source is not relevant, and as such the model should learn to be invariant to it.
The general applicability of the proposed strategy for data augmentation is shown by experimenting with three different popular self-supervised models, demonstrating consistent improvements.
We also proposed a solution that manages to exploit both color and shape information, by concatenating features learned with different modalities of self-supervision, achieving even superior performance.
Finally, we presented an in-depth analysis on the sensitivity of trained neurons to color information, to demonstrate the difference in models trained with and without our Planckian Jitter.
This produces a more general model for image classiﬁcation, since we obtain improvements also on non-color-sensitive general datasets.
R EPRODUCIBILITY S TATEMENT
The code of the Planckian Jitter data augmentation procedure, written in MATLAB and PyTorch 1.7.0, will be made available upon acceptance.
The training runs have been performed using Pytorch in combination with Pytorch Lightning Bolt framework, which provides an implementation of SimSiam methodology for backbone contrastive training.
The model has been trained using CUDA deterministic, and random seed set to 1234 .
All datasets used for the training and ﬁne-tuning are publicly available.
Only the CUB200 dataset has been pre-processed, by cropping each image using the given bounding box values, available alongside each image in the annotations ﬁles.
A. D ATASETS DETAILS
In section 4.2.2 we analyze the impact of our data augmentation, when using the features extracted by the backbone trained on T INY -I MAGE N ET dataset, on new datasets.
The datsets used in the ﬁne-tuning step are:
• F LOWERS -102 dataset by Nilsback & Zisserman (2008): dataset consisting of 102 ﬂower categories, commonly occuring in the United Kingdom.
Each class consists of betweenand 258 images, for a total amount of 8.189 images.
• V EG F RU dataset by Saihui Hou & Wang (2017): this dataset consists of more than 160, images of vegetables and fruits, divided in 292 classes.
• CUB-200 dataset by Welinder et al.
(2010): an image dataset made of 6033 photos ofbird species.
• T1K+ dataset by Cusano et al.
(2021): a dataset of textures divided in 1129 classes, organized in 5 groups of 266 super classes.
We adopted the 266 class labeling to ﬁne tune and test our models.
Few example images for each of the color tasks datasets are reported in Figure 6.
A. C OLOR SELECTIVITY INDEX
Color selectivity is deﬁned as the property of a neuron that activates highly when a speciﬁc color appears in the input image, and do not when the color is not present.
The color selectivity index of a neuron is computed by estimating the ratio between its global activation with input color images and the global activation with corresponding grayscale images.
By deﬁnition, the color selectivity index correspond to the complement to 1 of the ratio between the area under the activation curve given by the N-top grayscale images and the area under the activation curve given given by the N-top colored images.
[Equation]
In the equation, ( w j,i,L ) j =1: N are the neuron activation values from the N-top colored images, while ( w (cid:48)
j,i,L ) j
=1: N are the neuron activation values, by the same neurons, of the N-top grayscale images.
We can distinguish between neurons that are color blind or neurons that highly relies on color information by looking at the alpha value obtained: an alpha value more than 0 .
25 means that a speciﬁc neuron is high color selective, while an alpha value less than 0 .
1 means that the neuron is basically color blind.
These thresholds have been selected in relation to the work presented by Rafegas &
Vanrell (2018).
Using the deﬁnition of color selectivity index shown above, we collected alpha values for the neurons on the last layer of the encoders trained with different data augmentation conﬁgurations, in order to compare the models sensitivity to color and how it change in relation to the training procedure adopted.
To analyze the learned features robustness to different realistic illuminants, we tested the models with different versions of the C IFAR -100 and F LOWERS -102 datasets.
We applied the Planckian
Jitter on the two datasets, generating 25 different versions of each datasets, one for each illuminant sampled.
Using these different versions of the datasets we then test the models for each illuminant and collect the classiﬁcation accuracies.
The results on both C IFAR -100 and F LOWERS -102 can be seen in Figure 7.
A. E XECUTION TIME COMPARISON
Here we reported an execution time comparison performed to assess the usability of the proposed data augmentation, with respect to the already existent Color Jitter data augmentation algorithm. In order to analyse such aspect of the algorithms, we executed the two algorithms, the Color Jitter image transform from PyTorch Torchvision package (respectively at versions v1.8.1 and v0.9.1) and the proposed Planckian Jitter at different image resolutions. For each resolution, we run the code 40 times and averaged the execution time. Results can be seen in Figure 8.
The execution has been performed in CPU using a Intel i7-8700 processor.
As can be seen, the proposed augmentation algorithm is faster with respect to the Color Jitter algorithm.