Original: P LANCKIAN J ITTER : COUNTERING THE COLORCRIPPLING EFFECTS OF COLOR JITTER ON SELFSUPERVISED TRAINING
Revised: P LANCKIAN J ITTER : COUNTERING THE COLORCRIPPLING EFFECTS OF COLOR JITTER ON SELFSUPERVISED TRAINING

Original: Anonymous authors Paper under double-blind review
Revised: IDEAS NCBR, Warsaw, Poland

Original: A BSTRACT
Revised: A BSTRACT

Original: Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation.
Revised: Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation.

Original: The data augmentations used are of crucial importance to the quality of learned feature representations.
Revised: The data augmentations used are of crucial importance to the quality of learned feature representations.

Original: In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations.
Revised: In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations.

Original: To address this problem, we propose a more realistic, physics-based color data augmentation – which we call Planckian Jitter – that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information.
Revised: To address this problem, we propose a more realistic, physics-based color data augmentation – which we call Planckian Jitter – that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information.

Original: Experiments confirm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on a wide range of downstream datasets.
Revised: Experiments con ﬁ rm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to signi ﬁ cant performance gains on a wide range of downstream datasets.

Original: In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations.
Revised: In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations.

Revised: Of ﬁ cial code available at: https://github.com/TheZino/PlanckianJitter

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning enables the learning of representations without the need for labeled data (Doersch et al., 2015; Dosovitskiy et al., 2014).
Revised: [Table] -supervised learning enables the learning of representations without the need for labeled data (Doersch et al., 2015; Dosovitskiy et al., 2014).

Original: Several recent works learn representations that are invariant with respect to a set of data augmentations and have obtained spectacular results (Grill et al., 2020; Chen & He, 2021; Caron et al., 2020), significantly narrowing the gap with supervised learned representations.
Revised: Several recent works learn representations that are invariant with respect to a set of data augmentations and have obtained spectacular results (Grill et al., 2020; Chen & He, 2021; Caron et al., 2020), signi ﬁ cantly narrowing the gap with supervised learned representations.

Original: These works vary in their architectures, learning objectives, and optimization strategies, however they are similar in applying a common set of data augmentations to generate different image views.
Revised: These works vary in their architectures, learning objectives, and optimization strategies, however they are similar in applying a common set of data augmentations to generate different image views.

Original: These algorithms, while learning to map these different views to the same latent representation, learn rich semantic representations for visual data.
Revised: These algorithms, while learning to map these different views to the same latent representation, learn rich semantic representations for visual data.

Original: The set of transformations (data augmentations) used induces invariances that characterize the learned visual representation.
Revised: The set of transformations (data augmentations) used induces invariances that characterize the learned visual representation.

Original: Before deep learning revolutionized the way visual representations are learned, features were handcrafted to represent various properties, leading to research on shape (Lowe, 2004), texture (Manjunath & Ma, 1996), and color features (Finlayson & Schaefer, 2001; Geusebroek et al., 2001).
Revised: Before deep learning revolutionized the way visual representations are learned, features were handcrafted to represent various properties, leading to research on shape (Lowe, 2004), texture (Manjunath & Ma, 1996), and color features (Finlayson & Schaefer, 2001; Geusebroek et al., 2001).

Original: Color features were typically designed to be invariant to a set of scene-accidental events such as shadows, shading, and illuminant and viewpoint changes.
Revised: Color features were typically designed to be invariant to a set of scene-accidental events such as shadows, shading, and illuminant and viewpoint changes.

Original: With the rise of deep learning, feature representations that simultaneously exploit color, shape, and texture are learned implicitly and the invariances are a byproduct of end-to-end training (Krizhevsky et al., 2009).
Revised: With the rise of deep learning, feature representations that simultaneously exploit color, shape, and texture are learned implicitly and the invariances are a byproduct of end-to-end training (Krizhevsky et al., 2009).

Original: Current approaches to self-supervision learn a set of invariances implicitly related to the applied data augmentations.
Revised: Current approaches to self-supervision learn a set of invariances implicitly related to the applied data augmentations.

Original: In this work, we focus on the currently de facto choice for color augmentations.
Revised: In this work, we focus on the currently de facto choice for color augmentations.

Original: We argue that they seriously cripple the color quality of learned representations and we propose an alternative, physics-based color augmentation.
Revised: We argue that they seriously cripple the color quality of learned representations and we propose an alternative, physics-based color augmentation.

Original: Figure 1 (left) illustrates the currently used color augmentation on a sample image.
Revised: Figure 1 (left) illustrates the currently used color augmentation on a sample image.

Original: It is clear that the applied color transformation significantly alters the colors of the original image, both in terms of hue and saturation.
Revised: It is clear that the applied color transformation signi ﬁ cantly alters the colors of the original image, both in terms of hue and saturation.

Original: This augmentation results in a representation that is invariant with respect to surface reflectance – an invariance beneficial for recognizing classes whose surface reflectance varies significantly, for example many man-made objects such as cars and chairs.
Revised: This augmentation results in a representation that is invariant with respect to surface re ﬂ ectance – an invariance bene ﬁ cial for recognizing classes whose surface re ﬂ ectance varies signi ﬁ cantly, for example many man-made objects such as cars and chairs.

Original: However, such invariance is expected to hurt performance on downstream tasks for which color is an important feature, like natural classes such as birds or food.
Revised: However, such invariance is expected to hurt performance on downstream tasks for which color is an important feature, like natural classes such as birds or food.

Original: One of the justifications is that without large color changes, mapping images to the same latent representation can be purely done based on color and no complex shape or texture features are learned.
Revised: One of the justi ﬁ cations is that without large color changes, mapping images to the same latent representation can be purely done based on color and no complex shape or texture features are learned.

Original: However, as a result the quality of the color representation learned with such algorithms is inferior and important information on surface reflectance might be absent.
Revised: However, as a result the quality of the color representation learned with such algorithms is inferior and important information on surface re ﬂ ectance might be absent.

Original: Additionally, some traditional supervised learning methods propose domain-specific variations of color augmentation Galdran et al.
Revised: Additionally, some traditional supervised learning methods propose domain-speci ﬁ c variations of color augmentation Galdran et al.

Original: ; Xiao et al.
Revised: ; Xiao et al.

Original: In this paper we propose an alternative color augmentation (Figure 1, right) and we assess its impact on self-supervised learning.
Revised: In this paper we propose an alternative color augmentation (Figure 1, right) and we assess its impact on self-supervised learning.

Original: We draw on the existing color imaging literature on designing features invariant to illuminant changes commonly encountered in the real world (Finlayson & Schaefer, 2001).
Revised: We draw on the existing color imaging literature on designing features invariant to illuminant changes commonly encountered in the real world (Finlayson & Schaefer, 2001).

Original: Our augmentation, called Planckian Jitter , applies physically-realistic illuminant variations.
Revised: Our augmentation, called Planckian Jitter , applies physically-realistic illuminant variations.

Original: We consider the illuminants described by Planck’s Law for black-body radiation, that are known to be similar to illuminants encountered in real-life (Tominaga et al., 1999).
Revised: We consider the illuminants described by Planck’s Law for black-body radiation, that are known to be similar to illuminants encountered in real-life (Tominaga et al., 1999).

Original: The aim of our color augmentation is to allow the representation to contain valuable information about the surface reflectance of objects – a feature that is expected to be important for a wide range of downstream tasks.
Revised: The aim of our color augmentation is to allow the representation to contain valuable information about the surface re ﬂ ectance of objects – a feature that is expected to be important for a wide range of downstream tasks.

Original: Combining such a representation with the already high-quality shape and texture representation learned with standard data augmentation leads to a more complete visual descriptor that also describes color.
Revised: Combining such a representation with the already high-quality shape and texture representation learned with standard data augmentation leads to a more complete visual descriptor that also describes color.

Original: Our experiments show that self-supervised representations learned with Planckian Jitter are robust to illuminant changes.
Revised: Our experiments show that self-supervised representations learned with Planckian Jitter are robust to illuminant changes.

Original: In addition, depending on the importance of color in the dataset, the proposed Planckian jitter outperforms the default color jitter.
Revised: In addition, depending on the importance of color in the dataset, the proposed Planckian jitter outperforms the default color jitter.

Original: Moreover, for all evaluated datasets the combination of features of our new data augmentation with standard color jitter leads to significant performance gains of over 5% on several downstream classification tasks.
Revised: Moreover, for all evaluated datasets the combination of features of our new data augmentation with standard color jitter leads to signi ﬁ cant performance gains of over 5% on several downstream classi ﬁ cation tasks.

Original: Finally, we show that
Revised: Finally, we show that

Original: Planckian Jitter can be applied to several state-of-the-art self-supervised learning methods.
Revised: Planckian Jitter can be applied to several state-of-the-art self-supervised learning methods.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Self-supervised learning and contrastive learning.
Revised: Self-supervised learning and contrastive learning.

Original: Recent improvements in self-supervision learn semantically rich feature representations without the need for labelled data.
Revised: Recent improvements in self-supervision learn semantically rich feature representations without the need for labelled data.

Original: In SimCLR (Chen et al., 2020a) similar samples are created by augmenting an input image, while dissimilar are chosen  randomly (Chen et al., 2020a).
Revised: In SimCLR (Chen et al., 2020a) similar samples are created by augmenting an input image, while dissimilar are chosen [Figure] randomly (Chen et al., 2020a).

Original: To improve efficiency, MoCo (He et al., 2020) and its enhanced version (Chen et al., 2020b) use a memory bank for learned embeddings which makes sampling efficient.
Revised: To improve ef ﬁ ciency, MoCo (He et al., 2020) and its enhanced version (Chen et al., 2020b) use a memory bank for learned embeddings which makes sampling ef ﬁ cient.

Original: This memory is kept in sync with the rest of the network during training via a momentum encoder.
Revised: This memory is kept in sync with the rest of the network during training via a momentum encoder.

Original: Several methods do not rely on explicit contrastive pairs.
Revised: Several methods do not rely on explicit contrastive pairs.

Original: BYOL uses an asymmetric network incorporating an additional MLP predictor between the outputs of the two branches (Grill et al., 2020).
Revised: BYOL uses an asymmetric network incorporating an additional MLP predictor between the outputs of the two branches (Grill et al., 2020).

Original: One of the branches is kept “offline” and is updated by a momentum encoder.
Revised: One of the branches is kept “of ﬂ ine” and is updated by a momentum encoder.

Original: SimSiam defines a simplified solution without a momentum encoder (Chen & He, 2021).
Revised: SimSiam de ﬁ nes a simpli ﬁ ed solution without a momentum encoder (Chen & He, 2021).

Original: It obtains similar high-quality results and does not require a large minibatch size, in contrast to other methods.
Revised: It obtains similar high-quality results and does not require a large minibatch size, in contrast to other methods.

Original: We use the SimSiam method to verify our proposed color augmentation (we also apply it to SimCLR (Chen et al., 2020a) and
Revised: We use the SimSiam method to verify our proposed color augmentation (we also apply it to SimCLR (Chen et al., 2020a) and

Original: Barlow Twins (Zbontar et al., 2021) in the experiments).The main [Figure] op
Revised: Barlow Twins (Zbontar et al., 2021) in the experiments).

Original: [Figure] ad [Figure] component of the network is a CNN-based asymmetric siamese image encoder.
Revised: The main component of the network is a CNN-based asymmetric siamese image encoder.

Original: One branch has an additional MLP predictor whose output aims to be as close as possible to the other (Figure 2).
Revised: One branch has an additional MLP predictor whose output aims to be as close as possible to the other (Figure 2).

Original: The second branch is not updated during backpropagation.
Revised: The second branch is not updated during backpropagation.

Original: A negative cosine loss function is used:
Revised: A negative cosine loss function is used:

Original: [Equation]
Revised: [Equation]

Original: where z 1 , z 2 are representations for two different augmented versions, x 1 and x 2 , of the same image x .
Revised: where z 1 , z 2 are representations for two different augmented versions, x 1 and x 2 , of the same image x .

Original: The MLP is applied in alternation to either z 1 or z 2 , producing respectively p 1 or p 2 .
Revised: The MLP is applied in alternation to either z 1 or z 2 , producing respectively p 1 or p 2 .

Original: Note that
Revised: Note that

Original: Figure 2 only shows an instance for x 1 and does not show p 2 .
Revised: Figure 2 only shows an instance for x 1 and does not show p 2 .

Original: The stopgrad ( · ) operation blocks the gradient during the backpropagation.
Revised: The stopgrad ( · ) operation blocks the gradient during the backpropagation.

Original: In SimSiam no contrastive term is used and only similarity is enforced during learning.
Revised: In SimSiam no contrastive term is used and only similarity is enforced during learning.

Original: Data augmentation
Revised: Data augmentation.

Original: Data augmentation plays a central role in the self-supervised learning process.
Revised: Data augmentation plays a central role in the self-supervised learning process.

Original: Authors Chen et al.
Revised: Authors Chen et al.

Original: (2020a) and Zbontar et al.
Revised: (2020a) and Zbontar et al.

Original: (2021) discuss the importance of the different data augmentations.
Revised: (2021) discuss the importance of the different data augmentations.

Original: A set of well-defined transformations was proposed for SimCLR (Chen et al., 2020a).
Revised: A set of well-de ﬁ ned transformations was proposed for SimCLR (Chen et al., 2020a).

Original: This set is commonly accepted and used in several later works.
Revised: This set is commonly accepted and used in several later works.

Original: The augmentations include: rotation, cutout, flip, color jitter, blur and Grayscale.
Revised: The augmentations include: rotation, cutout, ﬂ ip, color jitter, blur and Grayscale.

Original: These operations are randomly applied to an image to generate the different views x 1 , x 2 from which are extracted the features z 1 and z 2 used in the self-supervision loss in Eq. 2.
Revised: These operations are randomly applied to an image to generate the different views x 1 , x 2 from which are extracted the features z 1 and z 2 used in the self-supervision loss in Eq. 2.

Original: Applied to the same image, contrastive-like self-supervision learns representations invariant to such distortions.
Revised: Applied to the same image, contrastive-like self-supervision learns representations invariant to such distortions.

Original: This multiple view creation is task-related (Tian et al., 2020), however color jittering operating on hue, saturation, brightness and contrast, is one of the most important ones in terms of overall usefulness of the learned representation for downstream tasks (Chen et al., 2020a; Zbontar et al., 2021).
Revised: This multiple view creation is task-related (Tian et al., 2020), however color jittering operating on hue, saturation, brightness and contrast, is one of the most important ones in terms of overall usefulness of the learned representation for downstream tasks (Chen et al., 2020a; Zbontar et al., 2021).

Original: Color jitter induces a certain level of color invariance (invariance to hue, saturation, brightnesss and contrast) which are consequently transferred to the downstream task.
Revised: Color jitter induces a certain level of color invariance (invariance to hue, saturation, brightnesss and contrast) which are consequently transferred to the downstream task.

Original: As a consequence, we expect these learned features to underperform on downstream tasks for which color is crucial.
Revised: As a consequence, we expect these learned features to underperform on downstream tasks for which color is crucial.

Original: Xiao et al.
Revised: Xiao et al.

Original: (2020) were the first to point out that the imposed invariances might not be beneficial for downstream tasks.
Revised: (2020) were the ﬁ rst to point out that the imposed invariances might not be bene ﬁ cial for downstream tasks.

Original: As a solution, they propose to learn different embedding spaces in parallel that capture each of the invariances.
Revised: As a solution, they propose to learn different embedding spaces in parallel that capture each of the invariances.

Original: Differently than them, we focus on the color distortion and propose a physics-based color augmentation that allows learning invariance to physically realistic color variations.
Revised: Differently than them, we focus on the color distortion and propose a physics-based color augmentation that allows learning invariance to physically realistic color variations.

Original: Color imaging has a long tradition in research on color features invariant to scene-accidental events such as shading, shadows, and illuminant changes (Geusebroek et al., 2001;
Revised: Color imaging has a long tradition in research on color features invariant to scene-accidental events such as shading, shadows, and illuminant changes (Geusebroek et al., 2001;

Original: Finlayson & Schaefer, 2001).
Revised: Finlayson & Schaefer, 2001).

Original: Invariant features were found to be extremely beneficial for object recognition.
Revised: Invariant features were found to be extremely bene ﬁ cial for object recognition.

Original: The invariance to hue and saturation changes induced by color jitter, however, is detrimental to object recognition for classes in which color characteristics are fundamentally discriminative.
Revised: The invariance to hue and saturation changes induced by color jitter, however, is detrimental to object recognition for classes in which color characteristics are fundamentally discriminative.

Original: Therefore, in this work we revisit early theory on illuminant invariance (Finlayson & Schaefer, 2001) to design an improved color augmentation that induces invariances common in the real world and that, when used during self-supervised learning, does not damage the color quality of the learned features.
Revised: Therefore, in this work we revisit early theory on illuminant invariance (Finlayson & Schaefer, 2001) to design an improved color augmentation that induces invariances common in the real world and that, when used during self-supervised learning, does not damage the color quality of the learned features.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: The image transformations introduced by default color jitter creates variability in training data that indiscriminately explores all hues at various levels of saturation.
Revised: The image transformations introduced by default color jitter creates variability in training data that indiscriminately explores all hues at various levels of saturation.

Original: The resulting invariance is useful for downstream tasks where chromatic variations are indeed irrelevant (e.g. car color in vehicle recognition), but is detrimental to downstream tasks where color information is critical (e.g. natural classes like birds and vegetables).
Revised: The resulting invariance is useful for downstream tasks where chromatic variations are indeed irrelevant (e.g. car color in vehicle recognition), but is detrimental to downstream tasks where color information is critical (e.g. natural classes like birds and vegetables).

Original: The main motivation for applying strong color augmentations is that this it leads to very strong shape and texture representations.
Revised: The main motivation for applying strong color augmentations is that this it leads to very strong shape and texture representations.

Original: Indiscriminately augmenting color information in the image requires that the representation solve the matching problem using shape (Chen et al., 2020a) 1 .
Revised: Indiscriminately augmenting color information in the image requires that the representation solve the matching problem using shape (Chen et al., 2020a) 1 .

Original: As an alternative to color jitter, we propose a physics-based color augmentation that mimics color variations due to illuminant changes commonly encountered in the real world.
Revised: As an alternative to color jitter, we propose a physics-based color augmentation that mimics color variations due to illuminant changes commonly encountered in the real world.

Original: The aim is to reach a representation that does not have the color crippling effects of color jitter, and that better describes classes for which surface reflectance is a determining feature.
Revised: The aim is to reach a representation that does not have the color crippling effects of color jitter, and that better describes classes for which surface re ﬂ ectance is a determining feature.

Original: When combined with default color jitter, this representation should also provide a high-quality shape/texture and color representation.
Revised: When combined with default color jitter, this representation should also provide a high-quality shape/texture and color representation.

Original: 3. P LANCKIAN J ITTER
Revised: 3. P LANCKIAN J ITTER

Original: We call our color data augmentation procedure Planckian Jitter because it exploits the physical description of a black-body radiator to re-illuminate training images within a realistic illuminant distribution (Finlayson & Schaefer, 2001; Tominaga et al., 1999).
Revised: We call our color data augmentation procedure Planckian Jitter because it exploits the physical description of a black-body radiator to re-illuminate training images within a realistic illuminant distribution (Finlayson & Schaefer, 2001; Tominaga et al., 1999).

Original: The resulting augmentations are more realistic than those of the default color jitter (see Fig. 1).
Revised: The resulting augmentations are more realistic than those of the default color jitter (see Fig. 1).

Original: The resulting learned, self-supervised feature representation is thus expected to be robust to illumination changes commonly observed in real-world images, while simultaneously maintaining the ability to discriminate the image content based on color information.
Revised: The resulting learned, self-supervised feature representation is thus expected to be robust to illumination changes commonly observed in real-world images, while simultaneously maintaining the ability to discriminate the image content based on color information.

Original: Given an input RGB training image I , our Planckian Jitter procedure applies a chromatic adaptation transform that simulates realistic variations in the illumination conditions.
Revised: Given an input RGB training image I , our Planckian Jitter procedure applies a chromatic adaptation transform that simulates realistic variations in the illumination conditions.

Original: The data augmentation procedure is as follows:
Revised: The data augmentation procedure is as follows:

Original: we sample a new illuminant spectrum σ T ( λ ) from the distribution of a black-body radiator; 2.
Revised: we sample a new illuminant spectrum σ T ( λ ) from the distribution of a black-body radiator; 2.

Original: we transform the sampled spectrum σ T ( λ ) into its sRGB representation ρ T ∈ R 3 ; 3.
Revised: we transform the sampled spectrum σ T ( λ ) into its sRGB representation ρ T ∈ R 3 ; 3.

Original: we create a jittered image I ′ by reilluminating I with the sampled illuminant ρ T ; and 4.
Revised: we create a jittered image I ′ by reilluminating I with the sampled illuminant ρ T ; and 4.

Original: we introduce brightness and contrast variation, producing a Planckian-jittered image I ′′ .
Revised: we introduce brightness and contrast variation, producing a Planckian-jittered image I ′′ .

Original: A radiating black body at temperature T can be synthesized using Planck’s Law (Andrews, 2010):
Revised: A radiating black body at temperature T can be synthesized using Planck’s Law (Andrews, 2010):

Original: [Equation]
Revised: [Equation]

Original:  where c = 2 . 

Revised: (3) where c = 2 .

Original: × 10 8 m/s is the speed of light, h = 6 . 

Revised: × 10 8 m/s is the speed of light, h = 6 .

Original: × 10 − 34 Js is Planck’s constant, and k = 1 .
Revised: × 10 − 34 Js is Planck’s constant, and k = 1 .

Original: 380662 × 10 − 23 J/K is Boltzmann’s constant.
Revised: 380662 × 10 − 23 J/K is Boltzmann’s constant.

Original: We sampled T in the interval between 3000 K and 15000 K which is known to result in a set of illuminants that can be encountered in real life (Tominaga et al., 1999).
Revised: We sampled T in the interval between 3000 K and 15000 K which is known to result in a set of illuminants that can be encountered in real life (Tominaga et al., 1999).

Original: Then, we discretized wavelength λ in 10nm steps ( ∆ λ ) in the interval between 400nm and 700nm.
Revised: Then, we discretized wavelength λ in 10nm steps ( ∆ λ ) in the interval between 400nm and 700nm.

Original: The resulting spectra are visualized in Figure 4 (left) in Appendix A.1.
Revised: The resulting spectra are visualized in Figure 4 (left) in Appendix A.1.

Original: The conversion from spectrum into sRGB is obtained according to Wyszecki & Stiles (1982):
Revised: The conversion from spectrum into sRGB is obtained according to Wyszecki & Stiles (1982):

Original: we first map the spectrum into the corresponding XYZ stimuli, using the 1931 CIE standard observer color matching functions c { X,Y,Z } ( λ ) , in order to bring the illuminant into a standard color space that represents a person with average eyesight; 2.
Revised: we ﬁ rst map the spectrum into the corresponding XYZ stimuli, using the 1931 CIE standard observer color matching functions c { X,Y,Z } ( λ ) , in order to bring the illuminant into a standard color space that represents a person with average eyesight; 2.

Original: we normalize this tristimulus by its Y component, convert it into the CIE 1976 L*a*b color space, and fix its L component to 50 in a 0-to-100 scale, allowing us to constrain the intensity of the represented illuminant in a controlled manner as a separate task; and 3.
Revised: we normalize this tristimulus by its Y component, convert it into the CIE 1976 L*a*b color space, and ﬁ x its L component to 50 in a 0-to-100 scale, allowing us to constrain the intensity of the represented illuminant in a controlled manner as a separate task; and 3.

Original: we then convert the resulting values to sRGB, applying a gamma correction and obtaining ρ T 
Revised: we then convert the resulting values to sRGB, applying a gamma correction and obtaining ρ T =

Revised: R,

Revised: G

Original: = { R, G, B } ; the resulting distribution of illuminants is visualized with the AngleRetaining Chromaticity diagram (Buzzelli et al., 2020) in Figure right) in Appendix A.
Revised: B } ; the resulting distribution of illuminants is visualized with the AngleRetaining Chromaticity diagram (Buzzelli et al., 2020) in Figure 4 (right) in Appendix A.1.

Original: All color space conversions assume a D65 reference white, which means that a neutral surface illuminated by average daylight conditions would appear achromatic.
Revised: All color space conversions assume a D65 reference white, which means that a neutral surface illuminated by average daylight conditions would appear achromatic.

Original: Once the new illuminant has been converted in sRGB, it is applied to the input image I by resorting to a Von-Kries-like transform (von Kries, 1902) given by the following channel-wise scalar multiplication:
Revised: Once the new illuminant has been converted in sRGB, it is applied to the input image I by resorting to a Von-Kries-like transform (von Kries, 1902) given by the following channel-wise scalar multiplication:

Original: [Equation]
Revised: [Equation]

Original: where we assume the original scene illuminant to be white (1,1,1).
Revised: where we assume the original scene illuminant to be white (1,1,1).

Original: Finally, brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination:
Revised: Finally, brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination:

Original: [Equation]
Revised: [Equation]

Original: where c B = 0 .
Revised: where c B = 0 .

Original: 8 and c C = 0 .
Revised: 8 and c C = 0 .

Original: 8 represent, respectively, brightness and contrast coefficients, and µ is a spatial average function.
Revised: 8 represent, respectively, brightness and contrast coef ﬁ cients, and µ is a spatial average function.

Original: 3. C OMPLIMENTARITY OF SHAPE , TEXTURE AND COLOR REPRESENTATIONS
Revised: 3. C OMPLIMENTARITY OF SHAPE , TEXTURE AND COLOR REPRESENTATIONS

Original: The self-supervised learning paradigm involves a pretraining phase that relies on data augmentation to produce a set of features with certain invariance properties.
Revised: The self-supervised learning paradigm involves a pretraining phase that relies on data augmentation to produce a set of features with certain invariance properties.

Original: These features are then used as the representation for a second phase, where we learn a given supervised downstream task.
Revised: These features are then used as the representation for a second phase, where we learn a given supervised downstream task.

Original: The default color jitter augmentation generates features that are strongly invariant to color information, resulting in high-quality representations of shape and texture, but that is an inferior descriptor of surface reflectances (i.e. the color of objects).
Revised: The default color jitter augmentation generates features that are strongly invariant to color information, resulting in high-quality representations of shape and texture, but that is an inferior descriptor of surface re ﬂ ectances (i.e. the color of objects).

Original: Our augmentation based on Planckian Jitter (see Figure 1) is based on transformations mimicking the physical color variations in the real world due to illuminant changes.
Revised: Our augmentation based on Planckian Jitter (see Figure 1) is based on transformations mimicking the physical color variations in the real world due to illuminant changes.

Original: As a result, the learned representation yields a high-quality color description of scene objects (this is also verified in Appendix A.8).
Revised: As a result, the learned representation yields a high-quality color description of scene objects (this is also veri ﬁ ed in Appendix A.8).

Original: However, it likely leads to a drop in the quality of the shape and texture representation (since color can be used to solve cases where previously shape/texture were required).
Revised: However, it likely leads to a drop in the quality of the shape and texture representation (since color can be used to solve cases where previously shape/texture were required).

Original: To exploit the complimentarity of the two representations, we propose to learn both – one with color jitter and one with Planckian Jitter – and to then concatenate the results in a single representation vector (of 1024 dimensions, i.e. twice the original size of 512).
Revised: To exploit the complimentarity of the two representations, we propose to learn both – one with color jitter and one with Planckian Jitter – and to then concatenate the results in a single representation vector (of 1024 dimensions, i.e. twice the original size of 512).

Original: We call this Latent space combination (LSC) .
Revised: We call this Latent space combination (LSC) .

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: In this section, we analyze the color sensitivity of the learned backbone networks, verify the superiority of the proposed color data augmentation method compared to the default color jitter on color datasets, and evaluate the impact on downstream classification tasks.
Revised: In this section, we analyze the color sensitivity of the learned backbone networks, verify the superiority of the proposed color data augmentation method compared to the default color jitter on color datasets, and evaluate the impact on downstream classi ﬁ cation tasks.

Original: We report additional results on computational time of the proposed Planckian augmentation in Appendix A.5.
Revised: We report additional results on computational time of the proposed Planckian augmentation in Appendix A.5.

Original: 4. T RAINING AND EVALUATION SETUP
Revised: 4. T RAINING AND EVALUATION SETUP

Original: We perform unsupervised training on two datasets: CIFAR-100 (Krizhevsky et al., 2009) ( 32 × 32 ) and ImageNet ( 224 × 224 ).
Revised: We perform unsupervised training on two datasets: CIFAR-100 (Krizhevsky et al., 2009) ( 32 × 32 ) and ImageNet ( 224 × 224 ).

Original: 2 We slightly modify the ResNet18 architecture to accommodate 32 ×images: the kernel size of the first convolutional was reduced from 7 × 7 to 3 × 3 and the first max  pooling layer was removed.
Revised: 2 We slightly modify the ResNet18 architecture to accommodate 32 ×images: the kernel size of the ﬁ rst convolutional was reduced from 7 × 7 to 3 × 3 and the ﬁ rst max agile manner on low-resolution images then pooling layer was removed.

Original: SimSiam training was performed using Stochastic Gradient Descent with a starting learning rate of 0 .
Revised: SimSiam training was performed using Stochastic Gradient Descent with a starting learning rate of 0 .

Original: 03 , a cosine annealing learning rate scheduler, and mini-batch size of 512 (as in original SimSiam work by Chen & He (2021)).
Revised: 03 , a cosine annealing learning rate scheduler, and mini-batch size of 512 (as in original SimSiam work by Chen & He (2021)).

Original: For the training on the 1000-class
Revised: For the training on the 1000-class

Original: ImageNet training set, we follow the same procedure as Chen & He (2021) with ResNet50.
Revised: ImageNet training set, we follow the same procedure as Chen & He (2021) with ResNet50.

Original: The linear classifier training at resolution 32 × 32 was performed on C IFAR -100 and F LOWERS102 (Nilsback & Zisserman, 2008).
Revised: The linear classi ﬁ er training at resolution 32 × 32 was performed on C IFAR -100 and F LOWERS102 (Nilsback & Zisserman, 2008).

Original: C IFAR -100 is used as a baseline for the classification task.
Revised: C IFAR -100 is used as a baseline for the classi ﬁ cation task.

Original: The linear classifier training for C IFAR -100 is done with Stochastic Gradient Descent for 500 epochs with a starting learning rate 0 .
Revised: The linear classi ﬁ er training for C IFAR -100 is done with Stochastic Gradient Descent for 500 epochs with a starting learning rate 0 .

Original: 1 , a cosine annealing learning rate scheduler, and mini-batch size of 512.
Revised: 1 , a cosine annealing learning rate scheduler, and mini-batch size of 512.

Original: The F LOWERS -102 dataset with 102 classes was selected to assess the quality of the features extracted in scenarios where color information plays an important role.
Revised: The F LOWERS -102 dataset with 102 classes was selected to assess the quality of the features extracted in scenarios where color information plays an important role.

Original: Images from F LOWERSare resized to 32 × 32 pixels to match the input dimensions of the pretrained model.
Revised: Images from F LOWERSare resized to 32 × 32 pixels to match the input dimensions of the pretrained model.

Original: Here we used the Adam optimizer with initial learning rate of 0 . 03 .
Revised: Here we used the Adam optimizer with initial learning rate of 0 . 03 .

Original: the investigative part of our research in an agile manner on low-resolution images, then transfer the most significant configurations to a higher-resolution, to ether confirm or refute the initial hypotheses

Original: [Figure] For training linear classifiers at resolution 224 × 224 for downstream tasks we follow the evaluation protocol of Chen & He (2021).
Revised: For training linear classi ﬁ ers at resolution 224 × 224 for downstream tasks we follow the evaluation protocol of Chen & He (2021).

Original: We use six different datasets: I MAGE N ET , F LOWERS -102, the finegrained V EG F RU (Saihui Hou & Wang, 2017), CUB-200 (Welinder et al., 2010), T1K+ (Cusano et al., 2021), and USED (Ahmad et al., 2016), all resized to 224 × 224 pixels.
Revised: We use six different datasets: I MAGE N ET , F LOWERS -102, the ﬁ negrained V EG F RU (Saihui Hou & Wang, 2017), CUB-200 (Welinder et al., 2010), T1K+ (Cusano et al., 2021), and USED (Ahmad et al., 2016), all resized to 224 × 224 pixels.

Original: More details about the datasets are provided in Appendix A.2.
Revised: More details about the datasets are provided in Appendix A.2.

Original: In the case of CUB-200, each image was cropped using the bounding boxes given in the dataset annotations.
Revised: In the case of CUB-200, each image was cropped using the bounding boxes given in the dataset annotations.

Original: For T1K+, we used the 266 class labeling to train and test the linear classifier.
Revised: For T1K+, we used the 266 class labeling to train and test the linear classi ﬁ er.

Original: To assess the impact of color data augmentations we define six different configurations:
Revised: To assess the impact of color data augmentations we de ﬁ ne six different con ﬁ gurations:

Original: • Default Color Jitter (CJ): the default configuration, as used in SimSiam and SimCLR, uses both Random Color Jitter and Random Grayscale operations.
Revised: • Default Color Jitter (CJ): the default con ﬁ guration, as used in SimSiam and SimCLR, uses both Random Color Jitter and Random Grayscale operations.

Original: • Default Color Jitter w/o Grayscale (CJ-) : same as Default , without Random Grayscale.
Revised: • Default Color Jitter w/o Grayscale (CJ-) : same as Default , without Random Grayscale.

Original: • Planckian Jitter (PJ)
Revised: • Planckian Jitter (PJ)

Original: : uses the complete proposed Planckian Jitter operating on chromaticy, brightness, and contrast aspects of the images.
Revised: : uses the complete proposed Planckian Jitter operating on chromaticy, brightness, and contrast aspects of the images.

Original: No Random Grayscale is applied.
Revised: No Random Grayscale is applied.

Original: • LSC Default Color Jitter + Planckian Jitter ([CJ,PJ] : This latent space combination (simple concatenation of representations) combines the default color jitter with our Planckian jitter.
Revised: • LSC Default Color Jitter + Planckian Jitter ([CJ,PJ] : This latent space combination (simple concatenation of representations) combines the default color jitter with our Planckian jitter.

Original: It allows evaluation of the complimentary nature of the representations.
Revised: It allows evaluation of the complimentary nature of the representations.

Original: • LSC Default Color Jitter + Default Color Jitter w/o Grayscale ([CJ,CJ-]) :
Revised: • LSC Default Color Jitter + Default Color Jitter w/o Grayscale ([CJ,CJ-]) :

Original: We combine the default color jitter with a version without the Grayscale augmentation, since this representation is also expected to result in a better color representation.
Revised: We combine the default color jitter with a version without the Grayscale augmentation, since this representation is also expected to result in a better color representation.

Original: • LSC of two Default Color Jitter Models ([CJ,CJ]) : We also show results of simply concatenating two independently trained models (trained from different seeds) with default color jitter (an ensemble of two models).
Revised: • LSC of two Default Color Jitter Models ([CJ,CJ]) : We also show results of simply concatenating two independently trained models (trained from different seeds) with default color jitter (an ensemble of two models).

Original: In all experiments, these are combined with the other default augmentations (crop, flip, and blur).
Revised: In all experiments, these are combined with the other default augmentations (crop, ﬂ ip, and blur).

Original: 4. C OLOR SENSITIVITY ANALYSIS
Revised: 4. C OLOR SENSITIVITY ANALYSIS

Original: We perform a robustness analysis on the VegFru and CUB-200 datasets with realistic illuminant variations, and analyzed sensitivity to color information.
Revised: We perform a robustness analysis on the VegFru and CUB-200 datasets with realistic illuminant variations, and analyzed sensitivity to color information.

Original: This experiment is driven by two motivations: to verify that we obtain invariance to the transformation applied during training, and to characterize the degradation of different non-Planckian training modalities.
Revised: This experiment is driven by two motivations: to verify that we obtain invariance to the transformation applied during training, and to characterize the degradation of different non-Planckian training modalities.

Original: We assume as reference point the D65 illuminant, which for the purpose of this test is considered the default illuminant in every image.
Revised: We assume as reference point the D65 illuminant, which for the purpose of this test is considered the default illuminant in every image.

Original: Given the different backbones pretrained on I MAGE N ET , we then train a linear classifier on this dataset (assumed to be under white illumination).
Revised: Given the different backbones pretrained on I MAGE N ET , we then train a linear classiﬁ er on this dataset (assumed to be under white illumination).

Original: For testing we create different versions of the dataset, each illuminated by illuminants of differing color temperature.
Revised: For testing we create different versions of the dataset, each illuminated by illuminants of differing color temperature.

Original: This allows us to evaluate the robustness of the learned representations with respect to these illuminant changes.
Revised: This allows us to evaluate the robustness of the learned representations with respect to these illuminant changes.

Original: Results are given in Figure 3(a) (more results are provided in Figure 7 from Appendix A.4).
Revised: Results are given in Figure 3(a) (more results are provided in Figure 7 from Appendix A.4).

Original: Planckian Jitter obtains a remarkably stable performance between 4000K and 14000K, while Default Color
Revised: Planckian Jitter obtains a remarkably stable performance between 4000K and 14000K, while Default Color

Original: [Table]

Original: Jitter is more sensitive to the illumination color and the classification accuracy decreases when the scene illuminant moves away from white.
Revised: Jitter is more sensitive to the illumination color and the classi ﬁ cation accuracy decreases when the scene illuminant moves away from white.

Original: We also see that the combination of default and Planckian
Revised: We also see that the combination of default and Planckian

Original: Jitter obtains the best results for all illuminants and manages to maintain a high-level of invariance with respect to the illuminant color.
Revised: Jitter obtains the best results for all illuminants and manages to maintain a high-level of invariance with respect to the illuminant color.

Original: Among the non-Planckian curves, default color jitter (CJ) is the most invariant, followed by CJ+CJ- (although showing better performance at D65), and finally CJ-. 
Revised: Among the non-Planckian curves, default color jitter (CJ) is the most invariant, followed by CJ+CJ- (although showing better performance at D65), and ﬁ nally CJ-. [Figure] AUGMENTATION [Table]

Original: In order to understand the impact of the color information on each neuron in trained models
Revised: In order to understand the impact of the color information on each neuron in trained models

Original: [Table]
Revised: [Table]

Original: we conducted an analysis using the color selectivity index described by Rafegas & Vanrell (2018).
Revised: we conducted an analysis using the color selectivity index described by Rafegas & Vanrell (2018).

Original: This index measures neuron activation when color is present or absent in input images.
Revised: This index measures neuron activation when color is present or absent in input images.

Original: We computed the index for the last layer of different backbones, and high values indicate color-sensitive neurons.
Revised: We computed the index for the last layer of different backbones [Table] and high values indicate color-sensitive neurons.

Original: See
Revised: See

Original: Appendix A.3 for more details on color selectivity.
Revised: Appendix A.3 for more details on color selectivity.

Original: The results are shown in Figure 3(b) and indicate the number of color-sensitive neurons for each of the considered models.
Revised: The results are shown in Figure 3(b) and indicate the number of color-sensitive neurons for each of the considered models.

Original: It is clear that the default color jitter has far fewer neurons dedicated to color description.
Revised: It is clear that the default color jitter has far fewer neurons dedicated to color description.

Original: This result confirms the hypothesis that models trained in this way are color invariant, a property that negatively affects the model in scenarios where color information has an important role as seen in our experiments.
Revised: This result con ﬁ rms the hypothesis that models trained in this way are color invariant, a property that negatively affects the model in scenarios where color information has an important role as seen in our experiments.

Original: We have also analyzed the results for the default color jitter without Grayscale augmentation (CJ-).
Revised: We have also analyzed the results for the default color jitter without Grayscale augmentation (CJ-).

Original: These results show that removing the Grayscale augmentation improves color sensitivity significantly.
Revised: These results show that removing the Grayscale augmentation improves color sensitivity signi ﬁ cantly.

Original: We therefore also consider this augmentation in future experiments.
Revised: We therefore also consider this augmentation in future experiments.

Original: 4. A BLATION STUDY
Revised: 4. A BLATION STUDY

Original: Six different models were trained and evaluated with a linear classification for image classification.
Revised: Six different models were trained and evaluated with a linear classi ﬁ cation for image classi ﬁ cation.

Original: For resolution 32 × 32 the model is evaluated on C IFAR -100 and F LOWERS -102.
Revised: For resolution 32 × 32 the model is evaluated on C IFAR -100 and F LOWERS -102.

Original: The results in terms of accuracy are reported in Table  2.
Revised: The results in terms of accuracy are reported in Table 1 and Table 2.

Original: We identify two different trends when interpreting these results.
Revised: We identify two different trends when interpreting these results.

Original: On C IFAR -100, removing color augmentations makes the model less powerful, due to the loss of color invariance in the features extracted by the encoder.
Revised: On C IFAR -100, removing color augmentations makes the model less powerful, due to the loss of color invariance in the features extracted by the encoder.

Original: This behaviour is consistent with what was reported by Chen et al.
Revised: This behaviour is consistent with what was reported by Chen et al.

Original: (2020a).
Revised: (2020a).

Original: We see in Table 1 that if color augmentations (i.e. brightness/contrast and Random Grayscale) are removed completely (the None configuration), the accuracy drops by 18% .
Revised: We see in Table 1 that if color augmentations (i.e. brightness/contrast and Random Grayscale) are removed completely (the None con ﬁ guration), the accuracy drops by 18% .

Original: On F LOWERS -102 the behavior is the opposite however: removing color augmentations helps the model to better classify images, obtaining an improvement of 12 . 
Revised: On F LOWERS -102 the behavior is the opposite however: removing color augmentations helps the model to better classify images, obtaining an improvement of 12 . [Table]

Original: 75% of accuracy with respect to the default color jitter.
Revised: of accuracy with respect to the default color jitter.

Original: This behavior confirms that color invariance negatively impacts downstream tasks where color information plays an important role.
Revised: This behavior con ﬁ rms that color invariance negatively impacts downstream tasks where color information plays an important role.

Original: Taking a closer look at the various augmentation on F LOWERS -102, we see that introducing more realistic color augmentations positively impacts contrastive training and produces models that achieveAUGMENTATION [Table] even better results with respect to the configuration without any kind of image color manipulation.
Revised: Taking a closer look at the various augmentation on F LOWERS -102, we see that introducing more realistic color augmentations positively impacts contrastive training and produces models that achieve even better results with respect to the con ﬁ guration without any kind of image color manipulation.

Original: Removing all color augmentations (None) improves results already by over 6%.
Revised: Removing all color augmentations (None) improves results already by over 6%.

Original: Then, by simply reducing the jittering operation to influence brightness and contrast, leaving hue and saturation unchanged, yields another boost in accuracy of 5 .
Revised: Then, by simply reducing the jittering operation to in ﬂ uence brightness and contrast, leaving hue and saturation unchanged, yields another boost in accuracy of 5 .

Original: 49% (to 41.96).
Revised: 49% (to 41.96).

Original: When we start modifying chromaticity using a more realistic transformation (i
Revised: When we start modifying chromaticity using a more realistic transformation (i

Original: [Table]
Revised: [Table]

Original: e Planckian Jitter ), the final result is a boost of 6 .
Revised: e Planckian Jitter ), the ﬁ nal result is a boost of 6 .

Original: 28% in accuracy with respect to the None configuration.
Revised: 28% in accuracy with respect to the None con ﬁ guration.

Original: Also, on C IFAR -100 we see an improvement of 5 .
Revised: Also, on C IFAR -100 we see an improvement of 5 .

Original: 38% from Planckian Jitter with respect no color augmentation.
Revised: 38% from Planckian Jitter with respect no color augmentation.

Original: Despite this improvement, in this scenario the contrastive training with the realistic augmentation does not yield better results with respect to the Default configuration because color only plays a minor role on this dataset.
Revised: Despite this improvement, in this scenario the contrastive training with the realistic augmentation does not yield better results with respect to the Default con ﬁ guration because color only plays a minor role on this dataset.

Original: Given the results obtained using the data augmentations reported in Table
Revised: Given the results obtained using the data augmentations reported in Table

Original: [Table]
Revised: [Table]

Original: and given the considerations made in Section 3. [Table] we evaluate the complementarity of the learned representation by combining latent spaces from different backbones.
Revised: and given the considerations made in Section 3.2, we evaluate the complementarity of the learned representation by combining latent spaces from different backbones.

Original: Results for two different latent space combinations are given in Table 4.
Revised: Results for two different latent space combinations are given in Table 4.

Original: On both datasets the Latent space combination of Default and Planckian
Revised: On both datasets the Latent space combination of Default and Planckian

Original: Jitter achieves the best results.
Revised: Jitter achieves the best results.

Original: On the original C IFAR -100 task, this combination achieves a total accuracy of 63 .
Revised: On the original C IFAR -100 task, this combination achieves a total accuracy of 63 .

Original: 54% , a 3 . 61% improvement over Default and 16 .
Revised: 54% , a 3 . 61% improvement over Default and 16 .

Original: 23% more compared to Planckian
Revised: 23% more compared to Planckian

Original: Jitter alone.
Revised: Jitter alone.

Original: Comparing to the LSC using the Default ColorJitter w/o Grayscale, the version with
Revised: Comparing to the LSC using the Default ColorJitter w/o Grayscale, the version with

Original: Planckian Jitter achieves a small improvement of 1 . 27% in classification accuracy.
Revised: Planckian Jitter achieves a small improvement of 1 . 27% in classi ﬁ cation accuracy.

Original: On the downstream F LOWERS -102 task, Latent space combination reaches an accuracy value of 51 .
Revised: On the downstream F LOWERS -102 task, Latent space combination reaches an accuracy value of 51 .

Original: 66% : an improvement of 21 .
Revised: 66% : an improvement of 21 .

Original: 66% and 8 .
Revised: 66% and 8 .

Original: 91% in accuracy respectively compared to the two original configurations.
Revised: 91% in accuracy respectively compared to the two original con ﬁ gurations.

Original: Compared to the LSC using Default ColorJitter w/o Grayscale, the combination with Planckian Jitter achieves a higher result, and a bigger gap in terms of accuracy with respect to the C IFAR -100 scenario.
Revised: Compared to the LSC using Default ColorJitter w/o Grayscale, the combination with Planckian Jitter achieves a higher result, and a bigger gap in terms of accuracy with respect to the C IFAR -100 scenario.

Original: Here the use of Planckian Jitter brings an improvement of 4 .
Revised: Here the use of Planckian Jitter brings an improvement of 4 .

Original: 01% , confirming the impact of using realistic augmentation on classification tasks for which color is important.
Revised: con ﬁ rming the impact of using realistic augmentation on classi ﬁ cation tasks for which color is important.

Original: 4. E VALUATION ON DOWNSTREAM TASKS
Revised: 4. E VALUATION ON DOWNSTREAM TASKS

Original: Given the ablation study results, we performed the analysis of the proposed configurations on other downstream tasks using the backbone trained on higher resolution images ( 224 × 224 pixels).
Revised: Given the ablation study results, we performed the analysis of the proposed con ﬁ gurations on other downstream tasks using the backbone trained on higher resolution images ( 224 × 224 pixels).

Original: We report in Table 

Revised: We report in Table

Original: the results for: Default Color Jitter , Planckian Jitter , and latent space combinations.
Revised: the results for: Default Color Jitter , Planckian Jitter , and latent space combinations.

Original: Looking at the results, we see that the Planckian Jitter augmentation outperforms default color jitter on three datasets (CUB-200, T1K+, and USED).
Revised: Looking at the results, we see that the Planckian Jitter augmentation outperforms default color jitter on three datasets (CUB-200, T1K+, and USED).

Original: Comparing the results on F LOWERS -102 with those reported above at (32 × 32 ) pixels, we see that default color jitter actually obtains good results.
Revised: Comparing the results on F LOWERS -102 with those reported above at (32 × 32 ) pixels, we see that default color jitter actually obtains good results.

Original: We hypothesize that for high-resolution images the shape/texture information is very discriminative, and the additional color information yields little gain (for further analysis, see also A.7).
Revised: We hypothesize that for high-resolution images the shape/texture information is very discriminative, and the additional color information yields little gain (for further analysis, see also A.7).

Original: Table 3 also contains results for latent space combination, which confirm that the two learned representations are complementary.
Revised: Tablealso contains results for latent space combination, which con ﬁ rm that the two learned representations are complementary.

Original: Their combination yields gains of up to 9% on T1K+.
Revised: Their combination yields gains of up to 9% on T1K+.

Original: As a sanity check we also include the latent space combination of two networks separately trained with Color Jitter.
Revised: As a sanity check we also include the latent space combination of two networks separately trained with Color Jitter.

Original: This provides a small gain on some datasets, but yields significantly inferior results than LSC.
Revised: This provides a small gain on some datasets, but yields signi ﬁ cantly inferior results than LSC.

Original: AUGMENTATION

Original: [Table]
Revised: [Table]

Original: FRAMEWORK AUGMENTATION
Revised: FRAMEWORK AUGMENTATION

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  G ENERALITY AND L IMITATIONS OF P LANCKIAN J ITTER
Revised: 4. G ENERALITY AND L IMITATIONS OF P LANCKIAN J ITTER

Original: To show that our approach is generally applicable to self-supervised methods exploiting color augmentations, we report in Table 4 experiments using SimCLR, Barlow Twins, and the more recent
Revised: To show that our approach is generally applicable to self-supervised methods exploiting color augmentations, we report in Table 4 experiments using SimCLR, Barlow Twins, and the more recent

Original: VicReg Bardes et al.
Revised: VicReg Bardes et al.

Original: Independently of the model, Latent Space Combination consistently achieves the best results on both datasets.
Revised: Independently of the model, Latent Space Combination consistently achieves the best results on both datasets.

Original: A drawback of Planckian Jitter is the quality reduction of shape and texture representations, because the extreme color transformation of the standard Color Jitter force the network to solve the contrastive learning problem mainly using shape/texture information.
Revised: A drawback of Planckian Jitter is the quality reduction of shape and texture representations, because the extreme color transformation of the standard Color Jitter force the network to solve the contrastive learning problem mainly using shape/texture information.

Original: As we have shown, this problem can be addressed by exploiting their complimentary nature.
Revised: As we have shown, this problem can be addressed by exploiting their complimentary nature.

Original: Secondly, our current latent space combination requires the training of two separate backbones, which will also learn partially-overlapping features.
Revised: Secondly, our current latent space combination requires the training of two separate backbones, which will also learn partially-overlapping features.

Original: A training scenario with both augmentations simultaneously in a single network while reserving part of the latent space for each augmentation could be pursued to address this limitation.
Revised: A training scenario with both augmentations simultaneously in a single network while reserving part of the latent space for each augmentation could be pursued to address this limitation.

Original: Finally, object-specific augmentations that also take into account shadows, the type of reflectance, secondary light sources, inter-reflections, shadows, etc, could lead to further improvements.
Revised: Finally, object-speci ﬁ c augmentations that also take into account shadows, the type of re ﬂ ectance, secondary light sources, inter-re ﬂ ections, shadows, etc, could lead to further improvements.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Existing research on self-supervised learning mainly focuses on tasks where color is not a decisive feature, and consequently exploits data augmentation procedures that negatively affect colorsensitive tasks.
Revised: Existing research on self-supervised learning mainly focuses on tasks where color is not a decisive feature, and consequently exploits data augmentation procedures that negatively affect colorsensitive tasks.

Original: We propose an alternative color data augmentation, called Planckian Jitter, that is based on the physical properties of light.
Revised: We propose an alternative color data augmentation, called Planckian Jitter, that is based on the physical properties of light.

Original: Our experiments demonstrate its positive effects on a wide variety of tasks where the intrinsic color of the objects (related to their reflectance) is crucial for discrimination, while the illumination source is not.
Revised: Our experiments demonstrate its positive effects on a wide variety of tasks where the intrinsic color of the objects (related to their re ﬂ ectance) is crucial for discrimination, while the illumination source is not.

Original: We also proposed exploiting both color and shape information by concatenating features learned with different modalities of self-supervision, leading to significant overall improvements in learned representations.
Revised: We also proposed exploiting both color and shape information by concatenating features learned with different modalities of self-supervision, leading to signi ﬁ cant overall improvements in learned representations.

Original: Planckian Jitter can be easily incorporated into any self-supervised learning pipeline based on data augmentations, as shown by our results demonstrating improved performance for three self-supervised learning models.
Revised: Planckian Jitter can be easily incorporated into any self-supervised learning pipeline based on data augmentations, as shown by our results demonstrating improved performance for three self-supervised learning models.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The code of the Planckian Jitter data augmentation procedure, written in MATLAB and PyTorch 1.7.0, will be made available upon acceptance.
Revised: The code of the Planckian Jitter data augmentation procedure, written in MATLAB and PyTorch 1.7.0, will be made available upon acceptance.

Original: The training runs have been performed using Pytorch in combination with Pytorch Lightning Bolt framework, which provides an implementation of SimSiam methodology for backbone contrastive training.
Revised: The training runs have been performed using Pytorch in combination with Pytorch Lightning Bolt framework, which provides an implementation of SimSiam methodology for backbone contrastive training.

Original: The model has been trained using CUDA deterministic, and random seed set to 1234 .
Revised: The model has been trained using CUDA deterministic, and random seed set to 1234 .

Original: All datasets used for the training and fine-tuning are publicly available.
Revised: All datasets used for the training and ﬁ ne-tuning are publicly available.

Original: Only the CUB200 dataset has been pre-processed, by cropping each image using the given bounding box values, available alongside each image in the annotations files.
Revised: Only the CUB200 dataset has been pre-processed, by cropping each image using the given bounding box values, available alongside each image in the annotations ﬁ les. A CKNOWLEDGMENTS

Revised: We acknowledge the support from the Spanish Government funding for projects

Revised: PID2019-104174GB-I00, TED2021-132513B-I00, the support from the European Commission under the Horizon 2020 Programme, grant RYC2021-032765-I funded by

Revised: MCIN/AEI/10.13039/501100011033 and by European Union NextGenerationEU/PRTR and grant number 951911 (AI4Media).