Original: E QUIVARIANT S ELF -S UPERVISED L EARNING : E NCOURAGING E QUIVARIANCE IN R EPRESENTATIONS Anonymous authors
Revised: E QUIVARIANT C ONTRASTIVE L EARNING Rumen Dangovski MIT EECS rumenrd@mit.edu Li Jing

Original: Paper under double-blind review
Revised: Facebook AI Research ljng@fb.com Charlotte Loh MIT EECS cloh@mit.edu Seungwook Han

Revised: MIT-IBM Watson AI Lab sh3264@columbia.edu

Revised: Akash Srivastava

Revised: MIT-IBM Watson AI Lab akashsri@mit.edu

Revised: Brian Cheung

Revised: MIT CSAIL & BCS cheungb@mit.edu

Revised: Pulkit Agrawal

Revised: MIT CSAIL pulkitag@mit.edu

Revised: Marin Soljaˇci´c

Revised: MIT Physics soljacic@mit.edu

Original: A BSTRACT
Revised: A BSTRACT

Original: In state-of-the-art self-supervised learning (SSL) pre-training produces semantically good representations by encouraging them to be invariant under meaningful transformations prescribed from human knowledge.
Revised: In state-of-the-art self-supervised learning (SSL) pre-training produces semantically good representations by encouraging them to be invariant under meaningful transformations prescribed from human knowledge.

Original: In fact, the property of invariance is a trivial instance of a broader class called equivariance, which can be intuitively understood as the property that representations transform according to the way the inputs transform.
Revised: In fact, the property of invariance is a trivial instance of a broader class called equivariance, which can be intuitively understood as the property that representations transform according to the way the inputs transform.

Original: Here, we show that rather than using only invariance, pre-training that encourages non-trivial equivariance to some transformations, while maintaining invariance to other transformations, can be used to improve the semantic quality of representations.
Revised: Here, we show that rather than using only invariance, pre-training that encourages non-trivial equivariance to some transformations, while maintaining invariance to other transformations, can be used to improve the semantic quality of representations.

Original: Speciﬁcally, we extend popular SSL methods to a more general framework which we name Equivariant SelfSupervised Learning (E-SSL).
Revised: Speciﬁcally, we extend popular SSL methods to a more general framework which we name Equivariant SelfSupervised Learning (E-SSL).

Original: In E-SSL, a simple additional pre-training objective encourages equivariance by predicting the transformations applied to the input.
Revised: In E-SSL, a simple additional pre-training objective encourages equivariance by predicting the transformations applied to the input.

Original: We demonstrate E-SSL’s effectiveness empirically on several popular computer vision benchmarks.
Revised: We demonstrate E-SSL’s effectiveness empirically on several popular computer vision benchmarks, e.g. improving SimCLR to 72.5% linear probe accuracy on ImageNet.

Original: Furthermore, we demonstrate usefulness of E-SSL for applications beyond computer vision; in particular, we show its utility on regression problems in photonics science.
Revised: Furthermore, we demonstrate usefulness of E-SSL for applications beyond computer vision; in particular, we show its utility on regression problems in photonics science.

Revised: Our code, datasets and pre-trained models are available at https://github.com/rdangovs/essl to aid further research in E-SSL.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Human knowledge about what makes a good representation and the abundance of unlabeled data has enabled the learning of useful representations via self-supervised learning (SSL) pretext tasks.
Revised: Human knowledge about what makes a good representation and the abundance of unlabeled data has enabled the learning of useful representations via self-supervised learning (SSL) pretext tasks.

Original: State-of-the-art SSL methods encourage the representations not to contain information about the way the inputs are transformed, i.e. to be invariant to a set of manually chosen transformations.
Revised: State-of-the-art SSL methods encourage the representations not to contain information about the way the inputs are transformed, i.e. to be invariant to a set of manually chosen transformations.

Original: One such method is contrastive learning, which sets up a binary classiﬁcation problem to learn invariant features.
Revised: One such method is contrastive learning, which sets up a binary classiﬁcation problem to learn invariant features.

Original: Given a set of datapoints (say images), different transformations of the same data point constitute positive examples, whereas transformations of other data points constitute the negatives (He et al., 2020; Chen et al., 2020).
Revised: Given a set of data points (say images), different transformations of the same data point constitute positive examples, whereas transformations of other data points constitute the negatives (He et al., 2020; Chen et al., 2020).

Original: Beyond contrastive learning, many SSL methods also rely on learning representations by encouraging invariance (Grill et al., 2020; Chen & He, 2021;
Revised: Beyond contrastive learning, many SSL methods also rely on learning representations by encouraging invariance (Grill et al., 2020; Chen & He, 2021;

Original: Caron et al., 2021; Zbontar et al., 2021).
Revised: Caron et al., 2021; Zbontar et al., 2021).

Original: Here, we refer to such methods as Invariant-SSL (I-SSL).
Revised: Here, we refer to such methods as Invariant-SSL (I-SSL).

Original: The natural question in I-SSL is to what transformations should the representations be insensitive (Chen et al., 2020; Tian et al., 2020; Xiao et al., 2020).
Revised: The natural question in I-SSL is to what transformations should the representations be insensitive (Chen et al., 2020; Tian et al., 2020; Xiao et al., 2020).

Original: Chen et al.
Revised: Chen et al.

Original: (2020) highlighted the importance of transformations and empirically evaluated which transformations are useful for contrastive learning (e.g., see Figure 5 in their paper).
Revised: (2020) highlighted the importance of transformations and empirically evaluated which transformations are useful for contrastive learning (e.g., see Figure 5 in their paper).

Original: Some transformations, such as four-fold rotations , despite preserving semantic information, were shown to be harmful for contrastive learning.
Revised: Some transformations, such as four-fold rotations , despite preserving semantic information, were shown to be harmful for contrastive learning.

Original: This does not mean that four-fold rotations are not useful for I-SSL at all.
Revised: This does not mean that four-fold rotations are not useful for I-SSL at all.

Original: In fact, predicting four-fold rotations is a good proxy task for evaluating the representations produced with contrastive learning (Reed et al., 2021).
Revised: In fact, predicting four-fold rotations is a good proxy task for evaluating the representations produced with contrastive learning (Reed et al., 2021).

Original: Furthermore, instead of being insensitive to rotations (invariance), training a neural network to predict them, i.e. to be sensitive to four-fold rotations, results in good image representations (Gidaris et al., 2018; 2019).
Revised: Furthermore, instead of being insensitive to rotations (invariance), training a neural network to predict them, i.e. to be sensitive to four-fold rotations, results in good image representations (Gidaris et al., 2018; 2019).

Original: These results indicate that the choice of making features sensitive or insensitive to a particular group of transformations can have a substantial effect on the performance of downstream tasks.
Revised: These results indicate that the choice of making features sensitive or insensitive to a particular group of transformations can have a substantial effect on the performance of downstream tasks.

Original: However, the prior work in SSL has exclusively focused on being either entirely insensitive (Grill et al., 2020; Chen & He, 2021; Caron et al., 2021; Zbontar et al., 2021) or sensitive (Agrawal et al., 2015; Doersch et al., 2015; Zhang et al., 2016; Noroozi &
Revised: However, the prior work in SSL has exclusively focused on being either entirely insensitive (Grill et al., 2020; Chen & He, 2021; Caron et al., 2021; Zbontar et al., 2021) or sensitive (Agrawal et al., 2015; Doersch et al., 2015; Zhang et al., 2016; Noroozi &

Original: Favaro, 2016; Gidaris et al., 2018) to a set of transformations.
Revised: Favaro, 2016; Gidaris et al., 2018) to a set of transformations.

Original: In particular, the I-SSL literature has proposed to simply remove transformations that hurt performance when applied as invariance.
Revised: In particular, the I-SSL literature has proposed to simply remove transformations that hurt performance when applied as invariance.

Original: [Figure] To understand how sensitivity/ insensitivity to a particular transformation affects the resulting features, we ran a series of experiments summarized in Figure 1.
Revised: To understand how sensitivity/ insensitivity to a particular transformation affects the resulting features, we ran a series of experiments summarized in Figure 1.

Original: We trained and tested a simple I-SSL baseline, SimCLR (Chen et al., 2020), on CIFAR-10 using only the random resized cropping transformation (solid yellow line).
Revised: We trained and tested a simple I-SSL baseline, SimCLR (Chen et al., 2020), on CIFAR-10 using only the random resized cropping transformation (solid yellow line).

Original: The test accuracy is calculated as the retrieval accuracy of a k-nearest neighbors (kNN) classiﬁer with a memory bank consisting of the representations on the training set obtained after pre-training for 800 epochs.
Revised: The test accuracy is calculated as the retrieval accuracy of a k-nearest neighbors (kNN) classiﬁer with a memory bank consisting of the representations on the training set obtained after pre-training for 800 epochs.

Original: Next, in addition to being invariant to resized cropping, we additionally encouraged the model to be either sensitive (shown in pink) or insensitive (shown in blue) to a second transformation.
Revised: Next, in addition to being invariant to resized cropping, we additionally encouraged the model to be either sensitive (shown in pink) or insensitive (shown in blue) to a second transformation.

Original: We encourage insensitivity by adding the transformation to the
Revised: We encourage insensitivity by adding the transformation to the

Original: SimCLR data augmentation, and sensitivity by predicting it (see Section 4).
Revised: SimCLR data augmentation, and sensitivity by predicting it (see Section 4).

Original: We varied the choice of this second transformation.
Revised: We varied the choice of this second transformation.

Original: We found that for some transformations, such as horizontal ﬂips and grayscale , insenstivity results in better features, but is detrimental for transformations, such as fourfold rotations , vertical ﬂips , 2x2 jigsaws ( 4!
Revised: We found that for some transformations, such as horizontal ﬂips and grayscale , insenstivity results in better features, but is detrimental for transformations, such as fourfold rotations , vertical ﬂips , 2x2 jigsaws ( 4!

Original: = 24 classes), four-fold Gaussian blurs (4 levels of blurring) and color inversions .
Revised: = 24 classes), four-fold Gaussian blurs (4 levels of blurring) and color inversions .

Original: When we encourage sensitivity to these transformations, the trend is reversed.
Revised: When we encourage sensitivity to these transformations, the trend is reversed.

Original: In summary, we observe that if invariance to a particular transformation hurts feature learning, then imposing sensitivity to the same transformation may improve performance.
Revised: In summary, we observe that if invariance to a particular transformation hurts feature learning, then imposing sensitivity to the same transformation may improve performance.

Original: This leads us to conjecture that instead of choosing the features to be only invariant or only sensitive as done in prior work, it may be possible to learn better features by imposing invariance to certain transformations (e.g., cropping) and sensitivity to other transformations (e.g., four-fold transformations).
Revised: This leads us to conjecture that instead of choosing the features to be only invariant or only sensitive as done in prior work, it may be possible to learn better features by imposing invariance to certain transformations (e.g., cropping) and sensitivity to other transformations (e.g., four-fold transformations).

Original: The concepts of sensitivity and insensitivity are both captured by the mathematical idea of equivariance (Agrawal et al., 2015; Jayaraman & Grauman, 2015; Cohen & Welling, 2016).
Revised: The concepts of sensitivity and insensitivity are both captured by the mathematical idea of equivariance (Agrawal et al., 2015; Jayaraman & Grauman, 2015; Cohen & Welling, 2016).

Original: Let G be a group of transformations.
Revised: Let G be a group of transformations.

Original: For any g ∈ G let T g ( x ) denote the function with which g transforms an input image x .
Revised: For any g ∈ G let T g ( x ) denote the function with which g transforms an input image x .

Original: For instance, if G is the group of four-fold rotations then T g ( x ) rotates the image x by a multiple of π/ 2 .
Revised: For instance, if G is the group of four-fold rotations then T g ( x ) rotates the image x by a multiple of π/ 2 .

Original: Let f be the encoder network that computes feature representation, f ( x ) .
Revised: Let f be the encoder network that computes feature representation, f ( x ) .

Original: I-SSL encourages the property of “invariance to G ,” which states f ( T g ( x ))
Revised: I-SSL encourages the property of “invariance to G ,” which states f ( T g ( x ))

Original: =
Revised: =

Original: f
Revised: f

Original: ( x ) , i.e. the output representation, f ( x ) , does not vary with T g .
Revised: ( x ) , i.e. the output representation, f ( x ) , does not vary with T g .

Original: Equivariance, a generalization of invariance, is deﬁned as, ∀ x : f ( T g ( x ))
Revised: Equivariance, a generalization of invariance, is deﬁned as, ∀ x : f ( T g ( x ))

Original: =
Revised: =

Original: T (cid:48) g ( f ( x )) , where T (cid:48) g is a ﬁxed transformation (i.e., without any parameters).
Revised: T (cid:48) g ( f ( x )) , where T (cid:48) g is a ﬁxed transformation (i.e., without any parameters).

Original: Intuitively, equivariance encourages the feature representation to change in a well deﬁned manner to the transformation applied to the input.
Revised: Intuitively, equivariance encourages the feature representation to change in a well deﬁned manner to the transformation applied to the input.

Original: Thus, invariance is a trivial instance of equivariance, where
Revised: Thus, invariance is a trivial instance of equivariance, where

Original: T (cid:48) g is the identity function, i.e. T (cid:48) g ( f ( x ))
Revised: T (cid:48) g is the identity function, i.e. T (cid:48) g ( f ( x ))

Original: =
Revised: =

Original: f
Revised: f

Original: ( x ) .
Revised: ( x ) .

Original: While there are many possible choices for T (cid:48) g (Cohen & Welling, 2016; Bronstein et al., 2021), I-SSL uses only the trivial choice that encourages f to be insensitive to G .
Revised: While there are many possible choices for T (cid:48) g (Cohen & Welling, 2016; Bronstein et al., 2021), I-SSL uses only the trivial choice that encourages f to be insensitive to G .

Original: In contrast, if T (cid:48) g is not the identity, then f will be sensitive to G and we say that the “equivariance to G ” will be non-trivial.
Revised: In contrast, if T (cid:48) g is not the identity, then f will be sensitive to G and we say that the “equivariance to G ” will be non-trivial.

Original: Therefore, in order to encourage potentially more useful equivariance properties, we generalize SSL to an Equivariant Self-Supervised Learning (E-SSL) framework.
Revised: Therefore, in order to encourage potentially more useful equivariance properties, we generalize SSL to an Equivariant Self-Supervised Learning (E-SSL) framework.

Original: In our experiments on standard computer vision data, such as the small-scale CIFAR-10 (Torralba et al., 2008; Krizhevsky, 2009) and the large-scale ImageNet (Deng et al., 2009), we show that extending I-SSL to E-SSL by also predicting four-fold rotations improves the semantic quality of the representations.
Revised: In our experiments on standard computer vision data, such as the small-scale CIFAR-10 (Torralba et al., 2008; Krizhevsky, 2009) and the large-scale ImageNet (Deng et al., 2009), we show that extending I-SSL to E-SSL by also predicting four-fold rotations improves the semantic quality of the representations.

Original: We show that this approach works for other transformations too, such as vertical ﬂips, 2x2 jigsaws, four-fold Gaussian blurs and color inversions, but focus on four-fold rotations as the most promising improvement we obtain with initial E-SSL experiments in Figure 1.
Revised: We show that this approach works for other transformations too, such as vertical ﬂips, 2x2 jigsaws, four-fold Gaussian blurs and color inversions, but focus on four-fold rotations as the most promising improvement we obtain with initial E-SSL experiments in Figure 1.

Original: We also note that the applications of E-SSL in this paper are task speciﬁc, meaning that the representations from E-SSL may work best for a particular downstream task that beneﬁts from equivariance dictated by the available data.
Revised: We also note that the applications of E-SSL in this paper are task speciﬁc, meaning that the representations from E-SSL may work best for a particular downstream task that beneﬁts from equivariance dictated by the available data.

Original: E-SSL can be further extended to applications in science; in particular, we focus on predictive tasks using (unlabelled and labelled) data collected via experiments or simulations.
Revised: E-SSL can be further extended to applications in science; in particular, we focus on predictive tasks using (unlabelled and labelled) data collected via experiments or simulations.

Original: The downstream tasks in prediction problems in science are often ﬁxed and can be aided by incorporating scientiﬁc insights.
Revised: The downstream tasks in prediction problems in science are often ﬁxed and can be aided by incorporating scientiﬁc insights.

Original: Here, we also explore the generality of E-SSL beyond computer vision, on a different application: regression problems in photonics science and demonstrate examples where E-SSL is effective over I-SSL.
Revised: Here, we also explore the generality of E-SSL beyond computer vi- [Figure] sion, a : regression problems in photonics science and demonstrate examples where E-SSL is effective over I-SSL.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We introduce E-SSL, a generalization of popular SSL methods that highlights the complementary nature of invariance and equivariance.
Revised: • We introduce E-SSL, a generalization of popular SSL methods that highlights the complementary nature of invariance and equivariance.

Original: To our knowledge, we are the ﬁrst to create a method that beneﬁts from such complementarity.
Revised: To our knowledge, we are the ﬁrst to create a method that beneﬁts from such complementarity.

Original: • We improve state-of-the-art SSL methods on CIFAR-10 and ImageNet by encouraging equivariance to four-fold rotations.
Revised: • We improve state-of-the-art SSL methods on CIFAR-10 and ImageNet by encouraging equivariance to four-fold rotations.

Original: We also show that E-SSL is more general and works for many other transformations, previously unexplored in related works.
Revised: We also show that E-SSL is more general and works for many other transformations, previously unexplored in related works.

Original: • We demonstrate the usefulness of E-SSL beyond computer vision with experiments on regression problems in photonics science.
Revised: • We demonstrate the usefulness of E-SSL beyond computer vision with experiments on regression problems in photonics science.

Original: We also show that our method works both for ﬁnite and inﬁnite groups.
Revised: We also show that our method works both for ﬁnite and inﬁnite groups.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Revised: In Section 2 we elaborate on related work.

Original: In Section 3 we introduce our experimental method for E-SSL.
Revised: In Sectionwe introduce our experimental method for E-SSL.

Original: In Section 4 we present our main experiments in computer vision.
Revised: In Section 4 we present our main experiments in computer vision.

Original: In Section2 we elaborate on related work. In Section 5 provide a discussion around our work that extends our study beond computer vision. In Section 6 we conclude and point to future work.
Revised: In Section 5 provide a discussion around our work that extends our study beond computer vision. Beginning from Appendix A, we provide more details behind our ﬁndings and discuss several potential avenues of future work.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: To encourage non-trivial equivariance, we observe that a simple task that predicts the synthetic transformation applied to the input, works well and improves I-SSL already; some prediction tasks create representations that can be transferred to other tasks of interest, such as classiﬁcation, object detection and segmentation.
Revised: To encourage non-trivial equivariance, we observe that a simple task that predicts the synthetic transformation applied to the input, works well and improves I-SSL already; some prediction tasks create representations that can be transferred to other tasks of interest, such as classiﬁcation, object detection and segmentation.

Original: While prediction tasks alone have been realized successfully before in
Revised: While prediction tasks alone have been realized successfully before in

Original: SSL (Agrawal et al., 2015; Doersch et al., 2015; Zhang et al., 2016; Misra et al., 2016; Noroozi & Favaro, 2016; Zamir et al., 2016; Lee et al., 2017; Mundhenk et al., 2018; Gidaris et al., 2018;
Revised: SSL (Agrawal et al., 2015; Doersch et al., 2015; Zhang et al., 2016; Misra et al., 2016; Noroozi & Favaro, 2016; Zamir et al., 2016; Lee et al., 2017; Mundhenk et al., 2018; Gidaris et al., 2018;

Original: Zhang et al., 2019; Zhang, 2020), to our knowledge we are the ﬁrst to combine simple predictive objectives of synthetic transformations with I-SSL, and successfully improve the semantic quality of representations.
Revised: Zhang et al., 2019; Zhang, 2020), to our knowledge we are the ﬁrst to combine simple predictive objectives of synthetic transformations with I-SSL, and successfully improve the semantic quality of representations.

Original: We found that the notion of equivariance captures the generality of our method.
Revised: We found that the notion of equivariance captures the generality of our method.

Original: To improve representations with pretext tasks, Gidaris et al.
Revised: To improve representations with pretext tasks, Gidaris et al.

Original: (2018) use four-fold rotations prediction as a pretext task for learning useful visual representations via a new model named RotNet.
Revised: (2018) use four-fold rotations prediction as a pretext task for learning useful visual representations via a new model named RotNet.

Original: Feng et al.
Revised: Feng et al.

Original: (2019) learn decoupled representations: one part trained with four-fold rotations prediction and another with non-parametric instance discrimination (Wu et al., 2018) and invariance to four-fold rotations.
Revised: (2019) learn decoupled representations: one part trained with four-fold rotations prediction and another with non-parametric instance discrimination (Wu et al., 2018) and invariance to four-fold rotations.

Original: Yamaguchi et al.
Revised: Yamaguchi et al.

Original: (2021) use a joint training objective between four-fold rotations prediction and image enhancement prediction.
Revised: (2021) use a joint training objective between four-fold rotations prediction and image enhancement prediction.

Original: Xiao et al.
Revised: Xiao et al.

Original: (2020) propose to learn representations as follows: for each atomic augmentation from the contrastive learning’s augmentation policy, they leave it out and project to a new space on which I-SSL encourages invariance to all augmentations, but the leftout one.
Revised: (2020) propose to learn representations as follows: for each atomic augmentation from the contrastive learning’s augmentation policy, they leave it out and project to a new space on which I-SSL encourages invariance to all augmentations, but the leftout one.

Original: The resulting representation could either be a concatenation of all projected left-out views’ representations, or the representation in the shared space, before the individual projections.
Revised: The resulting representation could either be a concatenation of all projected left-out views’ representations, or the representation in the shared space, before the individual projections.

Original: Our  method differs from the above contributions in that E-SSL is the only hybrid framework that encourages both insensitive representations for some transformations and sensitive representations for others and does not require representations to be sensitive and insensitive to a particular transformation at the same time.
Revised: Our [Figure] method differs from the above contributions in that E-SSL is the only hybrid framework that encourages both insensitive representations for some transformations and sensitive representations for others and does not require representations to be sensitive and insensitive to a particular transformation at the same time.

Original: Thus, what distinguished our work is the complementary nature of invariance and equivariance for multiple transformations, including ﬁnite and inﬁnite groups.
Revised: Thus, what distinguishes our work is the complementary nature of invariance and equivariance for multiple transformations, including ﬁnite and inﬁnite groups.

Original: To obtain performance gains from transformations, Tian et al.
Revised: To obtain performance gains from transformations, Tian et al.

Original: (2020) study which transformations are the best for contrastive learning through the lens of mutual information.
Revised: (2020) study which transformations are the best for contrastive learning through the lens of mutual information.

Original: Reed et al.
Revised: Reed et al.

Original: (2021) use four-fold rotations prediction as an evaluation measure to tune optimal augmentations for contrastive learning.
Revised: (2021) use four-fold rotations prediction as an evaluation measure to tune optimal augmentations for contrastive learning.

Original: Wang & Qi (2021) use strong augmentations to improve contrastive learning by matching the distributions of strongly and weakly augmented views’ representation similarities to a 
Revised: Wang & Qi (2021) use strong augmentations to improve contrastive learning by matching the distributions of strongly and weakly augmented views’ representation similarities to a memory bank.

Original: [Figure]
Revised: Wang et al.

Original: memory bank.
Revised: (2021) provide an effective way to bridge transformation-insensitive and transformation-sensitive approaches in self-superived learning methods via residual relaxation.

Original: A growing body of work encourages invariance to domain agnostic transformations (Tamkin et al., 2021; Lee et al., 2021; Verma et al., 2021) or strengthens invariance with regularization (Foster et al., 2021).
Revised: A growing body of work encourages invariance to domain agnostic transformations (Tamkin et al., 2021; Lee et al., 2021; Verma et al., 2021) or strengthens invariance with regularization (Foster et al., 2021).

Original: Our framework is different from the above works, because we work with transformations that encourage equivariance beyond invariance.
Revised: Our framework is different from the above works, because we work with transformations that encourage equivariance beyond invariance.

Original: To understand and improve equivariant properties of neural networks, Lenc & Vedaldi (2015) study emerging equivariant properties of neural networks and (Cohen & Welling, 2016;
Revised: To understand and improve equivariant properties of neural networks, Lenc & Vedaldi (2015) study emerging equivariant properties of neural networks and (Cohen & Welling, 2016;

Original: Bronstein et al., 2021) construct equivariant neural networks.
Revised: Bronstein et al., 2021) construct equivariant neural networks.

Original: In contrast, our work does not enforce strict equivariance, but only encourages equivariant properties for the encoder network through the choice of the loss function.
Revised: In contrast, our work does not enforce strict equivariance, but only encourages equivariant properties for the encoder network through the choice of the loss function.

Original: While strict equivariance is concerned with groups, some of the transformations, such as random resized cropping and Gaussian blurs, may not even be groups, but they could still be analyzed in the E-SSL framework.
Revised: While strict equivariance is concerned with groups, some of the transformations, such as random resized cropping and Gaussian blurs, may not even be groups, but they could still be analyzed in the E-SSL framework.

Original: Thus, ours is a ﬂexible framework, which allows us to consider a variety of transformations and how the encoder might exhibit equivariant properties to them.
Revised: Thus, ours is a ﬂexible framework, which allows us to consider a variety of transformations and how the encoder might exhibit equivariant properties to them.

Original:  M ETHOD
Revised:  M ETHOD

Original: Our method is designed to test our primary conjecture that a hybrid approach of sensitive and insensitive representations learns better features.
Revised: Our method is designed to test our primary conjecture that a hybrid approach of sensitive and insensitive representations learns better features.

Original: Surprisingly, this hybrid approach is not yet present in SSL, as Figure 2 illustrates.
Revised: Surprisingly, this hybrid approach is not yet present in SSL, as Figure 2 illustrates.

Original: In this ﬁgure, we can view transformations in SSL as “levers.”
Revised: In this ﬁgure, we can view transformations in SSL as “levers.”

Original: Each downstream task has an optimal conﬁguration of the levers, which should be tuned in the SSL objective: left for insensitive and right for sensitive representations.
Revised: Each downstream task has an optimal conﬁguration of the levers, which should be tuned in the SSL objective: left for insensitive and right for sensitive representations.

Original: E.g., make representations insensitive to horizontal ﬂips and grayscale and sensitive to four-fold rotations, vertical ﬂips, 2x2 jigsaws, Gaussian blurs or color inversions.
Revised: E.g., make representations insensitive to horizontal ﬂips and grayscale and sensitive to four-fold rotations, vertical ﬂips, 2x2 jigsaws, Gaussian blurs or color inversions.

Original: Technically, insensitive and sensitive features correspond to trivial and regular group representations.
Revised: Formally, insensitive and sensitive features correspond to trivial and regular group representations, respectively.

Original: We continue with an effective method to achieve such properties.
Revised: Here, we present an effective method to achieve this control.

Original: Let f ( · ; θ f ) with trainable parameters θ f be a backbone encoder.
Revised: Let f ( · ; θ f ) with trainable parameters θ f be a backbone encoder.

Original: Analogously, let p 1 ( · ; θ p 1 ) be a projector network for the I-SSL loss.
Revised: Analogously, let p 1 ( · ; θ p 1 ) be a projector network for the I-SSL loss.

Original: There might be an extra prediction head and parameters, depending on the objective, which we omit for simiplicity.
Revised: There might be an extra prediction head and parameters, depending on the objective, which we omit for simiplicity.

Original: Let p 2 ( · ; θ p 2 ) be the predictor network for encouraging sensitivity, which we will call “predictor for equivariance.”
Revised: Let p 2 ( · ; θ p 2 ) be the predictor network for encouraging sensitivity, which we will call “predictor for equivariance.”

Original: We share the backbone encoder f jointly for I-SSL and the objective of predicting the transformations from the backbone representations.
Revised: We share the backbone encoder f jointly for I-SSL and the objective of predicting the transformations from the backbone representations.

Original: Let (cid:96) I-SSL be the I-SSL loss and (cid:96) E-SSL be the added E-SSL loss that encourages  sensitivity to a particular transformation.
Revised: Let (cid:96) I-SSL be the I-SSL loss and (cid:96) E-SSL be the added E-SSL loss that encourages [Figure] sensitivity to a particular transformation.

Original: Let the parameter λ be the strength of the E-SSL loss.
Revised: Let the parameter λ be the strength of the E-SSL loss.

Original: The optimization objective for an image x with views { x (cid:48) } in the batch is given as follows
Revised: The optimization objective for an image x with views { x (cid:48) } in the batch is given as follows

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) E-SSL (the expectation in the second summand) can take either one or all of the views, but we take only one for simplicity.
Revised: where (cid:96) E-SSL (the expectation in the second summand) can take either one or all of the views, but we take only one for simplicity.

Original: The goal of (cid:96) E-SSL is to predict g from the representation p 2 ( f (  g ( x (cid:48) )) ; θ f  ) , which encourages equivariance to the group of transformations G .
Revised: The goal of (cid:96) E-SSL is to predict g from the representation p 2 ( f ( T g ( x (cid:48) ); θ f ); θ p 2 ) , which encourages equivariance to the group of transformations G .

Original: The
Revised: The

Original: PredictionLoss could be either a cross entropy loss for ﬁnite groups or L1/ MSE loss for inﬁnite groups.
Revised: PredictionLoss could be either a cross entropy loss for ﬁnite groups or L1/ MSE loss for inﬁnite groups.

Original: In practice we replace the expectation with an unbiased estimate.
Revised: In practice we replace the expectation with an unbiased estimate.

Original: Most of our experiments in this paper focus on ﬁnite groups, but we show one example for an inﬁnite group in Appendix E.
Revised: Most of our experiments in this paper focus on ﬁnite groups, but we show one example for an inﬁnite group in Appendix F.

Original: E-SSL can be constructed for any semantically meaningful transformation (see for example, Figure 1).
Revised: E-SSL can be constructed for any semantically meaningful transformation (see for example, Figure 1).

Original: From Figure 1 we choose four-fold rotations as the most promising transformation and we ﬁx it for the upcoming section.
Revised: From Figure 1 we choose four-fold rotations as the most promising transformation and we ﬁx it for the upcoming section.

Original: As a minor motivation, we also present empirical results about[Figure] the similarities between four-fold rotations prediction and I-SSL in Appendix B. In particular, both tasks beneﬁt from the same data augmentation.
Revised: As a minor motivation, we also present empirical results about the similarities between four-fold rotations prediction and I-SSL in Appendix C. In particular, both tasks beneﬁt from the same data augmentation.

Original: Figure 3 sketches how our construction works for predicting four-fold rotations.
Revised: Figure 3 sketches how our construction works for predicting four-fold rotations.

Original: In particular,in Equation 1 we sample each of the 4 possible rotations uniformly and use the cross entropy loss for the PredictionLoss .
Revised: In particular, we sample each of the 4 possible rotations uniformly and use the cross entropy loss for the PredictionLoss in Equation 1.

Original: What transformations could work for E-SSL?
Revised: What transformations could work for E-SSL?

Original: A common property of the successful transformations we have studied up to this point is that they form groups in the mathematical sense, i.e.
Revised: A common property of the successful transformations we have studied up to this point is that they form groups in the mathematical sense, i.e.

Original: ( i ) each transformation is invertible, ( ii ) composition of two transformations is part of the set of transformations and ( iii ) compositions are associative.
Revised: ( i ) each transformation is invertible, ( ii ) composition of two transformations is part of the set of transformations and ( iii ) compositions are associative.

Original: In this paper, we encourage equivariance to a group of transformation by predicting them.
Revised: In this paper, we encourage equivariance to a group of transformation by predicting them.

Original: This does not guarantee that the encoder we learn will be strictly equivariant to the group.
Revised: This does not guarantee that the encoder we learn will be strictly equivariant to the group.

Original: In practice we observe that invariance and equivariance is well encouraged by the training objectives we use (see
Revised: In practice we observe that invariance and equivariance is well encouraged by the training objectives we use (see

Original: Appendix C.3 for detailed analysis).
Revised: Appendix D.3 for detailed analysis).

Original: In fact, even strict equivariance is possible, i.e. there exists an encoder that is non-trivially equivariant, under a reasonable assumption which is formulated as follows.
Revised: In fact, even strict equivariance is possible, i.e. there exists an encoder that is non-trivially equivariant, under a reasonable assumption which is formulated as follows.

Original: Let X be the set of all images.
Revised: Let X be the set of all images.

Original: Let G be a group whose elements g ∈ G transform X via the function T g : X → X .
Revised: Let G be a group whose elements g ∈ G transform X via the function T g : X → X .

Original: Let X (cid:48)
Revised: Let X (cid:48)

Original: = { T g ( x ) | g ∈ G, x ∈ X } be the set of all transformed images.
Revised: = { T g ( x ) | g ∈ G, x ∈ X } be the set of all transformed images.

Original: Let f ( · ; θ ): X (cid:48)
Revised: Let f ( · ; θ ): X (cid:48)

Original: → S be an encoder network that we learn with parameters θ .
Revised: → S be an encoder network that we learn with parameters θ .

Original: We write f ( · ) ≡ f ( · ; θ ) for simplicity.
Revised: We write f ( · ) ≡ f ( · ; θ ) for simplicity.

Original: Finally, let S =
Revised: Finally, let S =

Original: { f ( x (cid:48) ) | x (cid:48) ∈ X } be the set of all representations of the images in X (cid:48) .
Revised: { f ( x (cid:48) ) | x (cid:48) ∈ X } be the set of all representations of the images in X (cid:48) .

Original: The following is our statement.
Revised: The following is our statement.

Original: Proposition 1 (Non-trivial Equivariance) .
Revised: Proposition 1 (Non-trivial Equivariance) .

Original: Given T g : X (cid:48)
Revised: Given T g : X (cid:48)

Original: → X (cid:48) for the group G , there exists an encoder f : X (cid:48)
Revised: → X (cid:48) for the group G , there exists an encoder f : X (cid:48)

Original: → S that is non-trivially equivariant to the group G under the assumption that iff ( T g ( x )) =
Revised: → S that is non-trivially equivariant to the group G under the assumption that if

Original: f ( T g (cid:48)

Original: ( x (cid:48) )) then g = g (cid:48) and x = x (cid:48) for all g, g (cid:48) ∈ G and x , x (cid:48) ∈ X.

Original: We defer the proof to Appendix A.
Revised: We defer the proof to Appendix B.

Original: The signiﬁcance of this proof is that it explicitly constructs a nontrivially equivariant encoder network for groups G if the assumption is satisﬁed.
Revised: The signiﬁcance of this proof is that it explicitly constructs a nontrivially equivariant encoder network for groups G if the assumption is satisﬁed.

Original: The intuition of the assumption is that if the representations of two transformed inputs are the same, the inputs should coincide, and likewise the transformations.
Revised: The intuition of the assumption is that if the representations of two transformed inputs are the same, the inputs should coincide, and likewise the transformations.

Original: More formally, this assumption reﬂects the condition when the dataset contains only one element of each group orbit.
Revised: More formally, this assumption reﬂects the condition when the dataset contains only one element of each group orbit.

Original: We speculate that satisfying this assumption is reasonable for the datasets in this work, since we observe a natural setting of the data, e.g. horizontal mirror symmetry in Gpm, and we consider transformations that disturb this natural setting.
Revised: We speculate that satisfying this assumption is reasonable for the datasets in this work, since we observe a natural setting of the data, e.g. horizontal mirror symmetry in Gpm, and we consider transformations that disturb this natural setting.

Original: In Appendix F we also show that E-SSL is crucial for the Flowers-102 dataset for which this assumption might be less clear.
Revised: In Appendix G we also show that E-SSL is crucial for the Flowers-102 dataset for which this assumption might be less clear.

Revised: In Appendix H we also present a natural modiﬁcation of E-SSL for scenarios, where that assumption is violated.

Original: Could other transformations still help?
Revised: Could other transformations still help?

Original: To motivate our work, in Figure 1 we observed additional transformations that could be useful, such as vertical ﬂips, 2x2 jigsaws, four-fold Gaussian blurs and color inversions.
Revised: To motivate our work, in Figure 1 we observed additional transformations that could be useful, such as vertical ﬂips, 2x2 jigsaws, four-fold Gaussian blurs and color inversions.

Original: All of these transformations are groups, except for four-fold Gaussian blurs.
Revised: All of these transformations are groups, except for four-fold Gaussian blurs.

Original: Each element of Gaussian blurs is invertible (de-blurring), but the inverse is not a transformation in the set.
Revised: Each element of Gaussian blurs is invertible (de-blurring), but the inverse is not a transformation in the set.

Original: Interestingly, we observe that four-fold Gaussian blurs still improve the baseline, which means the success of E-SSL may not be limited to groups.
Revised: Interestingly, we observe that four-fold Gaussian blurs still improve the baseline, which means the success of E-SSL may not be limited to groups.

Original: We might also consider combining the prediction of multiple transformations to encourage sensitivity to all of them.
Revised: We might also consider combining the prediction of multiple transformations to encourage sensitivity to all of them.

Original: However, the gains we saw in Figure 1 may not add up when we combine transformations, because they may not be independent.
Revised: However, the gains we saw in Figure 1 may not add up when we combine transformations, because they may not be independent.

Original: The gains may also depend on the transformations that we choose for I-SSL.
Revised: The gains may also depend on the transformations that we choose for I-SSL.

Original: While we see combinations of transformations as promising future work, we focus on a single transformation to make a clear presentation of E-SSL.
Revised: While we see combinations of transformations as promising future work, we focus on a single transformation to make a clear presentation of E-SSL.

Original:  E XPERIMENTS 4. S ETUPS
Revised:  E XPERIMENTS 4. S ETUPS

Original: CIFAR-10 setup.
Revised: CIFAR-10 setup.

Original: We use the CIFAR-10 experimental setup from (Chen & He, 2021).
Revised: We use the CIFAR-10 experimental setup from (Chen & He, 2021).

Original: We consider two simple I-SSL methods: SimCLR (with InfoNCE loss (Oord et al., 2018) and temperature 0.5) and SimSiam (Chen & He, 2021).
Revised: We consider two simple I-SSL methods: SimCLR (with InfoNCE loss (Oord et al., 2018) and temperature 0.5) and SimSiam (Chen & He, 2021).

Original: We were able to obtain baseline results close to those in (Chen & He, 2021).
Revised: We were able to obtain baseline results close to those in (Chen & He, 2021).

Original: The predictor for equivariance takes a smaller crop with size 16x16.
Revised: The predictor for equivariance takes a smaller crop with size 16x16.

Original: We report performance on the standard linear probe.
Revised: We report performance on the standard linear probe.

Original: We tune λ to 0.4 both for SimCLR and SimSiam (full tuning in
Revised: We tune λ to 0.4 both for SimCLR and SimSiam (full tuning in

Original: Table 6 in Appendix C).
Revised: Table 6 in Appendix D).

Original: Remaining experimental details can be found in Appendix C.
Revised: Remaining experimental details can be found in Appendix D.

Original: ImageNet setup.
Revised: ImageNet setup.

Original: We use the original augmentation setting for each method.
Revised: We use the original augmentation setting for each method.

Original: The predictor for equivariance takes a smaller crop with size 96x96.
Revised: The predictor for equivariance takes a smaller crop with size 96x96.

Original: We use a ResNet-50 (He et al., 2016) backbone for each method.
Revised: We use a ResNet-50 (He et al., 2016) backbone for each method.

Original: In terms of optimizer and batch size settings, we follow the standard training recipe for each method.
Revised: In terms of optimizer and batch size settings, we follow the standard training recipe for each method.

Original: For our SimCLR experiments we use a slightly more optimal implementation that uses BYOL’s augmentations (i.e. it includes solarization ), initializes the ResNet with zero
Revised: For our SimCLR experiments we use a slightly more optimal implementation that uses BYOL’s augmentations (i.e. it includes solarization ), initializes the ResNet with zero

Original: BatchNorm weights and uses the InfoNCE loss with temperature 0.2.
Revised: BatchNorm weights and uses the InfoNCE loss with temperature 0.2.

Original: Photonic-crystals setup.
Revised: Photonic-crystals setup.

Original: Photonic crystals (PhC) are periodically-structured materials engineered for wide ranging applications by manipulating light waves (Yablonovitch, 1987; Joannopoulos et al., 2008).
Revised: Photonic crystals (PhC) are periodically-structured materials engineered for wide ranging applications by manipulating light waves (Yablonovitch, 1987; Joannopoulos et al., 2008).

Original: The density-of-states (DOS) is often used as a design metric to engineer the desired properties of these crystals and thus here, we consider the regression task of predicting the DOS of PhCs.
Revised: The density-of-states (DOS) is often used as a design metric to engineer the desired properties of these crystals and thus here, we consider the regression task of predicting the DOS of PhCs.

Original: Examples of this dataset are depicted in Section 5 and further details can be found in Appendix E.
Revised: Examples of this dataset are depicted in Section 5 and further details can be found in Appendix F.

Original: The use of symmetry or invariance knowledge is common in scientiﬁc problems; here, the DOS labels are invariant to several physical transformations of the unit cell, namely, rolling translations (due to its periodicity), operations arising from the symmetry group ( C 4 v ) of the square lattice, i.e. rotations and mirror ﬂips, and refractive scaling.
Revised: The use of symmetry or invariance knowledge is common in scientiﬁc problems; here, the DOS labels are invariant to several physical transformations of the unit cell, namely, rolling translations (due to its periodicity), operations arising from the symmetry group ( C 4 v ) of the square lattice, i.e. rotations and mirror ﬂips, and refractive scaling.

Original: We construct an encoder network comprising of simple convolutional and fully-connected layers (see Appendix E) and create various synthetic datasets to investigate encouragement of equivariance.
Revised: We construct an encoder network comprising of simple convolutional and fully-connected layers (see Appendix F) and create various synthetic datasets to investigate encouragement of equivariance.

Original: After SSL/ E-SSL, we ﬁne-tune the network with L1 loss; for better interpretability of prediction accuracies, we use a relative error metric (Liu et al., 2018; Loh et al., 2021) for evaluation, given by (cid:96) DOS
Revised: After SSL/ E-SSL, we ﬁne-tune the network with L1 loss; for better interpretability of prediction accuracies, we use a relative error metric (Liu et al., 2018; Loh et al., 2021) for evaluation, given by (cid:96) DOS

Original: = ( (cid:80) ω (cid:12)(cid:12)
Revised: = ( (cid:80) ω (cid:12)(cid:12)

Original: DOS pred − DOS (cid:12)(cid:12) ) /
Revised: DOS pred − DOS (cid:12)(cid:12) ) /

Original: ( (cid:80) ω DOS) , reported in (%).
Revised: ( (cid:80) ω DOS) , reported in (%).

Original: We defer the results to Section 5, because of the novelty of the experimental setup.
Revised: We defer the results to Section 5, because of the novelty of the experimental setup.

Original: The predictor p 2 for E-SSL.
Revised: The predictor p 2 for E-SSL.

Original: The predictor is a 2 layer MLP for CIFAR-10 and Photonic-crystals, and a 3 layer MLP for ImageNet, followed by a linear head that produces the logits for the an n-way classiﬁcation (for example four-fold rotations is 4-way classiﬁcation), or a single node for the continuous group experiment.
Revised: The predictor is a 2 layer MLP for CIFAR-10 and Photonic-crystals, and a 3 layer MLP for ImageNet, followed by a linear head that produces the logits for the an n-way [Table] [Table] SimCLR

Revised: [Table]

Revised: SSL augmentation

Original: The predictor’s hidden dimension is shared across all layers and it equals 2048 for CIFAR-10 and ImageNet and 512 for PhC.
Revised: [Table] Alternatives to [Table] -SSL

Revised: Disentangled representations

Revised: [Table]

Revised: Insensitive instead of sensitive

Revised: [Table]

Revised: classiﬁcation (for example four-fold rotations is 4-way classiﬁcation), or a single node for the continuous group experiment.

Revised: The predictor’s hidden dimension is shared across all layers and it equals 2048 for CIFAR-10 and ImageNet and 512 for PhC.

Original: After each linear layer, there is a Layer Normalization (Ba et al., 2016) followed by ReLU.
Revised: After each linear layer, there is a Layer Normalization (Ba et al., 2016) followed by ReLU.

Original: We experimented with Batch Normalization (Ioffe & Szegedy, 2015) (with trainable afﬁne parameters) instead of Layer Normalization, but did not observe any signiﬁcant changes.
Revised: We experimented with Batch Normalization (Ioffe & Szegedy, 2015) (with trainable afﬁne parameters) instead of Layer Normalization, but did not observe any signiﬁcant changes.

Original: For some experiments, we discovered that removing the last ReLU from the MLP improves the results slightly.
Revised: For some experiments, we discovered that removing the last ReLU from the MLP improves the results slightly.

Original: In particular, for SimSiam on CIFAR-10 and for all models on ImageNet we omit the last ReLU.
Revised: In particular, for SimSiam on CIFAR-10 and for all models on ImageNet we omit the last ReLU.

Original: Finally, Algorithm 1 presents pseudocode for E-SSL with four-fold rotations on ImageNet.
Revised: Finally, Algorithm 1 presents pseudocode for E-SSL with four-fold rotations on ImageNet.

Original: In our implementation, we use smaller resolution for the rotated images, so that we can ﬁt all views on the same batch and have minimal overhead for pre-training (additional details in Table 9 in Appendix D). [Table] [Table] SimCLR
Revised: In our implementation, we use smaller resolution for the rotated images, so that we can ﬁt all views on the same batch and have minimal overhead for pre-training (additional details in Table 9 in Appendix E).

Original: SimSiam

Original: [Table]

Original: Disentangled representations

Original: [Table]

Original: Insensitive instead of sensitive

Original: [Table]

Original: 4. M AIN RESULTS
Revised: 4. M AIN RESULTS

Original: CIFAR-10 results.
Revised: CIFAR-10 results.

Original: To highlight the beneﬁts of our method, Table 1 demonstrates the improvement we obtain by using E-SSL on top of SimCLR and SimSiam and then shows different ablations and alternative methods.
Revised: To highlight the beneﬁts of our method, Table 1 demonstrates the improvement we obtain by using E-SSL on top of SimCLR and SimSiam and then shows different ablations and alternative methods.

Original: We label the E-SSL extensions as E-SimCLR and E-SimSiam respectively.
Revised: We label the E-SSL extensions as E-SimCLR and E-SimSiam respectively.

Original: We observe that we can increase a tuned baseline accuracy by about 2 − 3% .

Original: When ablating E-SSL, we see that each component of E-SSL is important.

Original: Most useful is the SSL augmentation applied on top of the rotated views.

Original: We also study alternatives to E-SSL.

Original: With “Disentangled representations” we investigate whether a “middle ground” is optimal for E-SSL: half of the representation to be insensitive to a transformation and the other half to be sensitive to the same transformation.

Original: This results in degradation of performance, which reﬂects our hypothesis that the representations should be either insensitive or sensitive. Weconducted this experiment by using four-fold rotations in ISSL for half of the representation and E-SSL for the other half. Finally, making the representations “Insensitive instead of sensitive” to four-fold-rotations hurts the performance signiﬁcantly, as it is also observed in Figure 1, and in (Chen et al., 2020; Xiao et al., 2020). Figure 4 reveals that E-SSL is more robust to removing transformations for I-SSL or reducing the labels for training.
Revised: We

Original: For example, E-SimCLR and E-SimSiam with only random resized cropping

Original: 1.0 fraction of training data     t
Revised: 1.0 fraction of training data     t

Original: o pli n e a r p r o b e a cc .
Revised: o pli n e a r p r o b e a cc .

Original: SimCLR semi-supervised
Revised: SimCLR semi-supervised

Original: Baseline E-SSL (ours)
Revised: Baseline E-SSL (ours)

Original: N o A u g m e n t .
Revised: N o A u g m e n t .

Original: C r o p o n l y
Revised: C r o p o n l y

Original: C r o p + F li p o n l y
Revised: C r o p + F li p o n l y

Original: N o G r
Revised: N o G r

Original: a y s c a l e
Revised: a y s c a l e

Original: F u ll A u g m e n t .
Revised: F u ll A u g m e n t .

Original: SimCLR remove data aug.
Revised: SimCLR remove data aug.

Original: 1.0 fraction of training data
Revised: 1.0 fraction of training data

Original: SimSiam semi-supervised
Revised: SimSiam semi-supervised

Original: N o A u g m e n t .
Revised: N o A u g m e n t .

Original: C r o p o n l y
Revised: C r o p o n l y

Original: C r o p + F li p o n l y
Revised: C r o p + F li p o n l y

Original: N o G r
Revised: N o G r

Original: a y s c a l e
Revised: a y s c a l e

Original: F u ll A u g m e n t .
Revised: F u ll A u g m e n t .

Original: SimSiam remove data aug.
Revised: SimSiam remove data aug.

Original: [Table]
Revised: [Table]

Revised: observe that we can increase a tuned baseline accuracy by about 2 − 3% .

Revised: When ablating E-SSL, we see that each component of E-SSL is important.

Revised: Most useful is the SSL augmentation applied on top of the rotated views.

Original: obtain 83.5% and 84.6% accuracies.
Revised: We also study alternatives to E-SSL. With “Disentangled representations” we investigate whether a “middle ground” is optimal for E-SSL: half of the representation to be insensitive to a transformation and the other half to be sensitive to the same transformation. This results in degradation of performance, which reﬂects our hypothesis that the representations should be either insensitive or sensitive. We conducted this experiment by using four-fold rotations in ISSL for half of the representation and E-SSL for the other half.

Revised: Finally, making the representations “Insensitive instead of sensitive” to four-fold-rotations hurts the performance signiﬁcantly, as it is also observed in Figure 1, and in (Chen et al., 2020; Xiao et al., 2020).

Revised: Figure 4 reveals that E-SSL is more robust to removing transformations for I-SSL or reducing the labels for training.

Revised: For example, E-SimCLR and E-SimSiam with only random resized cropping obtain 83.5% and 84.6% accuracies.

Original: Encouraging sensitivity to one transformation, namely fourfold-rotations, can reduce the need for selecting many transformations for I-SSL and with only 1% of the training data, E-SimCLR and E-SimSiam achieve 90.0 ± 1.0% and 88.6 ± 1.0% respectively.
Revised: Encouraging sensitivity to one transformation, namely fourfold-rotations, can reduce the need for selecting many transformations for I-SSL and with only 1% of the training data, E-SimCLR and E-SimSiam achieve 90.0 ± 1.0% and 88.6 ± 1.0% respectively.

Original: [Table]
Revised: [Table]

Original: ImageNet results.
Revised: ImageNet results.

Original: Table 2 demonstrates our main results on the linear probe on ImageNet after pre-training with various state-of-the-art
Revised: Table 2 demonstrates our main results on the linear probe on ImageNet after pre-training with various state-of-the-art

Original: I-SSL methods and their E-SSL versions.
Revised: I-SSL methods and their E-SSL versions.

Original: By only sweeping λ and slightly reducing the original learning rate for SimSiam we obtain consistent 1%/ 0.5%/ 1.3% improvements for
Revised: By only sweeping λ and slightly reducing the original learning rate for SimSiam we obtain consistent 1%/ 0.5%/ 1.3% improvements for

Original: SimCLR/ SimSiam/ Barlow Twins respectively.
Revised: SimCLR/ SimSiam/ Barlow Twins respectively.

Original: Additionally, in Table 3 we observe consistent beneﬁts of using E-SSL with longer pretraining.
Revised: Additionally, in Table 3 we observe consistent beneﬁts of using E-SSL with longer pretraining.

Original: In sum, E-SSL can improve different types of state-of-the-art representations in computer vision.
Revised: Finally, after 800 epochs of pretraining E-SimCLR achieves 72.5% , which is 0.6% better than SimCLR’s 71.9% baseline. [Figure] D O S ( a r b i t r a r y u n i t s ) [Figure]

Revised: PhC Dataset

Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: To show that other domains beneﬁt from E-SSL in a qualitatively similar way to the applications in the previous section, here we introduce two datasets in photonics science.
Revised: To show that other domains beneﬁt from E-SSL in a qualitatively similar way to the applications in the previous section, here we introduce two datasets in photonics science.

Original: Figure 5 depicts the datasets, i.e. input-label pairs consisting of 2D square periodic unit cells of PhCs and their associated
Revised: Figure 5 depicts the datasets, i.e. input-label pairs consisting of 2D square periodic unit cells of PhCs and their associated

Original: DOS.
Revised: DOS.

Original: The physics of the problem dictates that the DOS is invariant to (rolling) translations, scaling of all pixels by a ﬁxed positive factor, and operations of the C 4 v symmetry group, i.e. rotations and mirror ﬂips.
Revised: The physics of the problem dictates that the DOS is invariant to (rolling) translations, scaling of all pixels by a ﬁxed positive factor, and operations of the C 4 v symmetry group, i.e. rotations and mirror ﬂips.

Original: In choosing the transformations that E-SSL should encourage sensitivity to, we observe that the transformations that have worked for CIFAR-10 and ImageNet disturb the natural setting of the data (e.g. rotations disturb the natural upright setting of images).
Revised: In choosing the transformations that E-SSL should encourage sensitivity to, we observe that the transformations that have worked for CIFAR-10 and ImageNet disturb the natural setting of the data (e.g. rotations disturb the natural upright setting of images).

Original: Thus, we encourage sensitivity to transformations that ﬁt this observation, and insensitivity to the rest of the transformations.[Figure]
Revised: Thus, we encourage sensitivity to transformations that ﬁt this observation, and insensitivity to the rest of the transformations.

Original: D O S ( a r b i t r a r y u n i t s )

Original: [Figure] [Table] In Figure 5, the top dataset is a “Blob” dataset where the shape variation in each image is centered.
Revised: In Figure 5, the top dataset is a “Blob” dataset where the shape variation in each image is centered.

Original: We encourage sensititivy to the group of four-fold translations , given by G = { e, h, v, hv } , where h and v are 1/2-unit cell translations in the horizontal and vertical axis, respectively, e is the unit element (no transformation) and hv is the composition of h and v .
Revised: We encourage sensititivy to the group of four-fold translations , given by G = { e, h, v, hv } , where h and v are 1/2-unit cell translations in the horizontal and vertical axis, respectively, e is the unit element (no transformation) and hv is the composition of h and v .

Original: In the bottom dataset of Figure 5, the PhC unit cells are generated to have a horizontal mirror symmetry, i.e. we use the 2D wallpaper (or crystallographic plane) group pm .
Revised: In the bottom dataset of Figure 5, the PhC unit cells are generated to have a horizontal mirror symmetry, i.e. we use the 2D wallpaper (or crystallographic plane) group pm .

Original: We encourage sensitivity to the group of four-fold rotations (the same group we used for CIFAR-10 and ImageNet), since rotating any of the images disturbs the (horizontal) mirror symmetry.
Revised: We encourage sensitivity to the group of four-fold rotations (the same group we used for CIFAR-10 and ImageNet), since rotating any of the images disturbs the (horizontal) mirror symmetry.

Original: More accurately, since only ± π/ 2 rotations disturb the symmetry, we separate them in two classes, { π/ 2 , − π/ 2 } and { 0 , π } , and perform binary prediction in E-SSL.
Revised: More accurately, since only ± π/ 2 rotations disturb the symmetry, we separate them in two classes, { π/ 2 , − π/ 2 } and { 0 , π } , and perform binary prediction in E-SSL.

Original: Table 4 shows the results of ﬁne-tuning the backbone and an additional DOS-predictor head (see
Revised: Table 4 shows the results of ﬁne-tuning the backbone and an additional DOS-predictor head (see

Original: Appendix E) with 3000 labelled samples for this regression task.
Revised: Appendix F) with 3000 labelled samples for this regression task.

Original: We observe that encouraging sensitivity to the selected transformations (via E-SimCLR) leads to the largest reduction in the error.
Revised: We observe that encouraging sensitivity to the selected transformations (via E-SimCLR) leads to the largest reduction in the error.

Original: On the contrary, including these transformations to SimCLR (indicated by “+ Transform”) increases the error.
Revised: On the contrary, including these transformations to SimCLR (indicated by “+ Transform”) increases the error.

Original: Furthermore, we explore scaling transformations and show that E-SSL can be generalized to inﬁnite groups (see Appendix E).
Revised: Furthermore, we explore scaling transformations and show that E-SSL can be generalized to inﬁnite groups (see Appendix F).

Original: This supports our observations about the usefulness of E-SSL over I-SSL and demonstrates E-SSL’s generality beyond computer vision. 
Revised: This supports our observations about the usefulness of E-SSL over I-SSL and demonstrates E-SSL’s generality beyond computer vision. A CKNOWLEDGEMENTS

Revised: We thank Shurong Lin, Kristian Georgiev, Alexander Atanasov, Peter Lu and the anonymous reviewers for fruitful conversations and support.

Revised: R.D. dedicates this work to the memory of Boyko

Revised: Dangovski.

Revised: The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center (Reuther et al., 2018) for providing HPC and consultation resources that have contributed to the research results reported within this paper/report.

Original: C ONCLUSION AND F UTURE W ORK
Revised: Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artiﬁcial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the United States Air Force or the U.S. Government.

Revised: The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

Revised: This material is also based in part upon work supported by the Air Force Ofﬁce of Scientiﬁc Research under the award number FA9550-21-1-0317 and the U. S. Army Research Ofﬁce through the

Revised: Institute for Soldier Nanotechnologies at MIT, under Collaborative Agreement Number W911NF18-2-0048.

Revised: This work is also supported in part by the the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artiﬁcial Intelligence and Fundamental

Revised: Interactions, http://iaifi.org/ ).

Revised: R EPRODUCIBILITY S TATEMENT

Revised: Algorithm 1, the original public code for each of the I-SSL methods we use in the paper and the experimental setups in Section 4.1, and in Appendices D, E and F, can be used for reproducibility.

Revised: Our code is available at https://github.com/rdangovs/essl .

Revised: A S UMMARY OF M AIN T EXT AND L AYOUT OF A PPENDIX

Original: In this paper we motivated the generalization of state-of-the-art methods in self-supervised learning to the more general framework of equivariant self-supervised learning (E-SSL).
Revised: In this paper we motivated the generalization of state-of-the-art methods in self-supervised learning to the more general framework of equivariant self-supervised learning (E-SSL).

Original: In E-SSL rather than using only invariance as a trivial case of equivariance, we encouraged non-trivial equivariance and improved state-of-the-art methods on common computer vision benchmarks and regressions tasks in photonics science.
Revised: In E-SSL rather than using only invariance as a trivial case of equivariance, we encouraged non-trivial equivariance and improved state-of-the-art methods on common computer vision benchmarks and regressions tasks in photonics science.

Original: We also discussed that there are many types of equivariance we can consider for
Revised: We also discussed that there are many types of equivariance we can consider for

Original: E-SSL.
Revised: E-SSL.

Original: We observed that most of the successful transformations for E-SSL that we explored form groups, but  that potentially many more transformations could be explored.
Revised: We observed that most of the successful transformations for E-SSL that we explored form groups, but forsee that potentially many more transformations could be explored.

Original: For future work one coulduse explicit constructions of equivariant properties as an objective to learn transformations, instead of setting them manually.
Revised: For future work one could learn transformations that are equivariances, instead of setting them manually.

Original: Thus, the concept of E-SSL could potentially be extended to natural language processing or other science domains, whose transformations for SSL are less well-understood.
Revised: Thus, the concept of E-SSL could potentially be extended to natural language processing or other science domains, whose transformations for SSL are less well-understood.

Original: Algorithm 1, the original public code for each of the I-SSL methods we use in the paper and  the experimental setups in Section 4.1, and in Appendices C, D and E, can be used for reproducibility. Code with annotation is available as supplementary material.
Revised: To facilitate further research in E-SSL, below we provide additional details and analysis of the experiments in the main text.

Original: We will make our code public if the paper is accepted.
Revised: We also discuss interesting avenues for future work.

Original: A P ROOF OF P ROPOSITION
Revised: B P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: To construct a non-trivially equivariant f , we ﬁrst need to show that both X (cid:48) and S are
Revised: To construct a non-trivially equivariant f , we ﬁrst need to show that both X (cid:48) and S are

Original: G -sets, i.e. that there is a group action T g of G on X (cid:48) , which is given by the statement of the proposition, and another (non-trivial) group action T (cid:48) g of G on S , which we will construct.
Revised: G -sets, i.e. that there is a group action T g of G on X (cid:48) , which is given by the statement of the proposition, and another (non-trivial) group action T (cid:48) g of G on S , which we will construct.

Original: Then, we need to show that f commutes with the group action, i.e. that f ( T g ( x (cid:48) ))
Revised: Then, we need to show that f commutes with the group action, i.e. that f ( T g ( x (cid:48) ))

Original: =
Revised: =

Original: T (cid:48)g ( f ( x (cid:48) )) .
Revised: T (cid:48)

Original: Group actions.
Revised: Group actions.

Original: Note that by the setup of the problem, we are already given how G acts on the input X (cid:48) , i.e. T g is known.
Revised: Note that by the setup of the problem, we are already given how G acts on the input X (cid:48) , i.e. T g is known.

Original: For example, if G is the group of four-fold rotations, then T g is the rotation of the input by a multiple of π/ 2 .
Revised: For example, if G is the group of four-fold rotations, then T g is the rotation of the input by a multiple of π/ 2 .

Original: We proceed to construct the non-trivial group action T (cid:48) g of G on S.
Revised: We proceed to construct the non-trivial group action T (cid:48) g of G on S.

Original: Deﬁne the function T (cid:48) : G × S → S as T (cid:48) ( g, s ) =
Revised: Deﬁne the function T (cid:48) : G × S → S as T (cid:48) ( g, s ) =

Original: f ( T g ( T g (cid:48)
Revised: f ( T g ( T g (cid:48)

Original: ( x (cid:48) ))) , where s = f ( T g (cid:48)
Revised: ( x (cid:48) ))) , where s = f ( T g (cid:48)

Original: ( x (cid:48) )) .
Revised: ( x (cid:48) )) .

Original: Note that T (cid:48) is well-deﬁned, because gg (cid:48) ∈ G by the closure of the group and s is uniquely written as s =
Revised: Note that T (cid:48) is well-deﬁned, because gg (cid:48) ∈ G by the closure of the group and s is uniquely written as s =

Original: f ( T g (cid:48)
Revised: f ( T g (cid:48)

Original: ( x (cid:48) )) .
Revised: ( x (cid:48) )) .

Original: To see why s is uniquely written, it sufﬁces to show that if f ( T g (cid:48) ( x (cid:48) ))
Revised: To see why s is uniquely written, it sufﬁces to show that if f ( T g (cid:48) ( x (cid:48) ))

Original: =
Revised: =

Original: f ( T g (cid:48)(cid:48)
Revised: f ( T g (cid:48)(cid:48)

Original: ( x (cid:48)(cid:48) )) then both g (cid:48)
Revised: ( x (cid:48)(cid:48) )) then both g (cid:48)

Original: =
Revised: =

Original: g (cid:48)(cid:48) and x (cid:48)
Revised: g (cid:48)(cid:48) and x (cid:48)

Original: =
Revised: =

Original: x (cid:48)(cid:48) , which follows directly from our assumption in the statement.
Revised: x (cid:48)(cid:48) , which follows directly from our assumption in the statement.

Original: Now, to prove that T (cid:48) is a group action, it sufﬁces to show two properties.
Revised: Now, to prove that T (cid:48) is a group action, it sufﬁces to show two properties.

Original: • Identity: T (cid:48)
Revised: • Identity: T (cid:48)

Original: ( e, s )
Revised: ( e, s )

Original: =
Revised: =

Original: s for s = f ( g (cid:48)
Revised: s for s = f ( g (cid:48)

Original: ( x (cid:48) )) and e is the unit element of the group.
Revised: ( x (cid:48) )) and e is the unit element of the group.

Original: To show that, note that by deﬁnition T (cid:48) ( e, s ) =
Revised: To show that, note that by deﬁnition T (cid:48) ( e, s ) =

Original: f ( T e ( T g (cid:48) ( x (cid:48) )))
Revised: f ( T e ( T g (cid:48) ( x (cid:48) )))

Original: =
Revised: =

Original: f ( T g (cid:48)
Revised: f ( T g (cid:48)

Original: ( x (cid:48) )) , because eg (cid:48) =
Revised: ( x (cid:48) )) , because eg (cid:48) =

Original: g (cid:48).
Revised: g (cid:48)

Original: • Compositionality: T (cid:48) ( g, T (cid:48) ( h, f
Revised: • Compositionality: T (cid:48) ( g, T (cid:48) ( h, f

Original: ( T g (cid:48)
Revised: ( T g (cid:48)

Original: ( x (cid:48) ))))
Revised: ( x (cid:48) ))))

Original: =
Revised: =

Original: T (cid:48)
Revised: T (cid:48)

Original: ( gh, f ( T g (cid:48)
Revised: ( gh, f ( T g (cid:48)

Original: ( x (cid:48) ))) .
Revised: ( x (cid:48) ))) .

Original: To show this, we expand the LHS and use the deﬁnition of T (cid:48) to obtain as follows 
Revised: To show this, we expand the LHS and use the deﬁnition of T (cid:48) to obtain as follows T (cid:48) ( g, T (cid:48)

Original: [Equation]
Revised: ( h, f

Revised: ( T g (cid:48)

Original:  )))) =[Equation] x
Revised: ( x (cid:48) )))) =

Original: [Equation]
Revised: T (cid:48) ( g, f

Original: [Equation] [Equation] x [Equation]
Revised: ( T h ( T g (cid:48) ( x (cid:48) ))))

Original: =
Revised: =

Original: [Equation] gh
Revised: f ( T g ( T hg (cid:48)

Original: [Equation]
Revised: ( x (cid:48) ))) =

Original: g (cid:48)
Revised: f ( T ghg (cid:48)

Original: ( x (cid:48) )))
Revised: ( x (cid:48) ))

Original: =
Revised: =

Revised: f ( T gh ( T g (cid:48)

Revised: ( x (cid:48) ))) =

Original: T (cid:48)
Revised: T (cid:48)

Original: ( gh, f ( T g (cid:48)
Revised: ( gh, f ( T g (cid:48)

Original: ( x (cid:48) ))) , because the group operation is associative.
Revised: ( x (cid:48) ))) , because the group operation is associative.

Original: Hence, T (cid:48) is a group action, and thus S is a G -set, and we can write T (cid:48)
Revised: Hence, T (cid:48) is a group action, and thus S is a G -set, and we can write T (cid:48)

Original: ( g, · ) ≡ T (cid:48) g ( · ) .
Revised: ( g, · ) ≡ T (cid:48) g ( · ) .

Original: Commuting with the group action.
Revised: Commuting with the group action.

Original: To see this property, note that T (cid:48) g (cid:48)
Revised: To see this property, note that T (cid:48) g (cid:48)

Original: ( f ( x (cid:48) )) =
Revised: ( f ( x (cid:48) )) =

Original: T (cid:48) g (cid:48)
Revised: T (cid:48) g (cid:48)

Original: ( f ( T g ( x )))
Revised: ( f ( T g ( x )))

Original: =
Revised: =

Original: f ( T g (cid:48)
Revised: f ( T g (cid:48)

Original: ( T g ( x )))
Revised: ( T g ( x )))

Original: =
Revised: =

Original: f ( T g (cid:48)
Revised: f ( T g (cid:48)

Original: ( x (cid:48) )) as desired.
Revised: ( x (cid:48) )) as desired.

Original: Note that T (cid:48) g is non-trivial.
Revised: Note that T (cid:48) g is non-trivial.

Original: Therefore, we can conclude that f , which satisﬁes the constructed group action T (cid:48) g , is not-trivially equivariant to the group G.
Revised: Therefore, we can conclude that f , which satisﬁes the constructed group action T (cid:48) g , is not-trivially equivariant to the group G.

Original: B R OTATION PREDICTION AND I-SSL BENEFIT FROM SIMILAR DATA AUGMENTATION .
Revised: C R OTATION PREDICTION AND I-SSL BENEFIT FROM SIMILAR DATA AUGMENTATION .

Original: Recently, rotation prediction with a linear head from the frozen backbone representations proved to be useful for validating the augmentation policies of contrastive learning (Reed et al., 2021).
Revised: Recently, rotation prediction with a linear head from the frozen backbone representations proved to be useful for validating the augmentation policies of contrastive learning (Reed et al., 2021).

Original: This shows that the two tasks of classiﬁcation of ground truth classes and synthetic rotation classes from frozen backbone representations beneﬁt from similar augmentation policies.
Revised: This shows that the two tasks of classiﬁcation of ground truth classes and synthetic rotation classes from frozen backbone representations beneﬁt from similar augmentation policies.

Original: We took this experiment a step further, and performed rotation prediction with the augmentation policies, typically used in contrastive learning.
Revised: We took this experiment a step further, and performed rotation prediction with the augmentation policies, typically used in contrastive learning.

Original: The result is in Table 5.
Revised: The result is in Table 5.

Original: Interestingly, RotNet beneﬁts from augmentations, typically used in contrastive learning, and the RotNet training shares the same sweet spot (Tian et al., 2020) as kNN classiﬁcation.
Revised: Interestingly, RotNet beneﬁts from augmentations, typically used in contrastive learning, and the RotNet training shares the same sweet spot (Tian et al., 2020) as kNN classiﬁcation.

Original: There are several takeaways from this experiment: ( i ) we can ﬁnd good augmentations for contrastive learning by doing RotNet alone , i.e. without doing any contrastive learning; ( ii )
Revised: There are several takeaways from this experiment: ( i ) we can ﬁnd good augmentations for contrastive learning by doing RotNet alone , i.e. without doing any contrastive learning; ( ii )

Original: RotNet beneﬁts from augmentations needed in contrastive learning; ( iii ) we may be able to combine four-fold rotations prediction and contrastive learning.
Revised: RotNet beneﬁts from augmentations needed in contrastive learning; ( iii ) we may be able to combine four-fold rotations prediction and contrastive learning.

Original: C CIFAR-10 E XPERIMENTS C. E XPERIMENTAL SETUP
Revised: D CIFAR-10 E XPERIMENTS D. E XPERIMENTAL SETUP

Original: Our experiments use the following architectural choices: ResNet-18 backbone (the CIFAR-10 version has kernel size 3, stride 1, padding 1 and there is no max pooling afterwards); 512 batch size 
Revised: Our experiments use the following architectural choices: ResNet-18 backbone (the CIFAR-10 version has kernel size 3, stride 1, padding 1 and there is no max pooling afterwards); 512 batch size (only our baseline SimSiam model uses batch size 1024); 0.03 base learning rate for the baseline

Original: [Table]
Revised: SimCLR and SimSiam and 0.06 base learning rate for E-SimCLR and E-SimSiam; 800 pre-training epochs; standard cosine decayed learning rate; 10 epochs for the linear warmup; two layer projector with hidden dimension 2048 and output dimension 2048; for SimSiam a two layer (bottleneck) predictor with hidden dimension 512 whose learning rate is not decayed; the last batch normalization for the projector does not have learnable afﬁne parameters; 0.0005 weight decay value; SGD with momentum 0.9 optimizer. The augmentation is Random Resized Cropping with scale (0.2, 1.0),

Original: [Table]
Revised: [Table]

Original: have learnable afﬁne parameters; 0.0005 weight decay value; SGD with momentum 0.9 optimizer.

Original: The augmentation is Random Resized Cropping with scale (0.2, 1.0), aspect ratio (3/4, 4/3) and size 32x32, Random horizontal Flips with probability 0.5, Color Jittering (0.4, 0.4, 0.4, 0.1) with probability 0.8 and Grayscale with probability 0.2.
Revised: aspect ratio (3/4, 4/3) and size 32x32, Random horizontal Flips with probability 0.5, Color Jittering (0.4, 0.4, 0.4, 0.1) with probability 0.8 and Grayscale with probability 0.2.

Original: Some of our evaluations use a kNN-classifer with 200 neighbors, cosine similarity and Gaussian kernel with temperature 0.1.
Revised: Some of our evaluations use a kNN-classifer with 200 neighbors, cosine similarity and Gaussian kernel with temperature 0.1.

Original: This evaluation correlates well with the standard linear probe, but it is more efﬁcient to calculate.
Revised: This evaluation correlates well with the standard linear probe, but it is more efﬁcient to calculate.

Original: We report the kNN accuracy in % at the end of the 800 epochs of training.
Revised: We report the kNN accuracy in % at the end of the 800 epochs of training.

Original: For our main results, we report a linear probe accuracy from training a linear classiﬁer for 100 epochs on top of the frozen representations with SGD with momentum 0.9 and cosine decay of the learning rate, batch size 256 and initial laerning rate of 30.
Revised: For our main results, we report a linear probe accuracy from training a linear classiﬁer for 100 epochs on top of the frozen representations with SGD with momentum 0.9 and cosine decay of the learning rate, batch size 256 and initial laerning rate of 30.

Original: For linear probe experiments we try 5 different initalizations of the linear head and report mean and standard deviations.
Revised: For linear probe experiments we try 5 different initalizations of the linear head and report mean and standard deviations.

Original: The deviations are negligible because the linear probe is robust to the random seed.
Revised: The deviations are negligible because the linear probe is robust to the random seed.

Original: All parameters are reported in a Pytorch-like style.
Revised: All parameters are reported in a Pytorch-like style.

Original: For Figure 1 we use resolution of 32x32 for the transformations studied.
Revised: For Figure 1 we use resolution of 32x32 for the transformations studied.

Original: The 4 levels of the Gaussian blur are for kernel sizes 0, 5, 9 and 15 in the default Gaussian blur torchvision implementation.
Revised: The 4 levels of the Gaussian blur are for kernel sizes 0, 5, 9 and 15 in the default Gaussian blur torchvision implementation.

Original: The prediction of the transformations follows the experimental setup in Section 3.
Revised: The prediction of the transformations follows the experimental setup in Section 3.

Original: When we apply the transformations in I-SSL, we add them in the beginning of the augmentation policy with probability 1.
Revised: When we apply the transformations in I-SSL, we add them in the beginning of the augmentation policy with probability 1.

Original: The same setup is used for “Disentangled representations” and “Insensitive instead of sensitive” in Table 1.
Revised: The same setup is used for “Disentangled representations” and “Insensitive instead of sensitive” in Table 1.

Original: C. A DDITIONAL EXPERIMENTS
Revised: D. A DDITIONAL EXPERIMENTS

Original: Explored hyperparameters.
Revised: Explored hyperparameters.

Original: Both for SimCLR and SimSiam we ran a grid search over the following hyperparameters: base learning rate: { 0.01, 0.03, 0.06 } , batch size: { 512, 1024 } , λ (for
Revised: Both for SimCLR and SimSiam we ran a grid search over the following hyperparameters: base learning rate: { 0.01, 0.03, 0.06 } , batch size: { 512, 1024 } , λ (for

Original: E-SSL): { 
Revised: E-SSL): { 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 } , predictor’s MLP depth: { 2, 3, 4 } , predictor’s normalization: { None, BatchNorm, LayerNorm } , nonlinearity at the last MLP layer of the predictor: { True, False } .

Original: Tuning λ .
Revised: Tuning λ .

Original: Table 6 shows tuning of the CIFAR-10 results.
Revised: Table 6 shows tuning of the CIFAR-10 results.

Original: We observe noticeable improvements over the SSL baselines by using E-SSL instead. 
Revised: We observe noticeable improvements over the SSL baselines by using E-SSL instead. [Table]

Original: Sensitivity to transformations for I-SSL.
Revised: Sensitivity to transformations for I-SSL.

Original: Table 7 demonstrates that E-SSL can produce good representation with as few SSL transformations for I-SSL as possible.
Revised: Table 7 demonstrates that E-SSL can produce good representation with as few SSL transformations for I-SSL as possible.

Original: We observe that E-SSL is less sensitive than SSL to the choice of data augmentation. 
Revised: We observe that E-SSL is less sensitive than SSL to the choice of data augmentation. [Table]

Original: The importance of complete invariance or sensitivity.
Revised: The importance of complete invariance or sensitivity.

Original: Table 8 studies whether a middle ground for the representations exist,i.e. whether it is possible to have part of the representation invariant
Revised: Table 8 studies whether a middle ground for the representations exist,

Original: [Table]
Revised: i

Original: [Table]
Revised: [Table]

Original:  and the other part sensitive to the transformation.
Revised: e. whether it is possible to have part of the representation invariant and the other part sensitive to the transformation.

Original: If we apply the E-SSL loss only to half of the representation [Table] then there is a very small drop in the performance.
Revised: If we apply the E-SSL loss only to half of the representation, then there is a very small drop in the performance.

Original: Furthermore, we observe that having a disjoint mix between insensitivity and sensitivity in the representation is noticeably harmful.
Revised: Furthermore, we observe that having a disjoint mix between insensitivity and sensitivity in the representation is noticeably harmful.

Original: [Table]
Revised: [Table]

Original: Fully connected backbone.
Revised: Fully connected backbone.

Original: We perform a simple experiment with a fully connected backbone, instead of a ResNet-18.
Revised: We perform a simple experiment with a fully connected backbone, instead of a ResNet-18.

Original: The hidden dimensions of the backbone are listed in order as { 3 × 32 × 32, 2048, 2048, 512 } with Batch Normalization and ReLUs in between.
Revised: The hidden dimensions of the backbone are listed in order as { 3 × 32 × 32, 2048, 2048, 512 } with Batch Normalization and ReLUs in between.

Original: The rest of the experimental setup is exactly the same.
Revised: The rest of the experimental setup is exactly the same.

Original: On the linear probe (%), we obtain 70.5 ± 0.0 for SimCLR and 73.8 ± 0.
Revised: On the linear probe (%), we obtain 70.5 ± 0.0 for SimCLR and 73.8 ± 0.

Original: for E-SimCLR, and 70.9 ± 0.0 for SimSiam and 73.5 ± 0.1 for E-SimSiam, highlighting noticeable gains from using E-SSL.
Revised: for E-SimCLR, and 70.9 ± 0.0 for SimSiam and 73.5 ± 0.1 for E-SimSiam, highlighting noticeable gains from using E-SSL.

Original: CIFAR-100 experiments.
Revised: CIFAR-100 experiments.

Original: We test our CIFAR-10 experimental setup directly on CIFAR-100.
Revised: We test our CIFAR-10 experimental setup directly on CIFAR-100.

Original: On the linear probe (%), we obtain 65.8 ± 0.0 for SimCLR and 69.5 ± 0.1 for E-SimCLR, and 65.8 ± 0.
Revised: On the linear probe (%), we obtain 65.8 ± 0.0 for SimCLR and 69.5 ± 0.1 for E-SimCLR, and 65.8 ± 0.

Original: for SimSiam and 69.3 ± 0.1 for E-SimSiam, highlighting sizable gains from using E-SSL.
Revised: for SimSiam and 69.3 ± 0.1 for E-SimSiam, highlighting sizable gains from using E-SSL.

Original: Large crop study.
Revised: Large crop study.

Original: We study whether using a large crop with a single rotation on CIFAR-10 can be just as good as a small crop.
Revised: We study whether using a large crop with a single rotation on CIFAR-10 can be just as good as a small crop.

Original: We obtain 93.9 ± 0.0 on the linear probe using E-SimCLR, which is only 0.2 absolute points below our best result of 94.1 ± 0.0 using four small crops.
Revised: We obtain 93.9 ± 0.0 on the linear probe using E-SimCLR, which is only 0.2 absolute points below our best result of 94.1 ± 0.0 using four small crops.

Original: C. N ORM - DIFFERENCES A NALYSIS
Revised: D. N ORM - DIFFERENCES A NALYSIS

Original: In Figure 6 we present analysis that shows our training objectives encourage invariance and equivariance to transformations.
Revised: In Figure 6 we present analysis that shows our training objectives encourage invariance and equivariance to transformations.

Original: We take our best performing E-SimCLR and E-SimSiam methods on
Revised: We take our best performing E-SimCLR and E-SimSiam methods on

Original: CIFAR-10.
Revised: CIFAR-10.

Original: During training we keep track of two measures that can capture how invariant/ equivariant the backbone representations are.
Revised: During training we keep track of two measures that can capture how invariant/ equivariant the backbone representations are.

Original: The “invariance mesure” computes the negative cosine similarity between two views of the backbone representations.
Revised: The “invariance measure” computes the negative cosine similarity between two views of the backbone representations.

Original: The lower this measure is, the higher the similarity between the two views, and thus the more invariant the backbone representations are to the transformations in I-SSL.
Revised: The lower this measure is, the higher the similarity between the two views, and thus the more invariant the backbone representations are to the transformations in I-SSL.

Original: We observe that during training high similarity between the two views is maintained (roughly between 0.8 and 0.9), which indicates that invariance is encouraged in the backbone representations, as desired.
Revised: We observe that during training high similarity between the two views is maintained (roughly between 0. and 0.9), which indicates that invariance is encouraged in the backbone representations, as desired.

Original: Likewise, the “equivariance measure” computes the average cosine similarity of the backbone represenations, between all six pairs of the four rotated views.
Revised: Likewise, the “equivariance measure” computes the average cosine similarity of the backbone represenations, between all six pairs of the four rotated views.

Original: The lower this measure is, the lower the similarity between the four views, and thus the more non-trivially equivariant the backbone representations are to the transformations for equivariance.
Revised: The lower this measure is, the lower the similarity between the four views, and thus the more non-trivially equivariant the backbone representations are to the transformations for equivariance.

Original: We observe that the measure decays to about 0.3 during training, which indicates that the backbone representations are encouraged to be equivariant to four-fold rotations, as desired
Revised: We observe that the measure decays to about 0.3 during training, which indicates that the backbone representations are encouraged to be equivariant to four-fold rotations, as desired.

Original: [Figure]
Revised: [Figure]

Original: D I MAGE N ET E XPERIMENTS
Revised: E I MAGE N ET E XPERIMENTS

Original: We had limited computational resources, so we kept the learning rates the same as in the original methods.
Revised: We had limited computational resources, so we kept the learning rates the same as in the original methods.

Original: Only for SimSiam we found that choosing a smaller learning rate 0.08 leads to better results for E-SimSiam.
Revised: Only for SimSiam we found that choosing a smaller learning rate 0.08 leads to better results for E-SimSiam.

Original: We only swept the λ parameter, where for SimCLR and SimSiam
Revised: We only swept the λ parameter, where for SimCLR and SimSiam

Original: the sweep was between 0 and 1 and for Barlow Twins it was between 0 and 100.
Revised: the sweep was between 0 and 1 and for Barlow Twins it was between 0 and 100.

Original: The optimal λ is 0.4 for
Revised: The optimal λ is 0.4 for

Original: SimCLR, 0.08 for SimSiam, 8 for Barlow Twins.
Revised: SimCLR, 0.08 for SimSiam, 8 for Barlow Twins.

Original: We use (0.05, 0.14) scale range for 100 pretraining epochs
Revised: We use (0.05, 0.14) scale range for 100 pretraining epochs.

Original: For more pre -training epochs we use (0.05, 0.14) for SimCLR and (0.08, 1.0) for
Revised: For more pre-training epochs we use (0.05, 0.14) for SimCLR and (0.08, 1.0) for

Original: Barlow Twins
Revised: Barlow Twins.

Original: Table 9 lists the overhead from using rotation prediction in our experiments. 
Revised: Table 9 lists the overhead from using rotation prediction in our experiments. [Table]

Original: E P H C E XPERIMENTS
Revised: F P H C E XPERIMENTS

Original: Dataset generation.
Revised: Dataset generation.

Original: 2D Photonic crystals (PhCs) are characterized by a periodically varying permitivitty ε ( x, y ) ; here, for simplicity we consider a “two-tone” permitivitty proﬁle i.e. ε ∈ { ε 1 , ε 2 } , with ε i ∈ [1 , 20] discretized to a resolution of 32 × 32 .
Revised: 2D Photonic crystals (PhCs) are characterized by a periodically varying permitivitty ε ( x, y ) ; here, for simplicity we consider a “two-tone” permitivitty proﬁle i.e. ε ∈ { ε 1 , ε 2 } , with ε i ∈ [1 , 20] discretized to a resolution of 32 × 32 .

Original: To generate the unit cells in the “blob” dataset, we follow the proceedure in Christensen et al.
Revised: To generate the unit cells in the “blob” dataset, we follow the proceedure in Christensen et al.

Original: For the Gpm dataset, the unit cells are deﬁned using a level set of a 2D Fourier sum function like in Kim et al.
Revised: For the Gpm dataset, the unit cells are deﬁned using a level set of a 2D Fourier sum function like in Kim et al.

Original: (2021); Loh et al.
Revised: (2021); Loh et al.

Original: (2021), with[Table] additional constraints applied to the lattice to create the mirror symmetry adopted from the method in Christensen et al.
Revised: (2021), with additional constraints applied to the lattice to create the mirror symmetry adopted from the method in Christensen et al.

Original: We then follow the procedure in Loh et al.
Revised: We then follow the procedure in Loh et al.

Original: (2021) to compute, and subsequently process, the density-of-states (DOS) of each unit cell, speciﬁcally, via the MIT Photonics
Revised: (2021) to compute, and subsequently process, the density-of-states (DOS) of each unit cell, speciﬁcally, via the MIT Photonics

Original: Bands (MPB) software (Johnson & Joannopoulos, 2001) and the Generalized Gilat-Raubenheimer method in an implementation from Liu et al.
Revised: Bands (MPB) software (Johnson & Joannopoulos, 2001) and the Generalized Gilat-Raubenheimer method in an implementation from Liu et al.

Original: Network architecture.
Revised: Network architecture.

Original: We use an encoder network composing of simple convolutional (CNN) and fully-connected (FC) layers for the backbone; speciﬁcally, our backbone begins with 3 CNN layers, all with a kernel size of 7 and channel dimensions given by [64 , 256 , 256] .
Revised: We use an encoder network composing of simple convolutional (CNN) and fully-connected (FC) layers for the backbone; speciﬁcally, our backbone begins with 3 CNN layers, all with a kernel size of 7 and channel dimensions given by [64 , 256 , 256] .

Original: The output is ﬂattened and fed into 2 FC layers each with 1024 nodes (i.e. the representations have dimension 1024).
Revised: The output is ﬂattened and fed into 2 FC layers each with 1024 nodes (i.e. the representations have dimension 1024).

Original: We include BatchNorm (Ioffe & Szegedy, 2015), ReLU and MaxPooling for the CNNs, and ReLU only for the ﬁrst FC layer.
Revised: We include BatchNorm (Ioffe & Szegedy, 2015), ReLU and MaxPooling for the CNNs, and ReLU only for the ﬁrst FC layer.

Original: The projector and predictor networks, p 1 and p 2 are 2-layer
Revised: The projector and predictor networks, p 1 and p 2 are 2-layer

Original: MLPs with hidden dimension 512, with BatchNorm and ReLU between each layer except the last and the projection dimension for p 1 is 256.
Revised: MLPs with hidden dimension 512, with BatchNorm and ReLU between each layer except the last and the projection dimension for p 1 is 256.

Original: Additionally, since this is a regression task and the label space is much larger than in image classiﬁcation tasks, we include a dense DOS-predictor head after the representations, which is ﬁne-tuned with 3000 labelled samples after SSL or E-SSL.
Revised: Additionally, since this is a regression task and the label space is much larger than in image classiﬁcation tasks, we include a dense DOS-predictor head after the representations, which is ﬁne-tuned with 3000 labelled samples after SSL or E-SSL.

Original: The DOSpredictor has 4 FC layers, with number of nodes given by [1024 , 1024 , 512 , 400] .
Revised: The DOSpredictor has 4 FC layers, with number of nodes given by [1024 , 1024 , 512 , 400] .

Original: We explore two ﬁne-tuning protocols of the DOS-predictor: freezing the backbone (discussed later in the Appendix) or ﬁne-tuning the backbone (discussed in the main text).
Revised: We explore two ﬁne-tuning protocols of the DOS-predictor: freezing the backbone (discussed later in the Appendix) or ﬁne-tuning the backbone (discussed in the main text).

Original: Hyperparameters.
Revised: Hyperparameters.

Original: For SSL and E-SSL, we performed 250 pre-training epochs using the SGD optimizer with a standard cosine decayed learning rate; the batch size was ﬁxed to 512.
Revised: For SSL and E-SSL, we performed 250 pre-training epochs using the SGD optimizer with a standard cosine decayed learning rate; the batch size was ﬁxed to 512.

Original: The pretrained model was saved at various epochs { 20, 50, 100, 180, 250 } for further ﬁne-tuning.
Revised: The pretrained model was saved at various epochs { 20, 50, 100, 180, 250 } for further ﬁne-tuning.

Original: Finetuning was performed for 100 epochs using Adam optimizer and a ﬁxed batch size of 64.
Revised: Finetuning was performed for 100 epochs using Adam optimizer and a ﬁxed batch size of 64.

Original: No transformations were applied to the input during ﬁne-tuning for both freezing or ﬁne-tuning the backbone.
Revised: No transformations were applied to the input during ﬁne-tuning for both freezing or ﬁne-tuning the backbone.

Original: We ran a grid search over the following hyperparameters; for pre-training, base learning rate: { 10 − 3 , 10 − 4 , 10 − 5 } , λ (for E-SSL): { 0.2, 1.0, 2.0, 5.0, 10.0 } , and for ﬁne-tuning: a learning rate in { 10 − 3 , 10 − 4 , 10 − 5 } .
Revised: We ran a grid search over the following hyperparameters; for pre-training, base learning rate: { 10 − 3 , 10 − 4 , 10 − 5 } , λ (for E-SSL): { 0.2, 1.0, 2.0, 5.0, 10.0 } , and for ﬁne-tuning: a learning rate in { 10 − 3 , 10 − 4 , 10 − 5 } .

Original: Frozen backbone experiment.
Revised: Frozen backbone experiment.

Original: In Table 10 we present our results from freezing the backbone encoder while ﬁne-tuning the DOS-predictor head.
Revised: In Table 10 we present our results from freezing the backbone encoder while ﬁne-tuning the DOS-predictor head.

Original: We observe similar trends as in Table 4 where we allowed ﬁne-tuning of the backbone.
Revised: We observe similar trends as in Table 4 where we allowed ﬁne-tuning of the backbone.

Original: Relative error is reported in % and the lower the error is, the better.
Revised: Relative error is reported in % and the lower the error is, the better.

Original: SimCLR for Blob includes C 4 v (rotations and ﬂips) and SimCLR for Gpm includes rolling translations and ﬂips.
Revised: SimCLR for Blob includes C 4 v (rotations and ﬂips) and SimCLR for Gpm includes rolling translations and ﬂips.

Original: E-SimCLR encourages the features to be sensitive to the selected transformation (four-fold translations for Blob and four-fold rotations for Gpm), which improves the performance of SimCLR.
Revised: E-SimCLR encourages the features to be sensitive to the selected transformation (four-fold translations for Blob and four-fold rotations for Gpm), which improves the performance of SimCLR.

Original: On the contrary, adding the selected transformation to SimCLR, as indicated by “+ Transform”, increases the error ofSimCLR.
Revised: On the contrary, adding the selected transformation to SimCLR, as indicated by “+ Transform”, increases the error of

Original: Error bars are reported for 3 different choices of training data.

Original: Supervised (frozen) refers to the impractical situation of freezing a random backbone and ﬁne-tuning the DOS-predictor.

Original: [Table]
Revised: [Table]

Original: Continuous group experiment.
Revised: Continuous group experiment.

Original: In all experiments shown so far, we dealt with ﬁnite groups of transformations.
Revised: In all experiments shown so far, we dealt with ﬁnite groups of transformations.

Original: To show that E-SSL generalizes beyond the ﬁnite group setting, we also explore transformations from a continuous group.
Revised: To show that E-SSL generalizes beyond the ﬁnite group setting, we also explore transformations from a continuous group.

Original: An example is the scaling transformation where every pixel of the input unit cell is scaled by the same positive factor.
Revised: An example is the scaling transformation where every pixel of the input unit cell is scaled by the same positive factor.

Original: More speciﬁcally, this set of positive scaling transformations g ( s ) x
Revised: More speciﬁcally, this set of positive scaling transformations g ( s ) x

Original: = s x deﬁnes a continuous group G = { g ( s ) | s ∈ R + } which leaves the DOS labels invariant due to the physics of the problem and normalization applied when preprocessing the dataset (Loh et al., 2021).
Revised: = s x deﬁnes a continuous group G = { g ( s ) | s ∈ R + } which leaves the DOS labels invariant due to the physics of the problem and normalization applied when preprocessing the dataset (Loh et al., 2021).

Original: In our experiment, we uniformly sample s ∈ (1 , s max ] and apply the inverse with probability 0.5 (i.e. we cap the scaling factor to a maximum of s max = { 5 , 10 } for numerical stability during training and we apply up-scaling and down-scaling with equal probability).
Revised: In our experiment, we uniformly sample s ∈ (1 , s max ] and apply the inverse with probability 0.5 (i.e. we cap the scaling factor to a maximum of s max = { 5 , 10 } for numerical stability during training and we apply up-scaling and down-scaling with equal probability).

Original: To encourage equivariance to this group, we simply predict the scale factor applied to the input using L1 loss (i.e. the ﬁnal layer of the predictor p 2 is a single node).
Revised: To encourage equivariance to this group, we simply predict the scale factor applied to the input using L1 loss (i [Table] e. the ﬁnal layer of the predictor p 2 is a single node).

Original: In Table 11, we show results after ﬁne-tuning the backbone and the DOS predictor network with 3000 labelled samples.
Revised: In Table [Table] we show results after ﬁne-tuning the backbone and the DOS predictor network with 3000 labelled samples.

Original: We observe similar trends to Table 4; encouraging sensitivity to scaling produces the lowest error and including scaling to SimCLR increases the error.
Revised: We observe similar trends to Table 4; encouraging sensitivity to scaling produces the lowest error and including scaling to SimCLR increases the error.

Original: To isolate the effect of scaling transformation, the remaining physics-governed invariances excluding scaling (translations, rotations and mirrors) are used in SimCLR and the invariance part of E-SimCLR for both datasets.(cid:96) [Table] [Table]
Revised: To isolate the effect of scaling transformation, the remaining physics-governed invariances excluding scaling (translations, rotations and mirrors) are used in SimCLR and the invariance part of E-SimCLR for both datasets.

Original: [Table]
Revised: [Table]

Original: better encourages be sensitive scaling are

Original: PhC Dataset

Original: Supervised

Original: [Table]
Revised: [Table]

Original: Blob ( s max = 10 )

Original: [Table]
Revised: [Table]

Original: ( s

Original: [Table]= [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: F F LOWERS -102 E XPERIMENTS
Revised: G F LOWERS -102 E XPERIMENTS

Original: In order to study the importance of the assumption in Proposition 1 we perform an experiment with a dataset that might not be amenable to such an assumption at ﬁrst sight.
Revised: In order to study the importance of the assumption in Proposition 1 we perform an experiment with a dataset that might not be amenable to such an assumption at ﬁrst sight.

Original: We choose the Flowers102 dataset (Nilsback & Zisserman, 2008), because at ﬁrst sight the dataset might not beneﬁt from four-fold rotations in E-SSL.
Revised: We choose the Flowers102 dataset (Nilsback & Zisserman, 2008), because at ﬁrst sight the dataset might not beneﬁt from four-fold rotations in E-SSL.

Original: Therefore, the datasets complements the experiments we performed for CIFAR-10 and ImageNet.
Revised: Therefore, this dataset complements the experiments we performed for CIFAR-10 and ImageNet.

Original: Experimental setup.
Revised: Experimental setup.

Original: We train SimCLR and E-SimCLR.
Revised: We train SimCLR and E-SimCLR.

Original: We use the same optimization hyperparamateres from the experimental setup for our CIFAR-10 experiments.
Revised: We use the same optimization hyperparameters from the experimental setup for our CIFAR-10 experiments.

Original: We downsize the images to 96x96 resolution and use the standard ResNet-18, instead of its modiﬁed version for CIFAR-10.
Revised: We downsize the images to 96x96 resolution and use the standard ResNet-18, instead of its modiﬁed version for CIFAR-10.

Original: For the data augmentation in I-SSL, we use the same RandomResizedCropping as in CIFAR- (with size 96 of the crops being the only difference), the same Color Jittering and Random horizontal ﬂips as in the CIFAR-10 experiment.
Revised: For the data augmentation in I-SSL, we use the same RandomResizedCropping as in CIFAR-10 (with size 96 of the crops being the only difference), the same Color Jittering and Random horizontal ﬂips as in the CIFAR-10 experiment.

Original: We report the kNN accuracy in (%) on the validation set.
Revised: We report the kNN accuracy in (%) on the validation set.

Original: We study both four-fold rotations and four-fold translations as transformations for invariance/ equivariance.
Revised: We study both four-fold rotations and four-fold translations as transformations for invariance/ equivariance.

Original: Four-fold rotations are chosen following the hypothesis that most of the data points should be invariant to rotation.
Revised: Four-fold rotations are chosen following the hypothesis that most of the data points should be invariant to rotation.

Original: Four-fold translations are chosen, because of our observation that most of the data points are centered, just like in the Blob PhC dataset.
Revised: Four-fold translations are chosen, because of our observation that most of the data points are centered, just like in the Blob PhC dataset.

Original: The λ for predicting four-fold translations is 0.01 and for four-fold rotations is 0.5 (chosen from a grid search among { 0.001, 0.01, 0.1, 0.5, 1.0, 2.0 } ).
Revised: The λ for predicting four-fold translations is 0.01 and for four-fold rotations is 0.5 (chosen from a grid search among { 0.001, 0.01, 0.1, 0.5, 1.0, 2.0 } ).

Original: Results.
Revised: Results.

Original: Following our observations in Figure 1, we observe that encouraging insensitivity to fourfold rotations and translations, by adding the transformations to the SimCLR data augmentation, worsens the SimCLR baseline.
Revised: Following our observations in Figure 1, we observe that encouraging insensitivity to fourfold rotations and translations, by adding the transformations to the SimCLR data augmentation, worsens the SimCLR baseline.

Original: In contrast, using these transformations for E-SSL improves the baselines and further shows the utility of E-SSL for real-world data.
Revised: In contrast, using these transformations for E-SSL improves the baselines and further shows the utility of E-SSL for real-world data.

Original: We even observe beneﬁt from encouraging equivariance to four-fold rotations, which is against the intuition that rotations should be invariant.
Revised: We even observe beneﬁt from encouraging equivariance to four-fold rotations, which is against the intuition that rotations should be invariant.

Original: This is probably due to the fact that some images in the dataset are not truly rotationally invariant (see examples of the data points in Figure 8). 
Revised: This is probably due to the fact that some images in the dataset are not truly rotationally invariant (see examples of the data points in Figure 8). [Figure]

Revised: H R ELATIVE O RIENTATION P REDICTION

Revised: In our experiments we demonstrate that if a shared biased between the train and test sets exists, we should exploit it via the E-SSL training objective.

Revised: However, in some scenarios, the class label of the downstream tasks depends on the orientation (e

Original: [Figure]
Revised: [Figure]

Revised: g., classifying road signs) and the current

Revised: E-SSL method may not be very useful

Revised: [Figure]

Revised: because both x and T g ( x ) exist in the data.

Revised: This situation invites a natural generalization of our method in the spirit of (Agrawal et al., 2015).

Revised: If x and T g ( x ) exist in the data then we can modify E-SSL minimally to form a useful objective.

Revised: In particular, we can set the objective for p 2 to predict the relative orientation between two data points, i.e. given x , we form T g (cid:48)

Revised: ( x ) by sampling T g (cid:48) from G uniformly, and then predict g (cid:48) from p 2 ( z ; θ p 2 ) , where z

Revised: =

Revised: [ f ( x ) , f ( T g (cid:48)

Revised: ( x ))] is the concatenation of the two representations.

Revised: This modiﬁcation requires minimal change to our framework.

Revised: To test the usefulness of the modiﬁed method when x and T g (cid:48) ( x ) exist in the data, we artiﬁcially modify CIFAR-10 so that any rotation of an image can appear in the dataset.

Revised: We consider the downstream task of predicting the rotation orientation of an image, which clearly depends on the orientation of the image.

Revised: Our hypothesis is that the modiﬁed E-SimCLR will be better than SimCLR, which is what we observe in our results below.

Revised: Intuitively, SimCLR is not able to capture the orientation of the image, while the modiﬁed E-SimCLR is able to, because the latter predicts relative orientation.

Revised: The experimental setting is as follows: we pretrain for 100 epochs.

Revised: The predictor for equivariance’s input dimension is doubled, because we concatenate two representations.

Revised: We set λ to 0.4.

Revised: All other hyperparameters are the same as the rest of the CIFAR-10 experiments.

Revised: The downstream task is 4way rotation orientation classiﬁcation.

Revised: Using pre-training with SimCLR on the standard CIFARdataset, we obtain baseline linear probe (%) accuracy 67.1 ± 0.1.

Revised: Using our modiﬁcation of E-SSL on the same experimental setting, we obtain 71.2 ± 0.1 .

Revised: linear probe accuracy, which is a sizable gain and points to the promise of the relative orientation prediction as future work.

Revised: The relative orientation prediction scenario is also highly relevant in other domains such as in photonic crystals.

Revised: For example, we can modify the PhC setup and remove the “orientation bias” of the datasets while predicting a different property, the band structures.

Revised: Unlike the DOS, the PhC band structures are not invariant to rotations and so we can consider the group of four-fold rotations.

Revised: This setup would ﬁt the scenario described above since 1) both x and T g (cid:48)

Revised: ( x ) exist in the data due to the lack of bias and 2) the downstream task is sensitive to rotation.

Revised: We will explore the above framework on this setup in future work.