Paper: Self-Supervised Detection of Perfect and Partial Input-Dependent Symmetries (ICLR-2024)

Reviewer comment (Experiments):
Although the spirit of this work is fundamentally setting the grounds of the technique and the theory behind it, I believe experiments should explore data beyond MNIST. In that sense, I would strongly suggest the authors to explore (at least) CIFAR-like data, with 32x32 RGB images.

Author response:
We updated our manuscript showing how our proposed framework can be applied to different distributions such as Gaussian or cyclic, and more generally, arbitrary (unimodal) symmetric distributions. We have been on a time crunch to extend our method to other distributions and groups, but we believe that our method can be extendable to other groups such as O(2) and to 3D rotations, and we are working on having these experiments on more realistic datasets for the camera-ready version. We also thank the reviewer for suggesting to test our model on data that does not strictly follow a uniform distribution, which resulted in new results as reflected on the updated manuscript. Lastly, the use of the NT-Xent loss has been removed from our method, as the improvements have proved to be marginal or non-existent after further hyperparameter optimization of the rest of the parts of the method. Therefore, no ablation study on this loss and the consistency loss is needed anymore, as both are no longer part of the method. Regarding CIFAR10, While the method promises to be applicable to other datasets such as the ones tested in the IE-AE work, in datasets with high intra-class variability e.g. CIFAR10, objects belonging to the same class may not share a G-invariant representation that connects them through a group action. In such cases, the standard IE-AE is unable to capture this relationship, and consequently, our method is also limited, as it relies on IE-AEs. We have included this discussion in the limitations section of our method (Section 6).

Reviewer comment (Experiments):
Another way to show the limitations of the method (and strengths hopefully!) is to perform an experiment on data that does not strictly obey a uniform distribution of group elements.

Author response:
Understanding the functioning of $L_2$ and how it leads to centers of symmetry in the canonical representation is important to address the raised concerns that follow. At first sight, minimizing $L_2$ in combination with the IE-AE loss seems to only ensure that the canonical representation of an equivalence class is a member of that class, as opposed to the standard IE-AE approach where the canonical representation is chosen arbitrarily. However, it can be proved that the element of the class that minimizes $L_2$ is in fact the center of symmetry of the class. This derivation is included in the updated manuscript (see Appendix B.1). Now let's connect this reasoning to your questions.

Reviewer comment (Experiments):
While the proposed method seems general enough to process any symmetry, the only setting discussed is learning symmetry boundary of SO(2). Moreover, the assumption that the rotation symmetry is uniformly distributed in an interval seems restrictive.

Author response:
We updated our manuscript showing how our proposed framework can be applied to different distributions such as Gaussian or cyclic, and more generally, arbitrary (unimodal) symmetric distributions. We hope that this update shows that our method is general enough to be compared with other methods such as the standard IE-AE. We recognise that we have been on a time crunch to extend our method to other distributions and groups, but we believe that our method can be further extendable to other groups such as O(2) and to 3D rotations, and we are working on having these experiments for the camera-ready version.

Reviewer comment (Evaluation):
It is not clear whether or to what extend different levels of symmetry exists in datasets. In the abstract, the authors motivated the task of detecting the level of symmetry by the example that pictures of cars and planes exhibit different levels of rotation in CIFAR-10 dataset. However, this difference is not quantified.

Author response:
The comparison between cars and planes intended to serve as an intuitive example of two objects that can appear with different symmetry levels in the same dataset, but we indeed did not quantify this difference in CIFAR10. The purpose of this is to offer an intuitive example of two objects that would usually appear with different symmetries: cars, showing very little rotational symmetries and planes, which when photographed from below, show full rotational symmetries.

Reviewer comment (Evaluation):
While experiments in the paper demonstrate that the proposed method works as intended, it is not clear how these tasks are relevant in real-world applications. The practical value of detecting symmetry levels could be made more convincing by discussing possible scenarios where predicting levels of symmetry gives useful information or is expected to improve downstream model performance.

Author response:
In Section 5, we presented some of the practical applications of detecting levels of symmetry: building out-of-distribution symmetry classifiers and using symmetry standardization as a pre-processing tool to improve the generalization and performance of non-equivariant models. The benefits of learning the levels of symmetry were further motivated by the works presented in the Introduction section.

Reviewer comment (Experiments):
Since the proposed method builds on invariant-equivariant autoencoder, comparing against it in experiments would help demonstrate the advantage of predicting input-dependent levels of symmetry, particularly for the out-of-distribution symmetry detection and improving models with standardization experiments. Currently there is no baseline comparison for any of the experiments.

Author response:
Comparison of the IE-AE with our method is included now in the updated manuscript for the symmetry standardization experiments (Section 5). Note that a comparison with the IE-AE for the out-of-distribution symmetry experiment can not be performed, as this type of practical application can only be obtained by jointly using the $\Theta$ network and the modified IE-AE. In effect, a regular IE-AE can not be used to build out-of-distribution symmetry detectors, as it lacks the ability to predict an input-dependent symmetry distribution to compare each input to. Lastly, the figures of the symmetry level predictions in Section 5 have been modified for improved readability, as suggested.

Reviewer comment (Theory):
Suppose a symmetry is outside the symmetry boundary, i.e. does not present in the training set. Do we want the neural network to be equivariant to it? If so, it seems detecting symmetry at a sample-level is less useful in avoiding over constraining the model. If not, doesn’t this limit the model’s generalization ability?

Author response:
As mentioned in our introduction, prior research suggests that full equivariance can overly constrain models, leading to reduced performance when the symmetry group defined for G-equivariance does not align with the actual symmetries present in the data. However, models exhibiting full equivariance demonstrate enhanced robustness when encountering test data with symmetry distributions that differ from the training data (e.g. an SO(2)-equivariant network trained on MNIST will have higher accuracy in RotMNIST during inference than a regular CNN trained on the same dataset). Conversely, these fully equivariant models may underperform compared to partially equivariant models, when exposed to test data that falls within the symmetry constraints of the training data (which is usually the case). In such cases, partially equivariant models are desired. This presents a trade-off: generalization to out-of-distribution symmetries versus optimized performance for in-distribution symmetries. Interestingly, combining our out-of-distribution symmetry detectors with both fully and partially equivariant models could be a promising approach. This might enable a model to adaptively balance between full and partial equivariance, leveraging the benefits of both for improved generalization and performance. We hope that this paragraph answers your question, and we thank you for raising an interesting point that poses a valuable addition to our manuscript and future work.

Reviewer comment (Theory):
In section 3.2, why does minimizing $d(\psi(x),e)$ encourage $\psi(c_{[x]})=e$?

Author response:
This is an excellent question. It can be somewhat tricky to understand why minimizing our proposed objective function $L_2 = d(\psi(x), e)$ results in fulfilling the constraint  $\psi(c_{[x]})=e$. At first sight, minimizing $L_2$ in combination with the IE-AE loss seems to only ensure that the canonical representation of an equivalence class is a member of that class, as opposed to the standard IE-AE approach where the canonical representation is chosen arbitrarily. However, it can be proved that the element of the class that minimizes $L_2$ is indeed the center of symmetry. This derivation is included in the updated manuscript (see Appendix B.1). As the new derivation shows, the objective $L_2$ is able to create the right conditions for symmetry level prediction, for any unimodal, symmetric distribution. Our method is not limited to this class of distributions only, as the same minimization objective can also lead to predict cyclic symmetry levels (as presented in the revised manuscript) and possibly other groups and distributions. We are working on having these experiments for the camera-ready version.

Reviewer comment (Writing):
Method section writing could be simplified, by taking rotation detection as a running example, and relating it with the notations intermittently.

Author response:
We recognise that it can be challenging to follow the definitions without a running example. Some of the most crucial terms defined in the method section now show a running example, in order to make the presented results easier to understand (please refer to the updated method section in the manuscript, new changes in the manuscript are coloured in red).

Reviewer comment (Novelty):
The novelty in the method is lacking as it builds up on the IE-AE with existing Partial G-CNN layers, coupled with contrastive learning approach. All of which are already existing in the literature.

Author response:
The method builds on top of the IE-AE and modifies it to provide it with new, desirable properties that are necessary to learn the level of symmetry of each input, as stated in Proposition 3.1. Partial G-CNNs layers are not used in any part of the method, but rather motivate both the need for determining the level of symmetry, as well as the approach of modelling the level of rotational symmetry as a single-parameter estimation of uniform rotations centered at the identity. This nuance is now made clearer in the updated manuscript (Section 2.3). Similarly, the use of contrastive learning is optional, does not contribute towards the novelty of the proposed method, and has been removed in the revised manuscript, as we found the improvements to be marginal after further hyperparameter optimization of all the proposed components.

Reviewer comment (Theory):
Motivation for applying contrastive learning is unclear, why can’t pseudo labels be directly used to predict the result? What is the benefit of applying contrastive learning?

Author response:
The motivation behind contrastive learning is that minimizing a contrastive objective gives greater separation in the latent space, which is useful to create higher quality neighbors and improve the accuracy of the proposed method. However, the application of contrastive learning is not necessary for the method to be able to calculate the level of symmetries. Importantly, the use of the NT-Xent loss has been removed from our method, as pointed out in the previous response.

Reviewer comment (Presentation):
First and third rows in Fig. 4 are same. What is the reason behind it? Is it the input?

Author response:
First and third rows are indeed inputs, while the rows underneath represent each input’s reorientation into their canonical representation. This figure is now modified to clearly define and name each row.

Reviewer comment (Novelty):
While the decision to generalize the setting of previous works (especially w.r.t. Partial-G-CNNs and IE-AE) to deal with partial, per-item distributions, in an unsupervised manner, the result is much more limited and less applicable as a result. The assumption of pure rotation, that is uniformly distributed over a single segment (practically a 1-parameter estimation problem), is very limited and unnatural. This can be compared to IE-AE that handle discrete and continuous groups, including rotations, translations, and permutations.

Author response:
+->We updated our manuscript showing how our proposed framework can be applied to different distributions such as Gaussian or cyclic, and more generally, arbitrary (unimodal) symmetric distributions. We hope that this update shows that our method is general enough to be compared with other methods such as the standard IE-AE.

Reviewer comment (Experiments):
The limited setting is very obvious in the testing, which includes only uniformly rotated versions of MNIST. The above mentioned methods include demonstrating the potential on some slightly more realistic datasets (such as CIFAR and Shapenet), where there is no control or no way to model the true underlying symmetries. Especially when considering the potential of unsupervised learning, it is a little disappointing to see it demonstrated on controlled data that could allow fully supervised training.

Author response:
+-> In our work, we followed experimental settings set by previous works such as the IE-AE, which used controlled symmetries in datasets like RotMNIST, Tetris Shape dataset with random SO(3) rotations, and Shapenet with random 90º rotations. Indeed, we recognize the importance of applying our method to more complex datasets such as the ones in this work, and the truth is that time constraints have limited the extent of our method in this initial phase, both regarding its extension to other distributions and experiments on more datasets. We believe that our method can be extendable to distributions in other groups such as O(2) and 3D rotations, and we are working on having these experiments for the camera-ready version.

Reviewer comment (Theory):
There is not much of a discussion as to how you could drop some of the assumptions, if desired. For example, the uniformity is currently what enables rather easily estimating the center of the range and its boundaries. What would happen with other types and distributions of symmetries?

Author response:
+-> As mentioned in the first response, it is indeed not necessary to have a uniform distribution in the data. The main assumption of the method is the existence of centers of symmetry, which enables our model to detect, more generally, arbitrary unimodal, symmetric distributions. The assumption can be further relaxed to detect the symmetry level of other distributions such as cyclic distributions (as shown in the updated manuscript) and possibly, other distributions and groups.

Reviewer comment (Experiments):
The need for (and the quality of) the self-supervision and the boundary prediction network $\Theta$ itself is unclear. It is not the case that the network $\Theta$ can generalize to a new dataset, so for the given dataset, you still have to run the pre-training and then, given a new sample, you could just predict the pseudo-label itself. This might be more precise, but perhaps less efficient because of the nearest neighbor computation. This should be further explained.

Author response:
+-> The $\Theta$ network is designed for generalization to new datasets (meaning generalization to test datasets, if we are understanding you correclty) without requiring retraining or pre-training on these datasets. Regarding the reasoning about relying on pseudo-labels for predictions during testing: If we were to rely solely on pseudo-labels as predictions during the test phase, we would indeed have to re-run the pre-training phase on the test dataset. This approach contradicts the standard testing methodology, where the model should generalize to new data without training again. Additionally, using pseudo-labels as predictions would be problematic for smaller test datasets or individual input samples. For instance, if we only have a small set of 200 test samples with several classes in it, using these for neighbor computations would likely yield poor-quality neighbors. More critically, predicting the symmetry level of single inputs would be impractical, as the input would be its own neighbor, rendering the approach ineffective. In contrast, in our approach, once the $\Theta$ network is trained, it is capable of inferring the symmetry level of new, unseen inputs independently. The IE-AE's role is primarily during the training phase, where it aids in estimating the symmetry levels that are subsequently used to train the $\Theta$ network. After this training phase, the IE-AE becomes redundant and can be discarded. The trained $\Theta$ network is what we use for direct inference on new data, which is how we obtained the symmetry level prediction results for the experiments shown. In summary, the design of the $\Theta$ network is such that it eliminates the need for recalculating or relying on pseudo-labels during the testing phase, addressing in this way the concerns of efficiency and practicality. We updated the manuscript to include this nuance in our explanation (see Section 3. All changes in the revised manuscript are coloured in red).

Reviewer comment (Reproducibility):
The way of calculating nearest neighbors is not specified (or perhaps I missed it). This is critical for identifying the equivalence classes.

Author response:
+-> The methodology for calculating nearest neighbors is indeed crucial for estimating the symmetries in the equivalence classes and is briefly mentioned in the manuscript. The explanation in the manuscript states: ``Let $N_{k,d}{=}N_k: X \rightarrow P(X)$ be a function that maps each input $x$ to the set $N_{k,d}(x)\subset X$ of $k$-neighbors around $x$ in the $G$-invariant latent space $Z_\mathrm{inv}$ as measured by some distance metric $d$." To put this idea in more detail: the set $N_k(x)$ consists of the $k$ elements $y$ of the dataset whose G-invariant embeddings $\eta(y)$ are nearest to the G-invariant embedding of the input $\eta(x)$, based on the chosen distance metric d. For instance, to compute $N_2(x)$, we first calculate the distance between every $\eta(y)$ in $\eta(X)$ and the embedding $\eta(x)$. Then, we select the two elements $y\in X$ that have the smallest distances in $\eta(X)$ to $\eta(x)$. We hope this answers the question, and have updated the manuscript to include a more explicit explanation of this computation (Section 3.4).

Reviewer comment (Reproducibility):
How are nearest-neighbors ("semantically similar inputs") found?

Author response:
+-> The methodology for calculating nearest neighbors is indeed crucial for estimating the symmetries in the equivalence classes and is briefly mentioned in the manuscript. The explanation in the manuscript states: ``Let $N_{k,d}{=}N_k: X \rightarrow P(X)$ be a function that maps each input $x$ to the set $N_{k,d}(x)\subset X$ of $k$-neighbors around $x$ in the $G$-invariant latent space $Z_\mathrm{inv}$ as measured by some distance metric $d$." To put this idea in more detail: the set $N_k(x)$ consists of the $k$ elements $y$ of the dataset whose G-invariant embeddings $\eta(y)$ are nearest to the G-invariant embedding of the input $\eta(x)$, based on the chosen distance metric d. For instance, to compute $N_2(x)$, we first calculate the distance between every $\eta(y)$ in $\eta(X)$ and the embedding $\eta(x)$. Then, we select the two elements $y\in X$ that have the smallest distances in $\eta(X)$ to $\eta(x)$. We hope this answers the question, and have updated the manuscript to include a more explicit explanation of this computation (Section 3.4).

Reviewer comment (Theory):
In generating the pseudo-labels: Why is the estimator that simply takes the mean angle considered to be robust, and in particular "more robust to outliers" than what?

Author response:
+-> The decision to use the method of moments estimator for generating pseudo-labels is motivated by its robustness to outliers, compared to other potential estimators. Let's consider the alternative of a maximum likelihood estimator (MLE) for the distribution of rotation angles. The MLE is the maximum angle observed in the nearest neighbors set $N_k(x)$. However, this approach is particularly sensitive to outliers. For example, if any element in $N_k(x)$ has an erroneously predicted high rotation angle, the pseudo-label for $x$ would be disproportionately influenced by this outlier. In contrast, the method of moments estimator in the absolute-valued distribution, which we employ, calculates the pseudo-label as two times the mean of all angles in $N_k(x)$. This averaging effect mitigates the impact of any occasional anomalous angles, as these outliers are diluted when calculating the mean. The previous reasoning is now included in the updated manuscript (see Appendix C).

Reviewer comment (Experiments):
In experiments: Have you tried to demonstrate the method on more realistic data? (which would perhaps require dropping the uniform or continuous assumptions, or perhaps trying other more general classes of symmetry).

Author response:
+-> While the method promises to be applicable to other datasets such as the ones tested in the IE-AE work, in datasets with high intra-class variability e.g. CIFAR10, objects belonging to the same class may not share a G-invariant representation that connects them through a group action. In such cases, the standard IE-AE is unable to capture this relationship, and consequently, our method is also limited, as it relies on IE-AEs. We have included this discussion in the limitations section of our method (Section 6).