Paper: Interpreting Equivariant Representations (ICLR-2024)

Reviewer comment (Novelty):
Are there other significant examples where an isometric cross-section can be found? Or is the latent graph representation a very unique example? If that is the case, maybe the paper should focus more on the random projection approach because it has more applications.

Author response:
**1. Are isometric cross-sections rare?**

Unfortunately, we do believe that isometric cross sections are rare. However, permutation actions are commonly encountered, and we therefore find that the result warrants inclusion.

Reviewer comment (Theory):
I had a hard time reading and contextualizing this paper. For example, I'm not fully sure I understand why one would use equivariant representations when the inputs do not follow the G-equivariance of the representations. For eg, if x and g.x are similar inputs, then an equivariant representation is perhaps not the correct one to analyze such data in the first place. Why wouldn't one, instead just choose some mapping f that is invariant?

Author response:
**1. Why don’t you just use an invariant mapping when x and g.x are similar?**

We don’t quite understand the reviewer’s comment – we don’t study cases where the inputs x and g.x are similar, we study cases where they are different – e.g. permuted graphs or rotated images. In such applications, when your model predicts a structured output (e.g. a reconstructed graph, or an image segmentation) you want your model to be equivariant, not invariant. 

You are right, however, that invariant models are also applied to this type of data – the difference consists of whether or not the output should be transformed analogously to the input (as when the output is a segmentation or a reconstruction), or whether the output does not change when the input is transformed (e.g. when the output is a class). When building classification networks, the standard approach is to stack equivariant layers and then end the model with an invariant layer thus obtaining a fully invariant model.

Reviewer comment (Presentation):
I did not fully understand Section 4, it read more like a related work section and felt out of place.

Author response:
**2. Isn’t Section 4 a related work section?**

Section 4 is indeed a related work section. We positioned this section here to aid the reader, as the mathematical framework introduced in the previous sections is needed to  fully explain how our methods relate to previously published work. Thank you for pointing out that this was confusing; we have added a comment to make this more clear – please see the updated paper.

Reviewer comment (Evaluation):
Isn't the interpolation experiment self-satisfying? You already claim that after the invariant transform the representation space is convex and thus interpolation makes total sense (and thus the nicer results in Fig 4), whereas this is not the case for equivariant representations (thus results are all over for these in Fig 4). I'm not quite sure what was the purpose of this experiment. Was it to empirically validate the theory?

Author response:
**4. Isn’t the interpolation experiment self satisfying?**

The equivariant representation X is convex. However, when performing convex interpolation, it might suffer from the problem of potentially running into non-aligned representatives of the interpolated graphs, images, etc. The cross section Z_s does not have this problem because it only contains one representative of every graph, and these are geometrically aligned (because Z_s is isometric to the quotient). The fact that the cross section Z_s is a convex subset of  the equivariant representation X (see Proposition 2) makes the interpolation computationally and mathematically very easy. The purpose of the interpolation experiment is to empirically demonstrate that this makes a real (and not just mathematical) difference. In our experience, this is important for conveying a message to the more empirical parts of the community.

Reviewer comment (Writing):
Overall, the paper was a tough read for me. It would be very helpful to start by contextualizing a use case where equivariant representations might need a downstream analysis where equivariance can be misleading, this would help motivate the paper better.

Author response:
**5. Starting by contextualizing the paper would help readability**

Thank you for pointing this out. We have adapted the introduction to emphasize the use of autoencoders for learning self- and semi-supervised embeddings as a prominent use case. Note that equivariant representations are either enforced or encouraged (via augmentation) in any autoencoder, including the VAEs and masked autoencoders commonly used for semisupervised and self supervised learning. These representations are, e.g., typically used for downstream classification, discovery of novel structures like proteins or molecules, but also for specialized tasks such as counterfactual explanations or graph/image editing.