Paper: Efficient Model-Agnostic Multi-Group Equivariant Networks (ICLR-2024)

Reviewer comment (Experiments):
For the results in Table 1, why do you consider the input as ordered? What if instead we consider the input as a set of images, and apply Maron et al 2020? How would it compare?

Author response:
If we ensure that the fusion layers are identical and share across channels, then our framework directly extends for sets instead of sequences. But Maron et al. 2020 is not applicable here since they assume that the group elements acting on each image is the same. Thus, Maron et al. is not equivariant to the group actions considered here. Whereas, in Tab. 1, the group elements applied are chosen independently.

In the appendix of Maron et al. Sec. B, they also provide some results on applying independent group actions to each image/data point, but they assume the group actions are transitive. We make no such assumptions. In fact, the transitive assumption does not hold on the image classification tasks considered.

Reviewer comment (Experiments):
Related to Point 2, I think the multi-image task does not really test the first scenario, as the input groups are the same and therefore Maron et al 2020 can be directly applied.

Author response:
As discussed above, please note the framework of Maron et al. 2020 does not work in our problem setting.

Reviewer comment (Novelty):
The motivation for the work is unclear. It is not evident why one would want to make an existing trained model equivariant. While this assumption underlies the downstream language experiments, these experiments are not a convincing demonstration. The task appears contrived and does not align with the broader equivariant literature.

Author response:
**Reviewer:** The first one is on the motivation. It is unclear to me why we would want to make an existing trained model equivariant.

**Response:** Please note that making pretrained models equivariant is of crucial importance with the rise of large pretrained models. These large models are usually not trained with equivariances required for certain downstream tasks, but training them from scratch with group equivariance is extremely expensive and hence not often an option. Yet, using these large foundation models in downstream tasks shows excellent performance. Thus, the motivation of this work (focused on making pretrained model equivariant) is of great importance. Indeed, this research direction has gained recent importance as evident from several papers [1-4] in this direction. These are clearly mainstream papers on group equivariance published in respected conferences (NeurIPS and AAAI). Hence, we respectfully disagree that the considered task does not fit in the broader equivariant literature.

Further, we do not believe that the language task is contrived. The problem of addressing social biases in generated text from language models and the used framework [5] is widely studied. Further, addressing the issue of bias in natural language generation is not trivial, as shown in several works such as [6]. Taking inspiration from the formulation of [1] that shows that equivariance can provably debias natural language with respect to defined groups, we aim to extend these equivariance techniques to larger social groups much more efficiently to help safer deployments of language models. One can further note that intersectionality has been a pernicious problem in debiasing, and our mathematical framework is very natural therein. 

[1] Basu et al. "Equi-tuning: Group equivariant fine-tuning of pretrained models." AAAI 2023.

[2] Basu et al. "Equivariant few-shot learning from pretrained models." NeurIPS 2023

[3] Kim et al. "Learning probabilistic symmetrization for architecture agnostic equivariance." NeurIPS, 2023.

[4] Mondal et al. "Equivariant Adaptation of Large Pre-Trained Models." NeurIPS 2023.

[5] Sheng et al. "The Woman Worked as a Babysitter: On Biases in Language Generation." EMNLP-IJCNLP 2019

[6] Steed et al. "Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models." ACL 2022

Reviewer comment (Experiments):
The paper’s claims regarding large discrete product groups are unsupported. In the multi-input experiments (Table 1), the authors use $N=4$ and the $C_4$ group, which is neither a large product nor a large individual group. In the compositional generalization experiments, the product groups are composed of three smaller groups, each of size two, with the largest product group having size eight—again, not large. A similar criticism applies to the intersectional fairness experiments. The overall claim and motivation for working with large product groups is therefore weak. Moreover, simple methods like frame averaging and canonicalization could be applied in these settings and should at least be included as baselines.

Author response:
**Reviewer:** Large discrete product groups?
In addition, to the lack of coherent motivation, I also found the claims in the paper to be unsupported. A large theme of this work is on building equivariance across large discrete product groups. In the multi-input experiments (Table 1) you have N=4 and the C_4 group. This is not a large product or large individual group. Similarly, in the compositional generalization experiment ... largest product group considered is of size eight". Again this is not a large product group. A similar criticism can be attributed to the intersectional fairness experiments. So I find the entire claim and motivation for doing this work lacking. 

**Response:** Our main contribution is intended to be theoretical, where the experiments are given to validate our theoretical claims. Hence, we considered products of small groups rather than large discrete product groups, and show that the gap in computational complexity (in terms of memory consumption) is as comes from our theory. We are happy to rewrite the experiments section to indicate that the experiments are meant to validate the theoretical findings and that we did not work with very large product groups since that extra computational cost was unnecessary for theory validation.

Reviewer comment (Experiments):
The experimental design heavily relies on Basu et al. 2023’s results without considering that the experimental setup in that paper might also be problematic for validating the present work’s claims. The paper omits standard equivariant benchmarks from seminal works, such as molecular datasets with $S_n$ and $SE(3)$, where $S_n$ represents a much larger discrete group and discrete subgroups of $SE(3)$ could be considered. The lack of such benchmarks is concerning.

Author response:
**Reviewer:** Experimental design and results The entire choice of experiments in this paper leans heavily on Basu et. al 2023's results. But I don't think the authors thought that the experimental setup in that paper might also be problematic in terms of highlighting the claims and goals. 

**Response:** Please note that our intent in designing experiments is to validate the advantages gained from our algorithm as compared to existing model-agnostic frame-averaging-based methods such as equitune. Clearly, the theoretical advantage is huge (sum of group sizes vs. product of group sizes) and is validated in our experiments. 

It is unclear to us how the experimental setup in [2] is problematic and would appreciate further guidance from the reviewer. We believe that the setup in [2] perfectly fits our framework and allows us to demonstrate the advantage gained by using our method on product groups used in the setup of [2].

**Reviewer:** First, there is really no standard equivariant benchmark from many of the seminal equivariant papers ... the lack of this benchmark is alarming.

**Response:** Response: 
Please note that works such as [8] considered a similar problem of constructing approximately equivariant networks for larger groups using networks that are equivariant to smaller groups. Our work, just like [8], has mostly focused on images (and additionally, text).


We believe the reviewer is suggesting an SE(3)xS_n experiment in the context of our second formulation where one single model has group action applied that is a product of multiple smaller groups. However, we are unclear as to how this experiment would elucidate properties of our method. Our method is useful when we have large non-equivariant pretrained models and we want good finetuning or zero-shot performance. For example, if the zero-shot performance requires multiple passes through the pretrained model, our method shows how to reduce the number of forward passes while showing excellent equivariant performance.  We are happy to receive specific guidance from the reviewer on molecular experiments (e.g. with non-equivariant pretrained models) that would provide insights into our method.  

[8] Maile et al. "Equivariance-aware Architectural Optimization of Neural Networks." ICLR 2023

Reviewer comment (Novelty):
The paper ignores an existing body of work on latent space disentanglement via linear symmetries, initiated by Higgins et al. (2018), which comes with benchmarks well-suited to the setting of this work. This omission is questionable.

Author response:
**Reviewer:** ...to show discrete product group structure then there is a large body of work on latent space disentanglement via Linear symmetries which started from the seminal work of (Higgins et. al 2018)...

**Response:** Unfortunately, again, it is unclear to us how these suggested experiments fit with the problem we pursue. We would again kindly request the reviewer to provide further detailed guidance on suggested experiments.

Reviewer comment (Experiments):
The choice of compositional generalization and intersectional fairness tasks through group-theoretic notions appears contrived and runs counter to the practical goal of scaling up equivariant models for product groups.

Author response:
**Reviewer:** ... choice of compositional generalization and intersectional fairness via group theoretic notions quite a contrived ...

**Response:** We would reiterate that the chosen tasks of solving compositional generalization and intersectional fairness are important.


We would point the reviewer to a large literature on solving compositional generalization using group equivariance and data augmentation, e.g. [9-12], which use the SCAN dataset as a standard dataset to check compositional generalization.


Similarly, our framework for intersectional fairness and its evaluation is fully based on the popular work of Sheng et al. [12]. 


Moreover, we think these experiments perfectly fit the main goal of the paper: to provide an efficient method for performing equivariant finetuning and zero-shot evaluation from pretrained models. In fact, our entire theory was developed to tackle these problems: one of our main goals is to make large pretrained models fair and unbiased. Thus, showing that these fairness and compositional generalization tasks can be performed more efficiently with negligible drop in performance shows that these models can be more easily deployed in practice while guaranteeing their fairness and compositional generalization.


[9] Gordon et al. "Permutation equivariant models for compositional generalization in language."  ICLR 2019

[10] Zhao et al. "Toward Compositional Generalization in Object-Oriented World Modeling." ICML 2022.

[11] White, and Cotterell. "Equivariant Transduction through Invariant Alignment." Proceedings of the 29th International Conference on Computational Linguistics. 2022.

Reviewer comment (Evaluation):
Regarding results, the proposed approach performs worse than Equitune in some experiments (e.g., Fig 2) without a convincing justification, aside from claims about better scaling properties with respect to the number of groups in the product. Since the tested products are not large, this reasoning is unconvincing.

Author response:
**Reviewer:** With regard to results, ... the proposed approach does worse than equitune in some experiments (e.g. Fig 2). The authors do not have a convincing argument on why this is acceptable ...

**Response:** Our method is designed so that we obtain better much computational complexity at the expense of negligible drop in accuracy. The gained computational benefit (sum of group sizes vs. product of group sizes) is huge. Note that the improvement in computational complexity comes with a small drop in accuracy, which reduces with increasing model size (cf. Fig 4C). 


The drop in accuracy occurs for the following reason. Consider G to be a product of two smaller groups. Then equitune corresponds to a double summation of symmetrization over a pretrained model. On the other hand, we avoid this double summation by adding some invariance to the input and symmetry in the output in equation 6. This invariance in the input causes small drops in the accuracy. But, we believe that the gained computational complexity overshadows the negligible drop in performance that decreases with increase in model size (as seen from Fig. 4). The drop in accuracy is now explained in detail in the new Sec. G in the appendix.

Reviewer comment (Experiments):
Several obvious baselines are missing from the experiments. These include: (1) an actual equivariant architecture trained from scratch (not fine-tuned) post hoc in the same manner, and (2) baselines such as frame averaging and canonicalization.

Author response:
**Reviewer:** Secondly, obvious baselines are missing. These include having an actual equivariant (architecture) model that is trained (not fine-tuned) post hoc in this manner. Also, frame averaging and canonicalization should be included.

**Response:** It is confusing to us how we should add models trained for several experiments such as GPT2 or CLIP which are large pretrained models. The purpose of the work is to leverage the many advantages of pretrained models. Hence, we are not sure how training equivariant networks from scratch is a suitable baseline. Rather, the baseline, as we have already considered, should be other model-agnostic methods used with pretrained models.

Reviewer comment (Experiments):
Suggested additional experiments and evaluations include:  
1) An experiment where $S_n$ is in the product, possibly on a molecular task, as many examples of such benchmarks exist in the equivariant literature.  
2) An experiment with latent product symmetry, which could be more convincing in the authors' setting.  
3) Inclusion of the missing baselines mentioned in the weakness section.  
4) A comparison of computational cost (iterations/second, FLOPs, training time, inference time, etc.) between the proposed approach and frame averaging, as their costs are likely similar given the small group size.

Author response:
We thank the reviewer for their prompt response.

We did think through several setups of experiments including QM9 for molecules, and latent space product symmetries. However, we believe these experiments do not directly fit our framework. E.g., for the case of molecules with S_n x SO(2) symmetry, it is not clear if separating out S_n and SO(2) symmetries will lead to expressive neural networks. 

We think the best impact of the work is in the case of discrete product groups that are not parametrizable (hence, most methods designed for large groups such as canonicalization do not easily apply), e.g., our language-based experiments.

We would highly appreciate it if the reviewer could kindly elaborate on their suggestion for suitable experiments to improve the impact of our work.