Paper: Lie Group Decompositions for Equivariant Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
It does not matter too much to be frank, as SOTA is not everything, but please have a look at this paper which to my knowledge remains the SOTA on Affnist (figure 5 for instance) https://aaai.org/ojs/index.php/AAAI/article/download/5785/5641. With 174K parameters (64,16,16,16,10) they have achieved 98.1% accuracy on Affnist when trained at 99.7% of MNIST accuracy. So you might want to update that, considering the number of parameters entailed too.

Author response:
Not at all! We've updated the paper to make our contributions more clear.
The affNIST dataset was simply the standard benchmark for robustness to "larger" transformations (beyond rotations) on image classification tasks. We want to thank the reviewer for bringing the particular Capsule paper mentioned in "Weaknesses" to our attention. We will incorporate it in the next revision of the paper. Our models' performance can be further improved by taking more Monte Carlo samples of the group, however we will focus on this once we revise the paper with further experiments.

Reviewer comment (Novelty):
To what extent have you based the novelty of your paper on achieving SOTA performance on Affnist?

Author response:
Not at all! We've updated the paper to make our contributions more clear.
The affNIST dataset was simply the standard benchmark for robustness to "larger" transformations (beyond rotations) on image classification tasks. We want to thank the reviewer for bringing the particular Capsule paper mentioned in "Weaknesses" to our attention. We will incorporate it in the next revision of the paper. Our models' performance can be further improved by taking more Monte Carlo samples of the group, however we will focus on this once we revise the paper with further experiments.

Reviewer comment (Experiments):
Is there a scope that your method can be used in tandem with CapsNets?

Author response:
Thank you for the question, that is an interesting proposal! Capsule methods which rely on a discretization of the convolution integral such as [1] (see Section 4 of their paper) could in theory make use of the integral decomposition and sampling techniques proposed in our work. Further considering the framework of [1], one could make use of the Lie algebra parametrization to obtain tangent vectors which identify each specific group element, and allow for the calculation of the Frechet mean (in an equivariant manner) and the geodesic distance, which would play the role of the averaging operator $\mathcal{M}$ and the distance $\delta$ required by [1] (Section 2.1). We will consider this for possible future work.

[1] - Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. Advances in neural information processing systems, 31, 2018.

Reviewer comment (Writing):
My main concern of the paper is its mathematical denseness. I would recommend the authors to focus on the main message, and defer unimportant details to the appendix. For example, the condition of part (2) of Theorem 4.1 is probably too general, and it is not helpful for understanding the paper.

Author response:
We agree that the paper presentation can be improved and have started incorporating the suggested changes to make the presentation more clear (see updated version), and will continue to incorporate further changes in the final revision.
Part (2) of Theorem 4.1 is indeed a very general result, however we consider it an essential component of the overall framework as conditions (1) and (2) of this theorem categorize the major classes of decompositions that are employed in the context of invariant integration on topological/Lie groups. For the larger groups $\text{SL}(n, \mathbb{R})$ and $\text{GL}^{+}(n, \mathbb{R})$ which are decomposed both at the group and algebra level it is Theorem 4.1 that allows us to formalize the change of variables in the integral formula so as to eventually construct a sampling mechanism on these groups using their subcomponents. 
There are several ways these groups can be decomposed, with the Cartan decomposition being one such choice, which is what we've employed. The Cartan decomposition is a decomposition of the form $G = G/H \times H$ as in Theorem 4.1 (1), while the standard decomposition for the semi-direct product can be understood in terms of Theorem 4.1 (2). The reviewer is correct that presenting only one example for such a general theorem might not be well-motivated. In the revised manuscript we included included further discussion of possible decompositions which make use of Theorem 4.1 (2) (see Section B.6.1 of the Appendix), highlighting the fact that depending on the Lie group of interest, the integral decomposition can be realised in several ways. The decomposition into a product of subgroups (even if one of the subgroups is also a homogeneous space) will lead to a different set of tools being available when we go beyond the integral decompositions and try to understand what Lie algebra decomposition would correspond to the group decomposition.

Reviewer comment (Theory):
One of the issues the paper aims to address is the global non-surjectiveness of the exponential map. However, since most (group) convolutional filters are only defined locally (on a neighborhood of the identity), does the existence of local diffeomorphism already suffice?

Author response:
Our work fits within the recent class of methods which rely on constructing 
the convolution kernel analytically using the group Lie algebra.
Within this framework, to define a kernel taking in Lie algebra vectors 
corresponding to the group elements $g^{-1}\tilde{g}$ of the convolution, one 
has to construct a map \textbf{from the group to the algebra} (denoted by $\xi^{-1}: G \to \mathfrak{g}$ in our paper). 
One exception to this case is when the group is compact and Abelian, as in that 
case we can work only in the Lie algebra, as remarked in the paper.
If the Lie group exponential map is not surjective, we cannot formally 
make the change of variables of Equation (8), and the map $\xi^{-1}$ might 
not be defined. 
The reviewer is correct that in some cases one could address this 
by simply restricting the integration domain 
to the injectivity radius of the group exponential, which for the Lie groups employed 
in the context of machine learning will be the matrix exponential, 
making $\xi^{-1}$ the matrix logarithm.

To understand the motivation behind attacking the problem of non-surjectivty, first  
we must consider the degree to which the group exponential `covers' the 
group of interest, and for which group elements is $\xi^{-1}$ defined.
This is formally addressed by Proposition A.4 in the Appendix. For many groups of interest 
(e.g. scaling transformation, $\text{SL}(n, \mathbb{R})$ or $\text{GL}^{+}(n, \mathbb{R})$) the complement of the image 
of the group exponential is not a set of measure zero.
If one decides to opt for a restriction to the injectivity radius of the exponential, 
the expressivity of the model suffers, as the space of equivariant functions 
that can be approximated by the resulting method is reduced. Of course, the degree 
to which this is a problem is dependent on the particular application 
and the Lie group employed.
Another issue to consider is that if one is employing a Monte Carlo approximation 
of the convolution/cross-correlation integral then it is possible that 
random group elements $g^{-1}\tilde{g}$ might be sampled outside of domain 
of definition of the matrix logarithm which would then require the 
sampling mechanism to reject these samples (incurring an additional computational 
cost), as otherwise we would have numerical errors introduced in the 
approximation independent of optimization process. In some sense, the sampling 
mechanism itself has to be designed with this restriction in mind.
The reviewer is correct that in practice our filters/kernels are defined locally. 
We state this more precisely in the updated version of the paper and indicate that 
we can assume at least one of $f(\cdot)$ or $k(\cdot)$ in 
$\int_{G} f(\tilde{g})k(\tilde{g}^{-1}g) d\tilde{g}$ can be taken 
to have compact support. The advantage of our framework is that one does not have to 
assume that the support of the feature maps/kernels is tied in any way to the 
injectivity/surjectivity radius of the group exponential.

Reviewer comment (Writing):
Page 5, line 4 after "Factorizing the Haar measure", for general subsets $P\subset G$ and $K\subset G$, what does it mean to have "G-invariant" Radon measure. Although the authors later say that $P$ will always be a $G$-homogeneous space (so the $G$-action on $P$ is essentially an action on the quotient space), it would be much better to state this assumption in the beginning.

Author response:
Thank you for highlighting this particular part of the presentation. 
What we should have made more clear (and have clarified in the revision) is that a topological space $X$ on which a group $G$ acts can have a $G$-invariant Radon measure even when $X$ itself is not a group. If $X=G$ (and we are describing the action of the group $G$ on itself), then the $G$-invariant Radon measure on $X = G$ is simply the Haar measure of $G$ (Thm. A.1). When $X$ is a topological space on which $G$ acts but it is not necessarily $G$ itself, an invariant measure on $X$ (invariant with respect to the action of $G$) may still exist, and it is simply referred to as a $G$-invariant (Radon) measure. For example the SPD matrices are a homogeneous space of $\text{GL}^{+}(n, \mathbb{R})$, with the $\text{GL}^{+}(n, \mathbb{R})$-invariant measure being described in Section A.10 of the appendix. In any case, this particular distinction has now been made clear in the paper explicitly (see Remark A.2 of the Appendix).

Reviewer comment (Theory):
On page 4, the authors say $H = SO(n)$ is abelian. Is this the case for $n\ge 3$?

Author response:
Thank you for pointing this out. We have clarified that the BCH simplification was employed in previous work for $n=2$.

Reviewer comment (Theory):
The theory seems to apply only to $H = GL^+(n, R)$ or $H = SL(n, R)$. Does it apply to a general $H\subset GL(n, R)$?

Author response:
It is indeed the case that the tools are applicable in a more general setting, and the original purpose of the paper was to present a general framework that allows for one to deal with arbitrary groups in the context of integration/sampling/parametrization with the construction of conv. kernels being a primary application. We mentioned only briefly in the introduction that the methodology if applicable to any Lie group with finitely many connected components, however the degree of generality/applicability of our method was not clearly communicated. In the revised version of our paper we have made an attempt to clarify the 'framework' aspect of our approach and its components, as well as to point towards references which show that most matrix Lie groups $G \leq \textnormal{GL}(n, \mathbb{R})$ fulfill the conditions of this framework. Namely, non-compact groups can be decomposed into their maximal compact subgroup and some Euclidean factor. Affine groups can be decomposed/parametrised as indicated in the paper. Compact groups with surjective exp. maps can immediately be employed within this framework (and with tools employed by previous work).

Reviewer comment (Theory):
Can this construction be extended to representations beyond regular representations of $G$? For high-dimensional $G$, such as $E(n)$ with large $n$, sampling from $G$ will suffer from COD.

Author response:
Sampling efficiency is actually one requirement we had in mind when developing our construction. Note that the cases $n = 2$, or $n = 3$ cover most machine learning applications of interest. In this case the largest spaces we would need to sample from would be GL(2, R) (4 free parameters) and GL(3, R) (9 free parameters). Of course as the reviewer points out, because our approach involves a numerical integration procedure the convergence is depends on the number of MC samples. The advantages of our methodology is that if one can find a change of variables via the product space decomposition and the subcomponents are independent (which occurs when the measure decomposes as a product of measures), then the simulation problem can be transferred from the larger space to the smaller-dimensional factors. Then, we can consider the problem of sampling on these spaces. Note that, if we fully embrace the idea that the problem has reduced to a MC integration of the convolution integral, we can make use of classical variance-reduction techniques to improve our sample efficiency. This is already shown to some degree in our experiments, when we decompose GL+(2,R) as SPD(2,R) x SO(2,R), on the rotation group we use a quasi-monte carlo [1] approach and sample equi-distant points in [0, 2\pi]. A similar procedure could be done for SO(3) where we could pre-compute the samples (using for example some discretization such as the icosahedron) and keep the SPD factor as being sampled randomly to obtain better convergence rates. Other methods for variance reduction could be applied, as shown for example in [2]. In any case, the solution in the paper is in some sense the most basic form of MC integration, and there is still space to improve sample efficiency.

[1] - https://en.wikipedia.org/wiki/Quasi-Monte_Carlo_method

[2] - Lagrave, Pierre-Yves, and Frédéric Barbaresco. "Adaptive Importance Sampling for Equivariant Group-Convolution Computation." Physical Sciences Forum. Vol. 5. No. 1. MDPI, 2022. - https://www.mdpi.com/2673-9984/5/1/17

Reviewer comment (Novelty):
The advantage of this approach over MacDonald et al 2022 is not made sufficiently clear, since both approaches can also handle Lie groups with non-surjective exponential maps.

Author response:
> Could the authors elaborate on the primary advantages of this integration method over that of MacDonald et al 2022, which also handles the case of Lie groups with non-surjective exponential maps? The related work mentions memory requirements, but it is not clear to me how this method circumvents these issues (or what the precise issues are).

Besides the comments in the previous response (which point to Appendix A.3), we can make some remarks. Both methods employ a change of variables in the integration procedure. In our case, this happens at the group-level, and we reduce the numerical simulation problem from a larger space to a product of (independent!) factor spaces. This means that we can construct independent sampling mechanisms on the individual factors to obtain a sample on the original space. In general the smaller spaces will be easier to deal with, and for example the orthogonal groups SO(2) and SO(3) (which would appear in the most common applications of our approach for $n \in \{2,3\}$) already have a large amount of literature dedicated to methods which can be used to sample efficiently on these spaces with respect to their Haar measure. Without the decomposition tools specific to $SO(n)$ could not be employed. The same is true for the case of SPD matrices, which are also a manifold which has a wide variety of applications. 
The reason we avoid exponential memory constraints comes down to not having to reorder the convolution operation (as described in A.3). We only need to sample effectively from 1 layer before the current one, similar to previous proposals on regular group cnns.

Reviewer comment (Presentation):
The paper’s presentation could be significantly improved. In addition to typos, some of which are noted below, the writing leaves something to be desired overall. For example, the main body of the paper is very dense in mathematical background and details. It would be helpful to clearly and concisely express the main ideas of the paper earlier on and to give only the important intuitions, and to defer many such details to later in the paper or the appendix.

Author response:
> The paper’s presentation could be significantly improved. In addition to typos, some of which are noted below, the writing leaves something to be desired overall. For example, the main body of the paper is very dense in mathematical background and details. It would be helpful to clearly and concisely express the main ideas of the paper earlier on and to give only the important intuitions, and to defer many such details to later in the paper or the appendix.

Thank you for the feedback! We have incorporated the reviewer's feedback in the current updated version of the paper streamlining the presentation and only presenting the essential concepts in the main text, while also making more clear the our contributions.

Reviewer comment (Experiments):
Enforcing equivariance to Lie groups with non-surjective exponential maps could be much better motivated — applications are not discussed very often, and the datasets used are synthetic (e.g. affine transformations of MNIST). When does such equivariance arise in practice?

Author response:
> Enforcing equivariance to Lie groups with non-surjective exponential maps could be much better motivated — applications are not discussed very often, and the datasets used are synthetic (e.g. affine transformations of MNIST). When does such equivariance arise in practice?

We agree that further motivation for this class of transformations would benefit the paper and intend on discussing this in our next revision. There are several ways to motivate the applicability of such groups, one of them being as the reviewer points out specific problems where more exotic symmetries are present. Another way the applicability of these groups could be motivated is if on desires additional out-of-distribution robustness, independent of the degree of symmetry in the dataset. For example, if one is building a computer vision system and the geometric variability in the dataset is low, lets say only rotations, it does not mean that the robustness of the system has to be limited only to those transformations that are found in the dataset (consider the case of a self-driving car system trying to detect lanes, pedestrians, etc.). In some cases neither the training set nor the test set can indicate exactly what the symmetry group of the underlying real-world object is. An equivariant network is advantageous even in this case, as the entire network can be made invariant to all affine transformations, i.e. we can obtain a form of out-of-distribution robustness irrespective of the symmetry found in the present form of the dataset. We agree that there are objects especially in physics and chemistry which are assumed to be governed by rigid-body motions. However, even in this case deformations can occur for example in molecule interactions/dynamics. Other areas of application are pose estimation with full affine degrees of freedom, image registration (especially in the context of medical imaging) and robotic motion/manipulation.

Reviewer comment (Writing):
Here are several typos I encountered while reading:  
* Page 1, abstract: backwards quotation mark on the word larger  
* Page 2, fourth line: backwards quotation mark on the word larger  
* Page 2, related work: “Recent proposal” —> “Recent proposals”  
* Page 2, bottom line: “maps can be defined AS k-channel”  
* Page 3, between equations 5 and 6: “ecompasses”  
* Page 4, in Lie algebra parametrization: “approxiamtion” and “encompasses recent proposalS”  
* Page 7 before Theorem 4.4: “to obtain GL(n,R)-sampleS”

Author response:
Thank you for taking the time to review our paper and for the constructive feedback! We have updated the paper, with the current revision focusing on clarifying our contributions, streamlining the presentation and incorporating as much feedback as possible from the reviewer's comments. We plan on soon adding another revision which focuses on further experimental validation and numerical equivariance error analysis, while also providing a broader comparison with related work. The typos have also been address (with the exception of the abstract, as it can only be modified in case of acceptance for the Camera Ready version).

Reviewer comment (Presentation):
Mathematical rigor: very dense, difficult to read, too much re-derivation of previous results. A good part of sec. 3, Background, in __continuous group equivariance__ and some of __Lie algebra parametrization__ can be cut and moved to the appendix. Sec. 4 can also be shortened, like derivations in __layer definition__ in 4.2. The density of the math makes the paper hard to read.

Author response:
We have updated the manuscript based on the reviewer's feedback. We agree that the original presentation was too dense in non-essential details and made an effort to streamline the presentation.

Reviewer comment (Theory):
The paper calls $SO(n)$ abelian, which is incorrect for $n>2$ (page 4, sec 4).

Author response:
Thank you for pointing this out. We have clarified that the BCH simplification was employed in previous work for $n=2$.

Reviewer comment (Novelty):
Other literature: Many Gconv approaches do not sample the group this way (see question): Clebsch-Gordon Nets (Kondor et al NeurIPS 2018), Lorentz group (Bogatskiy, ICML 2020), EMLP (Finzi 2021) or Lie algebra conv (Dehmamy, NeurIPS 2021).

Author response:
We will discuss these works in our next revision.

Reviewer comment (Theory):
How are methods relying on irreps affected by the expm not being surjective? Does it matter in those cases?

Author response:
Generally irrep methods are used in the context of compact groups where the exponential map is surjective. In the work of EMLP (Finzi 2021) which you mention, the authors construct equivariant MLPs by solving a system of constraints where the resulting linear system is solved using the singular value decomposition (if the representation is finite dimensional, whereas we work with an infinite-dimensional representation). From their paper, one can review Appendix H where they also employ a form of product of exponentials in the case where the group of interest $G$ is "larger" (and have non-surjective exp maps).

Reviewer comment (Theory):
For the Haar measures to factorize to $\mu_K \otimes \mu_P$, do we need $G=PK$ or semidirect or direct product? A number of different decompositions are used in 4.1 and 4.2 and the requirements are not fully clear.

Author response:
Both cases are possible! In the semi-direct product case the decomposition is somewhat trivial in the euclidean space case, as this is simply the translation factor of the convolutions. The idea of the paper is that we have a general framework where we can perform a decomposition G = PK for multiple cases : P is homogeneous space, P is a subgroup (and we have a direct product decomposition) and P is a subgroup that is normal (and we have a semi-direct product decomposition). All cases are covered by this framework. The specific decomposition described by the reviewer is an application of the more general theorem, and a specified decomposition of the groups G = GL(n) and G = SL(n). The decomposition leading to a decomposition of the Haar measure into a product of invariant measures is actually a stronger condition than just a decomposition at the group or algebra level. The objective was to not only find a decomposition, but one which leads to a product space decomposition of independent factors, so as to be able to sample on the individual space and produce a sample on the product space and therefore on the original group. We have tried to make this more clear in the revision as well as provide further examples of possible decompositions (see Appendix B.6.1).

Reviewer comment (Theory):
I see that $G=PK$ can be done when $P$ or $K$ is a normal subgroup. In the $SL(n)$ to $\mathrm{SPos} \times SO(n)$, is $SO(n)$, is one of the factors also a normal subgroup?

Author response:
The original presentation in the paper was indeed not clear and the exact representation of each of the factor spaces might be confused. It is not necessary for one of the components to be a normal subgroup (this is the case when we are working with the semi-direct product $\mathbb{R}^{n} \rtimes G$ for example). Actually in the decompositions employed one of the components is not even a group (the SPD factor), it is a homogeneous space of the original group which we decompose. So one the P factor can be a group (normal or not) or a homogeneous space. In the case of the Cartan Decomposition we should also mention that the SPD factor is indeed diffeomorphic to another group (upper triangular), which could be used and would lead to another decomposition of GL(n) (as shown in B.6.1).

Reviewer comment (Theory):
Even when expm is not surjective, the product of multiple $g_i = \mathrm{expm}(X_i)$ can produce any arbitrary group element, as I understand it. Doesn't this mean that multi-layer Gconv using expm should in principle be sufficient to get good sample coverage on the group? In other words, is your decomposition really necessary?

Author response:
Thank you for the question! We have attempted to clarify this more precisely in the revision of the paper. Considering each question separately:

> Even when expm is not surjective, the product of multiple $g_i = \mathrm{expm}(X_i)$ can produce any arbitrary group element, as I understand it. 

In some sense this is already happening. The reviewer is correct that when the group exponential is not surjective, it is still possible to take a **product** of exponentials to represent an element. The problem is that this simple representation will in general not offer any way for the parametrization map $\xi^{-1}: G \to \mathfrak{g}$ to be constructed, and will not simplify the integration problem (see for example Remark B.11 in the updated manuscript which explains that a first alternative that was considered for the group exponential is the Riemannian exponential. However the issues lies with the fact that the Riemannian logarithm does not have a close form expression and we would then require an optimization process to happen at each layer for each forward pass). The Cartan/Polar decomposition is actually used to represented elements $g \in G$ as $g = e^{X}e^{Y}$ (see Section B.7 for the full details), only that X, Y are not just elements of $\mathfrak{g}$, but rather we decompose $\mathfrak{g}$ into a direct sum of vector spaces and represent each element separately, which allows for the construction of the map $\xi^{-1}$ and the decomposition corresponds to the group-level decomposition which also realizes a decomposition of the integral!

> Doesn't this mean that multi-layer Gconv using expm should in principle be sufficient to get good sample coverage on the group? In other words, is your decomposition really necessary?

The issue is that once one single layer is not invariant/equivariant the property is lost by induction for the later parts of the network.

Reviewer comment (Reproducibility):
The number of samples for Monte Carlo integration is not clear to me. When the authors say 10 samples are used for the affine group, do they mean 10 samples for both rotations and scales combined? How many rotations and scales?

Author response:
The number of rotation and SPD matrices sampled are the same in this sampling scheme. For example, let $n = 2$ and suppose we want to produce 10 samples. To produce a sample $A \in G$ ($G = GL+(2, R)$ or $G = SL(2, R)$), we sample on the individual factors and have a 10 x 2 x 2 tensor R of 10 random SO(2) matrices and a 10 x 2 x 2 tensor S of 10 random SPD matrices which we multiply (S^{1/2}R = A) to get a 10 x 2 x 2 tensor A of G-matrices, i.e. 10 samples of GL+(2, R) or SL(2, R) matrices distributed according to the Haar measure. This works if the rotation matrices are sampled uniformly with respect to the Haar measure and the SPD matrices are sampled with respect to the GL(n, R) or SL(n, R) invariant measure on this manifold, as described in detail in Appendix B.6. In this case the individual factors of the product space are independent random variables and the joint distribution factorizes, which allows for this sampling scheme. We will try to make this more in clear in the next revision.

Reviewer comment (Theory):
I don't fully understand why the proposed group decomposition technique is able to estimate the correlation integral well with only 10 samples while the work by MacDonald et al. need 100.

Author response:
Thank you for the question! We have tried to make this more clear in the revision. Section A.3 in the appendix has a more in-depth analysis of this approach, focusing on the theoretical presentation of their method (as we cannot now if further implementation issues exist which lead to their reduced sample efficiency).

Reviewer comment (Evaluation):
The analysis of the numerical stability of the method is largely missing. In particular, methods that rely on approximating integrals over groups will always be approximately equivariant. While increasing the number of samples could reduce this error below floating-point precision, such an analysis is not present in the paper. A curve illustrating the equivariance error as a function of the number of Monte Carlo samples would be pertinent in this context. Although sizeable equivariance errors might be manageable in the context of image classification tasks, this is not the case for all types of application tasks where group convolutions are relevant. Specifically, in physics simulations, equivariance errors are related to the non-conservation of physical quantities. The proposed approach should be contrasted with approaches that use linear operators to evaluate the group integral, such as [1] and [2], which are exactly equivariant (even for non-compact groups) but require large tensorial operations (for [1]) or ad hoc representation theory for each group (for [2]). I encourage the authors to briefly discuss these points in the text and add an analysis of numerical stability in the form of a plot to the appendix.

Author response:
Thank you for the positive feedback! We have made an update to the manuscript focused on clarifying general details and plan to make another revision where the equivariance error analysis is presented as a factor of the number of samples, and related work is discussed more in-depth (addressing the other questions in your review which are not mentioned here).

Reviewer comment (Theory):
In the paper, only the symmetrization to the canonical group action is explicitly mentioned. Extending the matrix elements to other finite-dimensional group representations should be straightforward, provided that the correct basis is determined. I am curious why this was not presented in full generality.

Author response:
Because of the mathematical denseness already existing in the paper we decided to limit its scope and leave possible applications of these tools in the context of finite-dimensional representations for these groups to future work.

Reviewer comment (Theory):
See my point regarding numerical stability.

Author response:
Thank you for the positive feedback! We have made an update to the manuscript focused on clarifying general details and plan to make another revision where the equivariance error analysis is presented as a factor of the number of samples, and related work is discussed more in-depth (addressing the other questions in your review which are not mentioned here).

Reviewer comment (Theory):
This might be a standard result already, but I would like to have a formal view of the convergence of your approach. Do you think it would be possible to obtain an explicit bound on the equivariance error as a function of the number of Monte Carlo samples, the sizes of the factors in the group decomposition, and the size of the representation? To be more precise, I am looking for something akin to this bound: $|| \int_{H} \int_{K} f(\rho(h) \rho(k) \cdot x) d\mu_{H}(\rho(h)) d\mu_{K}(\rho(k)) - \frac{V}{N} \sum_{i} f(\rho(h_{i}) \rho(k_{i}) \cdot x), \rho(h_{i}) \sim \mu_{H}, \rho(k_{i}) \sim \mu_{K}|| \leq f(N, dim(\rho), dim(H), dim(K) )$. Using that, one could ask what kind of factorization gives the best bound.

Author response:
We will try to include a discussion on this aspect in the next revision. For certain groups [1] much more explicit bounds can be obtained, however we are not aware of a result of this form for the groups employed in the paper. 

[1] - https://en.wikipedia.org/wiki/Quantum_t-design#Unitary_t-Designs