Paper: Lie Neurons: A General Adjoint-Equivariant Neural Network for Semisimple Lie Algebras (ICLR-2024)

Reviewer comment (Evaluation):
Evaluation is the weakest point of this paper. There are three tasks: invariant regression, equivariant regression, and classification of polyhedrons. The first two tasks use synthetic data. Those are useful to know that the model is working as intended, but we cannot know the performance of real problems. The third task is more on the application side, but still synthetic and too clean (e.g. noise free).

Author response:
Thank you for the comments. We have identify several potential applications in question 1. In order to show the immediate connection with practical applications, we compare with Vector Neurons in the $\mathfrak{so}(3)$ regime. Our method can be viewed as a generalization of Vector Neurons to semi-simple Lie algebras, and mostly falls back to Vector Neurons for $\mathfrak{so}(3)$, except for the new Lie bracket nonlinear layer. We compare the performance of our network with the Lie bracket layer against the original Vector Neurons, in the point cloud classification \[3] and registration \[4] tasks. As expected, our performance is very similar to the baseline. This demonstrates that the proposed network generalizes Vector Neurons, while maintaining the performance on $SO(3)$ tasks.

|                |                            |                          |               |
| -------------- | -------------------------: | -----------------------: | ------------: |
|                | Point Cloud Classification | Point Cloud Registration |               |
|                |                            |           Gaussian Noise ($\sigma=0.01$) | Uniform Noise ($scale=0.05$) |
| Vector Neurons |                     73.70% |                   2.2263 |        1.6305 |
| Ours           |                     73.50% |                   2.5343 |        1.1324 |

\[3] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.

\[4] Zhu, Minghan, Maani Ghaffari, and Huei Peng. "Correspondence-free point cloud registration with SO (3)-equivariant implicit shape representations." _Conference on Robot Learning_. PMLR, 2022.

Reviewer comment (Novelty):
Another concern is the range of applications. The problem setting — the transformation itself is a data point — is interesting, but it is not clear what kind of real problems can be formulated in that way. In other words, it's not straightforward to recognize the benefits of this framework from the practitioner's view.

Author response:
Thanks for pointing this out. The motivation is to provide a general equivariant framework for representation learning in problems with Lie group geometric structures. The Lie algebraic representation (adjoint) arises from standard group representation, making the proposed network as general as equivariance with respect to Lie group actions on vector spaces. 

We think the proposed network will have broad applications and has the potential to be adopted for applications such as 3D data processing using Inertial Measurement Units \[1,2] and Point Clouds \[3,4] where data can be viewed directly in $\mathfrak{so}(3)$ Lie algebra.

There are also applications where the inputs are transformations of semi-simple groups, notably $SL(n)$. In computer vision, $SL(3)$ is the homography group \[5,6], which contains affine(2), and 2D affine transformations as its subgroup. Representation learning on $\mathfrak{sl}(3)$ will result in homography-invariant features applicable in numerous detection and classification tasks. For physical systems, $SL(3)$ is the volume-preserving transformation (a generalization of rigid body motion $SE(3)$ in kinematics and dynamics). We conjecture the proposed work can help predict the motion of deformable shapes, even though we do not have experimental results to back up this conjecture, and this is purely our geometric intuition from analytical mechanics.

We envision our network to be applied in different applications in two ways: 

  1. For inputs that are directly transformations of semi-simple groups, we can directly apply our framework. This type of problems often arises in robotic state estimation and tracking, where the states being estimated are represented using elements of Lie groups. An example of this is homography tracking \[7], where measured homography matrices are noisy and fusion across multiple measurements is required. 
  2. If the measurements are raw sensor inputs which lives in some vector space, we need to find an adjoint-equivariant lift similar to Equation 13 in \[8]. This equivariant lift can be found heuristically, or it can be another equivariant neural network. For the special case where the input vectors are subjected to $SO(3)$ actions, such a lift will be identity. This is because the adjoint matrix of $SO(3)$ is exactly equal to the rotation matrix, i.e. $Adm\_R = R$, and $\mathfrak{so}(3)$ is isomorphic to $R^3$. We plan to explore a more general lifting network design in the near future.

\[1] Hamel, Tarek, and Robert Mahony. "Attitude estimation on SO \[3] based on direct inertial measurements." _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._. IEEE, 2006.

\[2] Mahony, Robert, Tarek Hamel, and J-M. Pflimlin. "Complementary filter design on the special orthogonal group SO (3)." _Proceedings of the 44th IEEE Conference on Decision and Control_. IEEE, 2005.

\[3] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.

\[4] Zhu, Minghan, Maani Ghaffari, and Huei Peng. "Correspondence-free point cloud registration with SO (3)-equivariant implicit shape representations." _Conference on Robot Learning_. PMLR, 2022.

\[5] Manerikar, Ninad, Minh-Duc Hua, and Tarek Hamel. "Homography observer design on special linear group SL (3) with application to optical flow estimation." _2018 European Control Conference (ECC)_. IEEE, 2018.

\[6] Mei, Christopher, et al. "Homography-based tracking for central catadioptric cameras." _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2006.

\[7] Hamel, Tarek, et al. "Homography estimation on the special linear group based on direct point correspondence." _2011 50th IEEE Conference on Decision and Control and European Control Conference_. IEEE, 2011.

\[8] van Goor, Pieter, Tarek Hamel, and Robert Mahony. "Equivariant filter (eqf)." _IEEE Transactions on Automatic Control_ (2022).

Reviewer comment (Reproducibility):
I don't wholly understand the setting of platonic solid classification. The homography matrix H seems to be an input, but how is it computed from an image?

Author response:
In the platonic solids investigated in this paper, each face has three vertices. Thus, for each pair of neighboring faces, there are three possible homographies to map between them in the projected image, each assuming a different correspondence between the three vertices of the two faces by rotating the vertices. The three homographies form the input $\mathfrak{sl}(3)$ features for this pair of neighboring faces. For the classification task, the network samples a fixed number of pairs of neighboring faces to form the input feature of this platonic solid. Such homography features are subject to adjoint actions when the camera rotates, to which our model is equivariant.

Reviewer comment (Experiments):
In the platonic solid classification task, the number of input points (the possible face pairs) varies depending on the platonic solids. How can you deal with the difference?

Author response:
As mentioned above, a fixed number (12) of pair-wise homographies are sampled from each platonic solid.

Reviewer comment (Reproducibility):
What is the network architecture used in the platonic solid classification experiment?

Author response:
Thank you for pointing this out. We have added an illustration of the architecture being used in each experiment in Figure 2.

Reviewer comment (Theory):
How can we find $Adm$? For a given G (or its algebra g), is there any systematic way to calculate it?

Author response:
Yes, there is an systematic way to calculate the Adjoint matrix. In this work, we focuses on matrix Lie groups. Since Lie algebra is a vector space, we can always find a basis. With this basis, each element in the Lie algebra can be represented using a $R^n$ vector. The adjoint matrix $Adm$ that maps from an vector $v_1 \in R^n$ to $v_2 \in R^n$ can be computed following the procedure described in Chapter 10.5.2 in [9].

\[9] Chirikjian, Gregory S. Stochastic models, information theory, and Lie groups, volume 2: Analytic methods and modern applications. Vol. 2. Springer Science & Business Media, 2011.

Reviewer comment (Experiments):
From a purely mathematical perspective, the problem is interesting. However, this reviewer’s estimation is that to have a substantial impact, equivariant research needs to have some application to real data. This paper does not really have any experiments on real data, nor did this reviewer see a clear path towards an application. The Platonic solids experiment seemed to be aimed at real applications, but it was a little unclear to this reviewer what these would be. It would be helpful to outline how this type of problem arises in the real world even if no experiments are run with real data (in particular, if homography modeling is a real problem that one can imagine this network architecture being applied to, perhaps more details could be given).

Author response:
**1. Motivation: From a purely mathematical perspective, the problem is interesting. However, this reviewer’s estimation is that to have a substantial impact, equivariant research needs to have some application to real data. This paper does not really have any experiments on real data, nor did this reviewer see a clear path towards an application. The Platonic solids experiment seemed to be aimed at real applications, but it was a little unclear to this reviewer what these would be. It would be helpful to outline how this type of problem arises in the real world even if no experiments are run with real data (in particular, if homography modeling is a real problem that one can imagine this network architecture being applied to, perhaps more details could be given).**

We would like to thank the reviewer for the comment. The motivation is to provide a general equivariant framework for representation learning in problems with Lie group geometric structures. The Lie algebraic representation (with adjoint group action) arises from standard group representation, making the proposed network as general as equivariance with respect to Lie group actions on vector spaces.

We think the proposed network will have broad applications and has the potential to be adopted for applications such as 3D data processing using Inertial Measurement Units \[1,2] and Point Clouds \[3,4] where data can be viewed directly in $\mathfrak{so}(3)$ Lie algebra.

There are applications where the inputs are transformations of semi-simple groups, notably $SL(n)$. In computer vision, $SL(3)$ is the homography group \[5,6], which contains affine(2), and 2D affine transformations as its subgroup. Representation learning on $\mathfrak{sl}(3)$ will result in homography-invariant features applicable in numerous detection and classification tasks. For physical systems, $SL(3)$ is the volume-preserving transformation (a generalization of rigid body motion $SE(3)$ in kinematics and dynamics). We conjecture the proposed work can help predict the motion of deformable shapes, even though we do not have experimental results to back up this conjecture, and this is purely our geometric intuition from analytical mechanics.

 We envision our network to be applied in different applications in two ways: 
  1. For inputs that are directly transformations of semi-simple groups, we can directly apply our framework. This type of problem often arises in robotic state estimation and tracking, where the states being estimated are represented using elements of Lie groups. An example of this is homography tracking \[7], where measured homography matrices are noisy and fusion across multiple measurements is required.
  2. If the measurements are raw sensor inputs which lives in some vector space, we need to find an adjoint-equivariant lift similar to Equation 13 in \[8]. This equivariant lift can be found heuristically, or it can be another equivariant neural network. For the special case where the input vectors are subjected to $SO(3)$ actions, such a lift will be the identity map. This is because the adjoint matrix of $SO(3)$ is exactly equal to the rotation matrix, i.e. $Adm\_R = R$, and $\mathfrak{so}(3)$ is isomorphic to $R^3$. To be clear, a lift from the input to the symmetry group is always possiple, e.g., via sampling, but the challenge is in constructing an equivariant lift. We plan to explore a more general lifting network design in the near future.

\[1] Hamel, Tarek, and Robert Mahony. "Attitude estimation on SO \[3] based on direct inertial measurements." _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._. IEEE, 2006.

\[2] Mahony, Robert, Tarek Hamel, and J-M. Pflimlin. "Complementary filter design on the special orthogonal group SO (3)." _Proceedings of the 44th IEEE Conference on Decision and Control_. IEEE, 2005.

\[3] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.

\[4] Zhu, Minghan, Maani Ghaffari, and Huei Peng. "Correspondence-free point cloud registration with SO (3)-equivariant implicit shape representations." _Conference on Robot Learning_. PMLR, 2022.

\[5] Manerikar, Ninad, Minh-Duc Hua, and Tarek Hamel. "Homography observer design on special linear group SL (3) with application to optical flow estimation." _2018 European Control Conference (ECC)_. IEEE, 2018.

\[6] Mei, Christopher, et al. "Homography-based tracking for central catadioptric cameras." _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2006.

\[7] Hamel, Tarek, et al. "Homography estimation on the special linear group based on direct point correspondence." _2011 50th IEEE Conference on Decision and Control and European Control Conference_. IEEE, 2011.

\[8] van Goor, Pieter, Tarek Hamel, and Robert Mahony. "Equivariant filter (eqf)." _IEEE Transactions on Automatic Control_ (2022).

Reviewer comment (Evaluation):
The reviewer noted a lack of analysis in the experimental section. As this is a novel architecture, it would be useful to see analysis of model performance beyond a single metric. For example, training curves and convergence, confusion matrices, etc. This would allow the reader to better understand what the model is and is not actually good at. For instance, empirically, equivariant models are often more data efficient — is that the case here? It might also make sense to measure the extent to which the proposed model is actually invariant to the action that it is claimed to be invariant to. Some additional experiments would be useful, as the reviewer felt they did not have a strong feel for the method despite reading through three different experiments.

Author response:
**2. Limited experiments: This reviewer has two concerns about the experimental section. The first was the lack of analysis and the second was the lack of strong baselines and experimental breadth.** 

  **a. As this is a novel architecture, it would be useful to see an analysis of model performance beyond a single metric. For example, training curves and convergence, confusion matrices, etc. This would allow the reader to better understand what the model is and is not actually good at. For instance, empirically, equivariant models are often more data efficient, is that the case here? It might also make sense to measure the extent to which the proposed model is actually invariant to the action that it is claimed to be invariant to. Of course, no paper can run all possible experiments, but some additional experiments would be useful. This reviewer felt that they did not have a strong feel for the method, despite reading through three different experiments.**

Thank you for your valuable comments. We conducted extra experiments as suggested by the reviewers to provide extra information to demonstrate the properties of our method. 

First, in terms of data efficiency, we present the training curves of our method and the baseline MLP in Figure 3. We can see that in both invariant and equivariant regression tasks, our equivariant model converges much faster than MLPs, showing the data efficiency of our method. 

Second, in terms of the invariant property, it is shown through the invariance/equivariance error in our paper, which indicates the output difference when input undergoes the adjoint action. Our models present almost zero invariance/equivariance error, indicating that our model is exactly equivariant. Our models have identical performance between Id and SL(3) test sets, supporting that our model is exactly equivariant.

Third, in terms of the confusion matrix, we did not include it in our paper because the accuracy numbers show most information in the platonic solids classification task. Near 100% accuracy shows that our model performs almost perfectly in this task, while the 36% accuracy shows that the MLP is as good as random guesses when the data is augmented by camera rotation. 
Furthermore, to show the performance consistency, we trained each model 5 separate times, and calculated the mean and standard deviation of the performance, as shown in the tables below. Our networks present very small std values in the performance, indicating that the high performance of our networks is reliable and repeatable. 
|               |                       |                    |                   |          |              |          |                  |          |
| :-----------: | :-------------------: | :----------------: | :---------------: | :------: | :----------: | :------: | :--------------: | :------: |
|               |                       | **Invariant Task** |                   |          |              |          |                  |          |
|     Model     | Training augmentation |     Num Params     | Test augmentation |          |              |          | Invariance Error |          |
|               |                       |                    |         Id        |          |     SL(3)    |          |                  |          |
|               |                       |                    |        AVG        |    STD   |      AVG     |    STD   |        AVG       |    STD   |
|      MLP      |           Id          |       136,193      |       0.148       |   0.005  |     6.493    |   1.282  |       1.415      |   0.113  |
|      MLP      |         SL(3)         |       136,193      |       0.201       |   0.010  |     1.119    |   0.018  |       0.683      |   0.006  |
|     LN-LR     |           Id          |       66,562       |      1.30E-03     | 3.24E-05 |   1.30E-03   | 3.25E-05 |     3.60E-04     | 5.48E-05 |
|     LN-LB     |           Id          |       132,098      |       0.557       | 1.87E-04 |     0.557    | 1.87E-04 |   **1.43E-05**   | 1.42E-06 |
| LN-LR + LN-LB |           Id          |       263,170      |    **8.84E-04**   | 2.52E-05 | **8.84E-04** | 2.49E-05 |     4.00E-04     |   0.000  |

Reviewer comment (Experiments):
The reviewer found the paper lacking in strong baselines and experimental breadth. It would improve the work to compare against stronger baselines. For instance, what happens if we try to use some type of adjoint-action augmentation when training the vanilla MLP? It would also be interesting to see how the model performs with different Lie algebras; are some Lie algebras harder to learn on than others? How does performance scale with Lie algebra size?

Author response:
|         |      |             |       |          |        |     |          |    |
| :---------------: | :-------------------: | :------------------: | :---------------: | :------: | :----------: | :------: | :----------------: | :------: |
|          |         | **Equivariant Task** |          |          |         |          |           |          |
|       Model       | Training augmentation |      Num Params      | Test augmentation |          |         |        | Equivariance Error |          |
|      |       |      |         Id        |          |     SL(3)    |          |         |          |
|      |       |      |        AVG        |    STD   |      AVG     |    STD   |         AVG        |    STD   |
|        MLP        |           Id          |        538,120       |       0.011       | 3.53E-04 |     1.318    | 7.08E-02 |        0.424       |   0.003  |
|        MLP        |         SL(3)         |        538,120       |       0.033       | 2.86E-04 |     0.452    | 1.01E-02 |        0.389       |   0.001  |
|      2 LN-LR      |           Id          |        197,376       |       0.213       | 4.07E-05 |     0.213    | 4.08E-05 |      9.32E-05      | 6.65E-06 |
|      2 LN-LB      |           Id          |        328,448       |    **9.83E-10**   | 1.78E-11 | **4.55E-08** | 8.65E-11 |    **6.56E-05**    | 4.22E-07 |
| 2 LN-LR + 2 LN-LB |           Id          |        590,592       |      7.65E-09     | 3.54E-10 |   5.41E-08   | 4.08E-10 |      7.67E-05      | 1.56E-06 |

|      |      |      |     |      |     |
| :-----------: | :---------------------: | :----: | :---: | :-----------: | :---: |
|            | **Classification Task** |        |       |               |       |
|            |        Num Params       |   Acc  |       | Acc (Rotated) |       |
|            |                         |   AVG  |  STD  |      AVG      |  STD  |
|      MLP      |         206,339         | 95.76% | 0.65% |     36.54%    | 0.99% |
|     LN-LR     |         134,664         | 99.56% | 0.23% |     99.51%    | 0.28% |
|     LN-LB     |         200,200         | 99.14% | 0.21% |     98.78%    | 0.49% |
| LN-LR + LN-LB |         331,272         | **99.62%** | 0.25% |    **99.61%**    | 0.14% |

**b. It would also improve the work if stronger baselines were compared against. For instance, what happens if we try to use some type of adjoint-action augmentation when training the vanilla MLP?**

We agree that we can run MLP with adjoint data augmentation for comparison. We have conducted new experiments on this setup. The results are shown in the tables above. MLP with $SL(3)$ adjoint augmentation during training still largely underperforms our equivariant models. 

**c. It would also be interesting to see how the model performs with different Lie algebras, are some Lie algebras harder to learn on than others? How does performance scale with Lie algebra size?**

The model size is independent of the Lie algebra size since the dimension corresponding to the Lie algebra is kept throughout the layers. In terms of the performance, it is tricky to directly compare across different Lie algebras with a consistent metric, but we do not see a scaling problem for the network to process Lie algrbras of different sizes. 
To show the performance on other Lie algebras, we added experiments on $\mathfrak{so}(3)$. Our method can be viewed as a generalization of Vector Neurons to semi-simple Lie algebras, and mostly falls back to Vector Neurons for $\mathfrak{so}(3)$, except for the new Lie bracket nonlinear layer. We compare the performance of our network with the Lie bracket layer against the original Vector Neurons, in the point cloud classification [3] and registration [4] tasks. As expected, our performance is very similar to the baseline. This demonstrates that the proposed network generalizes Vector Neurons, while maintaining the performance on $SO(3)$ tasks.

****

|                |                            |                          |               |
| -------------- | -------------------------: | -----------------------: | ------------: |
|                | Point Cloud Classification (acc, \% ) |  |Point Cloud Registration      (angular error, deg)          |
|                |                                          |           Gaussian Noise $\sigma=0.01$  | Uniform Noise $scale=0.05$|
| Vector Neurons |                     73.70% |                   2.2263 |        1.6305 |
| Ours           |                     73.50% |                   2.5343 |        1.1324 |

****
[3] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[4] Zhu, Minghan, Maani Ghaffari, and Huei Peng. "Correspondence-free point cloud registration with SO (3)-equivariant implicit shape representations." Conference on Robot Learning. PMLR, 2022.

Reviewer comment (Writing):
One aspect of the paper most likely to cause reader confusion is the process of moving between the Lie algebra $\mathfrak{g}$ itself and the corresponding vector space $\mathbb{R}^m$. As pointed out in the Nitpicks section, one problem is that $E_i$ is never defined other than saying it is the image of $e_i$ under the map $\wedge$. But then the definition of $\wedge$ uses $E_i$. More broadly, it might be worth investigating sweeping the distinction between the Lie algebra and its underlying vector space under the rug to cut down on extra notation.

Author response:
**3. Writing and figures: While the reviewer certain aspects of the writing, there are other ways that the the paper could be made more clear:**

  **a. Moving between elements of the Lie algebra and corresponding vectors for processing by the model: One of the aspects of the paper most likely to cause the reader confusion is the process of moving between the Lie algebra g itself and the corresponding vector space R^m. As is pointed out in the Nitpicks section, one problem is that E_i is never defined other than saying it is the image of e_i under the map ∧. But then the definition of ∧ uses E_i. More broadly, it might be worth investigating sweeping the distinction between the Lie algebra and its underlying vector space under the rug to cut down on extra notation.**

We apologize for the confusion. In this work, we focus on finite-dimensional matrix Lie algebras. The wedge is an operation that generates the Lie algebra elements using the canonical basis. Because Lie algebra is a vector space, such a basis always exists. For example, the $\mathfrak{so}(3)$ elements has the form $\begin{bmatrix}0& -\\omega_z & \\omega_y \\\ \\omega_z & 0 & -\\omega_x \\\ -\\omega_y & \\omega_x & 0 \\end{bmatrix}$. One can find the canonical basis as $E_x = \begin{bmatrix} 0& 0 & 0\\\ 0&0&-1\\\ 0&1&0 \end{bmatrix}, E_y = \\begin{bmatrix} 0& 0 & 1\\\ 0&0&0\\\ -1&0&0 \\end{bmatrix}, E_z = \begin{bmatrix} 0& -1 & 0 \\\ 1&0&0\\\ 0&0&0 \end{bmatrix}$. 
With $v=\begin{bmatrix} \omega_x, \omega_y, \omega_z \end{bmatrix}$, we have $(v)^\wedge = \omega_x E_x + \omega_y E_y + \omega_z E_z = \begin{bmatrix}0& -\\omega_z & \\omega_y \\\ \\omega_z & 0 & -\\omega_x \\\ -\\omega_y & \\omega_x & 0 \\end{bmatrix} = W \in \mathfrak{so}(3)$.
The Vee operation is the inverse of this: $(W)^\vee = v$
We will update the paper in Section 3.1 with this example to better convey the concept.
For a detailed description, we kindly refer the reviewer to Equation 10.30 and Chapter 10.2.4 in [9]. In our experiments, we use the basis for $\mathfrak{sl}(3)$ from [10]. 

[9] Chirikjian, Gregory S. Stochastic models, information theory, and Lie groups, volume 2: Analytic methods and modern applications. Vol. 2. Springer Science & Business Media, 2011.

[10] Winternitz, Pavel. "Subalgebras of Lie algebras. Example of sl (3, R)." Centre de Recherches Mathématiques CRM Proceedings and Lecture Notes. Vol. 34. 2004.

Reviewer comment (Presentation):
The reviewer found Figure 1 hard to understand. It might be helpful to somehow include the model in the schematic so it is more clear what is the input/output, group action on the input vs group action on the output, etc.

Author response:
**b. Figure 1: This reviewer found Figure 1 hard to understand. It might be helpful to somehow include the model in the schematic so it is more clear what is the input/output, group action on the input vs group action on the output, etc.**

We thank the reviewers for the valuable suggestions. We have added a model schematic with annotations on input/output equivariant relationship in Figure 1.

Reviewer comment (Writing):
$E_i$ is never defined beyond $E_i = (e_i)^\vee \in \mathfrak{g}$. It would be helpful to say something more explicit.

Author response:
**4. E\_i is never defined beyond E\_i=(e\_i)∨∈g. It would be helpful to say something more explicit.**

We agree with the reviewer. In the experiments, we use the canonical basis defined in \[10]. We will cite this reference in the revised paper.

   \[10] Winternitz, Pavel. "Subalgebras of Lie algebras. Example of sl (3, R)." _Centre de Recherches Mathématiques CRM Proceedings and Lecture Notes_. Vol. 34. 2004.

Reviewer comment (Writing):
$\log$ isn’t actually used in the paper? Perhaps (2) could be removed.

Author response:
**5. Log isn’t actually used in the paper? Perhaps (2) could be removed.**

We agree with the reviewer that the Log is not needed in our context. We will remove it from the paper.

Reviewer comment (Theory):
For equalities in (13), it might make sense to state the reason all this works (the matrix $Adm_a$ and $W$ are being multiplied from different sides?).

Author response:
**6. For equalities in (13), it might make sense to state the reason all this works (the matrix Adm and w are being multiplied from different sides?**

We inherit the design from Vector Neurons. The feature is of shape K\*C, where K is the dimension of the Lie algebra (or the dimension of the Euclidean space for Vector Neurons), and C is the dimension of the feature channels. A linear layer can be viewed as a matrix multiplication on the right (C dimension), while the adjoint matrix acts on the left (K dimension). We thank the reviewer for the suggestion. We will add the above explanation in Section 4.1 in the revised paper.

Reviewer comment (Evaluation):
The experimental results should have confidence intervals attached to them.

Author response:
**7. The experimental results should have confidence intervals attached to them.**

Thank you for pointing this out. We have conducted new experiments to address this issue. Specifically, we train each network separately 5 times and compute the average and standard deviation over the same model trained in different instances. The result is shown in Table 1 - 3.

Reviewer comment (Experiments):
The problem of Lie algebra equivariance does not seem at all well-motivated by applications. This is evidenced by the datasets used in experiments: the first two are quite contrived, and the third one, although geometric in nature, remains synthetic. It is not clear in practice when one genuinely has input data that lives in a Lie algebra, rather than in the vector space associated with an ordinary group representation.

Author response:
**1. The problem of Lie algebra equivariance does not seem at all well-motivated by **applications** ... It is not clear in practice when one genuinely has input data that lives in a Lie algebra, rather than in the vector space associated with an ordinary group representation.**

Thanks for pointing this out. The motivation is to provide a general equivariant framework for representation learning in problems with Lie group geometric structures. The Lie algebraic representation (adjoint) arises from standard group representation, making the proposed network as general as equivariance with respect to Lie group actions on vector spaces. 

We think the proposed network will have broad applications and has the potential to be adopted for applications such as 3D data processing using Inertial Measurement Units \[1,2] and Point Clouds \[3,4] where data can be viewed directly in $\mathfrak{so}(3)$ Lie algebra.

There are also applications where the inputs are transformations of semi-simple groups, notably $SL(n)$. In computer vision, $SL(3)$ is the homography group \[5,6], which contains affine(2), and 2D affine transformations as its subgroup. Representation learning on $\mathfrak{sl}(3)$ will result in homography-invariant features applicable in numerous detection and classification tasks. For physical systems, $SL(3)$ is the volume-preserving transformation (a generalization of rigid body motion $SE(3)$ in kinematics and dynamics). We conjecture the proposed work can help predict the motion of deformable shapes, even though we do not have experimental results to back up this conjecture, and this is purely our geometric intuition from analytical mechanics.

We envision our network to be applied in different applications in two ways: 

  1. For inputs that are directly transformations of semi-simple groups, we can directly apply our framework. This type of problems often arises in robotic state estimation and tracking, where the states being estimated are represented using elements of Lie groups. An example of this is homography tracking \[7], where measured homography matrices are noisy and fusion across multiple measurements is required. 
  2. If the measurements are raw sensor inputs which lives in some vector space, we need to find an adjoint-equivariant lift similar to Equation 13 in \[8]. This equivariant lift can be found heuristically, or it can be another equivariant neural network. For the special case where the input vectors are subjected to $SO(3)$ actions, such a lift will be identity. This is because the adjoint matrix of $SO(3)$ is exactly equal to the rotation matrix, i.e. $Adm\_R = R$, and $\mathfrak{so}(3)$ is isomorphic to $R^3$. We plan to explore a more general lifting network design in the near future.

\[1] Hamel, Tarek, and Robert Mahony. "Attitude estimation on SO \[3] based on direct inertial measurements." _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._. IEEE, 2006.

\[2] Mahony, Robert, Tarek Hamel, and J-M. Pflimlin. "Complementary filter design on the special orthogonal group SO (3)." _Proceedings of the 44th IEEE Conference on Decision and Control_. IEEE, 2005.

\[3] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.

\[4] Zhu, Minghan, Maani Ghaffari, and Huei Peng. "Correspondence-free point cloud registration with SO (3)-equivariant implicit shape representations." _Conference on Robot Learning_. PMLR, 2022.

\[5] Manerikar, Ninad, Minh-Duc Hua, and Tarek Hamel. "Homography observer design on special linear group SL (3) with application to optical flow estimation." _2018 European Control Conference (ECC)_. IEEE, 2018.

\[6] Mei, Christopher, et al. "Homography-based tracking for central catadioptric cameras." _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2006.

\[7] Hamel, Tarek, et al. "Homography estimation on the special linear group based on direct point correspondence." _2011 50th IEEE Conference on Decision and Control and European Control Conference_. IEEE, 2011.

\[8] van Goor, Pieter, Tarek Hamel, and Robert Mahony. "Equivariant filter (eqf)." _IEEE Transactions on Automatic Control_ (2022).

Reviewer comment (Presentation):
There is no end-to-end description of the architecture. Also, the architecture is not motivated in a particularly natural or pedagogical way, e.g. with a discussion of universality, a comparison to other approaches one might take for linear equivariance, etc. Generally speaking, the presentation could be improved — to include more details, to better motivate the equivariance problem via applications, and conditioned on this setting, to better motivate this particular architecture.

Author response:
**3. There is no end-to-end description of the architecture... Generally speaking, the presentation could be improved — to include more details, to better motivate the equivariance problem via applications, and conditioned on this setting, to better motivate this particular architecture.**

As in the response to the previous question, our architecture is largely influence by Vector Neurons. We aim to generalize the Vector Neurons to semi-simple Lie algebras because of the potential applications in robotics, computer vision, and dynamic modeling, as described in our response to question 1. We added an end-to-end illustration of the proposed method as Fig. 2 in the revised paper.
Fig. 1 is also revised to highlight the input-output relationship. Thanks for your suggestions.

Reviewer comment (Experiments):
The only baseline included in experiments is an MLP. It would make sense to include an architecture equivariant/invariant to all linear transformations, for example, or an MLP augmented with Lie algebra transformations. On its own, the MLP seems like a weak baseline.

Author response:
**4. The only baseline included in experiments is an MLP. It would make sense to include an architecture equivariant/invariant to all linear transformations, for example, or an MLP augmented with Lie algebra transformations. On its own, the MLP seems like a weak baseline.**

A linear layer can be made equivariant to all linear transformations by formulating the linear layer and the group actions at matrix multiplications from both sides of the input, as is done in Vector Neurons and our Lie networks. However, there is no existing nonlinear layer that is equivariant to all linear transformations. Nonlinear layers are the key to the expressiveness of a neural network. Thus, there is no practical architecture equivariant/invariant to all linear transformations to serve as a baseline. 

On the other hand, we agree that running MLP with adjoint data augmentation for comparison is a good idea. We have conducted new experiments on this setup. The results are shown in the tables below and in the paper Table 1-3. MLP with adjoint augmentation during training still largely underperforms our equivariant models. 
|               |                       |                    |                   |          |              |          |                  |          |
| :-----------: | :-------------------: | :----------------: | :---------------: | :------: | :----------: | :------: | :--------------: | :------: |
|               |                       | **Invariant Task** |                   |          |              |          |                  |          |
|     Model     | Training augmentation |     Num Params     | Test augmentation |          |              |          | Invariance Error |          |
|               |                       |                    |         Id        |          |     SL(3)    |          |                  |          |
|               |                       |                    |        AVG        |    STD   |      AVG     |    STD   |        AVG       |    STD   |
|      MLP      |           Id          |       136,193      |       0.148       |   0.005  |     6.493    |   1.282  |       1.415      |   0.113  |
|      MLP      |         SL(3)         |       136,193      |       0.201       |   0.010  |     1.119    |   0.018  |       0.683      |   0.006  |
|     LN-LR     |           Id          |       66,562       |      1.30E-03     | 3.24E-05 |   1.30E-03   | 3.25E-05 |     3.60E-04     | 5.48E-05 |
|     LN-LB     |           Id          |       132,098      |       0.557       | 1.87E-04 |     0.557    | 1.87E-04 |   **1.43E-05**   | 1.42E-06 |
| LN-LR + LN-LB |           Id          |       263,170      |    **8.84E-04**   | 2.52E-05 | **8.84E-04** | 2.49E-05 |     4.00E-04     |   0.000  |

|         |      |             |       |          |        |     |          |    |
| :---------------: | :-------------------: | :------------------: | :---------------: | :------: | :----------: | :------: | :----------------: | :------: |
|          |         | **Equivariant Task** |          |          |         |          |           |          |
|       Model       | Training augmentation |      Num Params      | Test augmentation |          |         |        | Equivariance Error |          |
|      |       |      |         Id        |          |     SL(3)    |          |         AVG        |    STD   |
|        MLP        |           Id          |        538,120       |       0.011       | 3.53E-04 |     1.318    | 7.08E-02 |        0.424       |   0.003  |
|        MLP        |         SL(3)         |        538,120       |       0.033       | 2.86E-04 |     0.452    | 1.01E-02 |        0.389       |   0.001  |
|      2 LN-LR      |           Id          |        197,376       |       0.213       | 4.07E-05 |     0.213    | 4.08E-05 |      9.32E-05      | 6.65E-06 |
|      2 LN-LB      |           Id          |        328,448       |    **9.83E-10**   | 1.78E-11 | **4.55E-08** | 8.65E-11 |    **6.56E-05**    | 4.22E-07 |
| 2 LN-LR + 2 LN-LB |           Id          |        590,592       |      7.65E-09     | 3.54E-10 |   5.41E-08   | 4.08E-10 |      7.67E-05      | 1.56E-06 |

|      |      |      |     |      |     |
| :-----------: | :---------------------: | :----: | :---: | :-----------: | :---: |
|            | **Classification Task** |        |       |               |       |
|            |        Num Params       |   Acc  |       | Acc (Rotated) |       |
|            |                         |   AVG  |  STD  |      AVG      |  STD  |
|      MLP      |         206,339         | 95.76% | 0.65% |     36.54%    | 0.99% |
|     LN-LR     |         134,664         | 99.56% | 0.23% |     99.51%    | 0.28% |
|     LN-LB     |         200,200         | 99.14% | 0.21% |     98.78%    | 0.49% |
| LN-LR + LN-LB |         331,272         | **99.62%** | 0.25% |    **99.61%**    | 0.14% |

Reviewer comment (Writing):
As a minor comment, the analogy made to vector neurons is more confusing than enlightening. It would be helpful to include a self-contained description of vector neurons to make this comparison more clear. Vector neurons are just one example of an equivariant architecture whose input representations transform according to rotations. Another important feature of vector neurons is that one representation is maintained for each point in the input point cloud, but this does not appear to have an analogue here (subject to correction if this is a misunderstanding).

Author response:
**2. As a minor comment, I found the analogy made to vector neurons more confusing than enlightening. It would be helpful to include a self-contained description of vector neurons, to make this comparison more clear.**

We added a paragraph to briefly introduce Vector Neurons and compare it with our method in the paper (Sec. 4). For Vector Neurons, the input is a collection of 3D points in the Euclidean space. For our method, the input is a collection of vectors in the Lie algebra space. Each feature in our network is maintained for each input element (spatial point for VN, Lie algebra for our network), unless pooled globally. Vector Neurons is equivariant to SO(3) actions on input points, while our method is equivariant to the adjoint action of arbitrary semi-simple Lie groups on input Lie algebras. Notice that the adjoint matrix of SO(3) on its Lie algebras is the same as the rotation matrix (Adm(R)=R), and so(3) is isomorphic to R^3. Therefore, our method can be viewed as a generalization of Vector Neurons. Our method can be reduced to Vector Neurons when we identify the 3D points as so(3) elements.

Reviewer comment (Writing):
The motivation for the work is somewhat unclear. While there is one concrete example involving camera views, I would like to see additional examples to appreciate the broad applicability of this approach, even if only as citations in the introduction.

Author response:
**1. The motivation for the work is somewhat unclear. While there is one concrete example involving camera views, I would like to see additional examples to appreciate the broad applicability of this approach, even if only as citations in the introduction.**

 Thanks for pointing this out. The motivation is to provide a general equivariant framework for representation learning in problems with Lie group geometric structures. The Lie algebraic representation (with adjoint group action) arises from standard group representation, making the proposed network as general as equivariance with respect to Lie group actions on vector spaces.


We think the proposed network will have broad applications and has the potential to be adopted for applications such as 3D data processing using Inertial Measurement Units \[1,2] and Point Clouds \[3,4] where data can be viewed directly in $\mathfrak{so}(3)$ Lie algebra.


There are applications where the inputs are transformations of semi-simple groups, notably $SL(n)$. In computer vision, $SL(3)$ is the homography group \[5,6], which contains affine(2), and 2D affine transformations as its subgroup. Representation learning on $\mathfrak{sl}(3)$ will result in homography-invariant features applicable in numerous detection and classification tasks. For physical systems, $SL(3)$ is the volume-preserving transformation (a generalization of rigid body motion $SE(3)$ in kinematics and dynamics). We conjecture the proposed work can help predict the motion of deformable shapes, even though we do not have experimental results to back up this conjecture, and this is purely our geometric intuition from analytical mechanics.

 We envision our network to be applied in different applications in two ways: 

  1. For inputs that are directly transformations of semi-simple groups, we can directly apply our framework. This type of problems often arises in robotic state estimation and tracking, where the states being estimated are represented using elements of Lie groups. An example of this is homography tracking \[7], where measured homography matrices are noisy and fusion across multiple measurements is required. 
  2. If the measurements are raw sensor inputs which lives in some vector space, we need to find an adjoint-equivariant lift similar to Equation 13 in \[8]. This equivariant lift can be found heuristically, or it can be another equivariant neural network. For the special case where the input vectors are subjected to $SO(3)$ actions, such a lift will be the identity map. This is because the adjoint matrix of $SO(3)$ is exactly equal to the rotation matrix, i.e. $Adm\_R = R$, and $\mathfrak{so}(3)$ is isomorphic to $R^3$. To be clear, a lift from the input to the symmetry group is always possiple, e.g., via sampling, but the challenge is in constructing an equivariant lift. We plan to explore a more general lifting network design in the near future.

\[1] Hamel, Tarek, and Robert Mahony. "Attitude estimation on SO \[3] based on direct inertial measurements." _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._. IEEE, 2006.

\[2] Mahony, Robert, Tarek Hamel, and J-M. Pflimlin. "Complementary filter design on the special orthogonal group SO (3)." _Proceedings of the 44th IEEE Conference on Decision and Control_. IEEE, 2005.

\[3] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.

\[4] Zhu, Minghan, Maani Ghaffari, and Huei Peng. "Correspondence-free point cloud registration with SO (3)-equivariant implicit shape representations." _Conference on Robot Learning_. PMLR, 2022.

\[5] Manerikar, Ninad, Minh-Duc Hua, and Tarek Hamel. "Homography observer design on special linear group SL (3) with application to optical flow estimation." _2018 European Control Conference (ECC)_. IEEE, 2018.

\[6] Mei, Christopher, et al. "Homography-based tracking for central catadioptric cameras." _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2006.

\[7] Hamel, Tarek, et al. "Homography estimation on the special linear group based on direct point correspondence." _2011 50th IEEE Conference on Decision and Control and European Control Conference_. IEEE, 2011.

\[8] van Goor, Pieter, Tarek Hamel, and Robert Mahony. "Equivariant filter (eqf)." _IEEE Transactions on Automatic Control_ (2022).

Reviewer comment (Theory):
In many useful applications, the mapping between vector spaces is not defined on the canonical basis but through representations of the group. Finding bases for these representations is a challenging task, and I think a discussion on this topic is missing.

Author response:
**2. In many useful applications, the mapping between vector spaces is not defined on the canonical basis but through representations of the group. Finding bases for these representations is a challenging task, and I think a discussion on this topic is missing.**

Thanks for the insight. In this work, we use the standard group representation in its Lie algebra, which gives rise to the adjoint (conjugation) representation. We agree this is one particular representation of the group and if input data comes in the form of other group representation that is unknown, the current framework might not be directly applicable. At the time of developing the Lie algebraic network, we did not consider these cases and assumed the input is lifted to the group standard representation in the Lie algebra. We will add this discussion in Sec. 6 to clarify this for readers. Learning the group irreps might be a promising direction to address this limitation. https://proceedings.mlr.press/v32/cohen14.html

Reviewer comment (Theory):
The fact that the Killing form for non-compact groups is not an inner product makes it ill-suited for measuring distances in the nonlinear activation. I am concerned about the potential numerical stability issues this could cause.

Author response:
**3. The fact that the Killing form for non-compact groups is not an inner product makes it ill-suited for measuring distances in the nonlinear activation. I am concerned about the potential numerical stability issues this could cause.**

Thanks for pointing out this concern. We conducted extensive experiments on the three tasks described in the initial submission and did not observe a single case of training failure (NaN in weights). 
However, in new experiments that we added during the rebuttal (the point cloud registration task where we deal with $\mathfrak{so}(3)$-equivariance, where the network and loss functions are more complicated. More detail in Q4), we did observe training failure (NaN weights) caused by the numerical instability of the Killing-form layer. 
Our understanding of the potential risk of the Killing form not being an inner product for non-compact groups is that we cannot have the normalization term $||d||$ in the denominator, causing the output scale and thus the gradient scale to be unstable. Accordingly, we modified the Killing-form layer in the following form: 
$$
f(x) = x, if B(x,d) <= 0;
$$
$$
f(x) = x + tanh(B(x,d))d, otherwise. 
$$
The added tanh() function constrains the Killing-form output to (-1, 1). This modified layer fixed the numerical stability problem, while the performance is consistent with models using the original Killing-form layer. We never observed training failures again. We appreciate your comment, which helped improve our network design.

Reviewer comment (Experiments):
The only baselines presented in the paper are standard MLPs. Although I understand that this is primarily a theoretical work, I would expect to see at least one strong baseline in one of the tasks.

Author response:
**4. The only baselines presented in the paper are standard MLPs. Although I understand that this is primarily a theoretical work, I would expect to see at least one strong baseline in one of the tasks.**

Unfortunately, there is no existing network equivariant to adjoint operations on semi-simple Lie Algebras. Therefore, it is not straightforward to find a comparable strong baseline. We try to address your concerns in two ways. 

First, we added MLP with augmentation as a stronger baseline, in which the training data undergoes $SL(3)$ adjoint action augmentation. In this way, the MLP is forced to generalize to $SL(3)$ adjoint actions, and we compare this with equivariance-by-design in our method. The results on the invariant and equivariant regression tasks are shown in the tables below. We can see that MLP with augmentations still underperforms our equivariant solutions. 

****

|                   |                       |                    |                   |          |              |          |                      |          |
| :---------------: | :-------------------: | :----------------: | :---------------: | :------: | :----------: | :------: | :------------------: | :------: |
|                   |                       | **Invariant Task** |                   |          |              |          |                      |          |
|       Model       | Training augmentation |   Num Params   | Test augmentation |          |   SL(3)  |          |                      |          |
|                   |                       |                    |       Id      |          |    AVG   |  STD |        AVG     |  STD |
|      MLP     |         Id        |       136,193      |       0.148       |   0.005  |     6.493    |   1.282  |         1.415        |   0.113  |
|      MLP    |       SL(3)       |       136,193      |       0.201       |   0.010  |     1.119    |   0.018  |         0.683        |   0.006  |
|     LN-LR     |         Id       |       66,562       |      1.30E-03     | 3.24E-05 |   1.30E-03   | 3.25E-05 |     3.60E-04     | 5.48E-05 |
|     LN-LB     |         Id        |       132,098      |       0.557       | 1.87E-04 |     0.557    | 1.87E-04 |         **1.43E-05**        | 1.42E-06 |
| LN-LR + LN-LB|         Id        |       263,170      |    **8.84E-04**   | 2.52E-05 | **8.84E-04** | 2.49E-05 |       4.00E-04       |   0.000  |

****

|                       |                       |                      |                   |          |           |          |                        |          |
| :-------------------: | :-------------------: | :------------------: | :---------------: | :------: | :-------: | :------: | :--------------------: | :------: |
|                       |                       | **Equivariant Task** |                   |          |           |          |                        |          |
|         Model         | Training augmentation |    Num Params    | Test augmentation |          |           |          | Equivariance Error |          |
|                       |                       |                      |       Id      |          | SL(3) |          |                        |          |
|                       |                       |                      |     AVG    |STD | AVG| STD |        AVG       | STD|
|        MLP        |         Id        |        538,120       |       0.011       | 3.53E-04 |   1.318   | 7.08E-02 |          0.424         |   0.003  |
|        MLP        |       SL(3)       |        538,120       |       0.033       | 2.86E-04 |   0.452   | 1.01E-02 |          0.389         |   0.001  |
|      2 LN-LR      |         Id        |        197,376       |       0.213       | 4.07E-05 |   0.213   | 4.08E-05 |        9.32E-05        | 6.65E-06 |
|      2 LN-LB      |         Id        |        328,448       |      **9.83E-10**     | 1.78E-11 |  **4.55E-08** | 8.65E-11 |        **6.56E-05**        | 4.22E-07 |
| 2 LN-LR + 2 LN-LB |         Id        |        590,592       |      7.65E-09     | 3.54E-10 |  5.41E-08 | 4.08E-10 |        7.67E-05        | 1.56E-06 |


**4. The only baselines presented in the paper are standard MLPs. Although I understand that this is primarily a theoretical work, I would expect to see at least one strong baseline in one of the tasks. (Cont'd)**

Second, we compare with Vector Neurons in the $\mathfrak{so}(3)$ regime. Our method can be viewed as a generalization of Vector Neurons to semi-simple Lie algebras, and mostly falls back to Vector Neurons for $\mathfrak{so}(3)$, except for the new Lie bracket nonlinear layer. We compare the performance of our network with the Lie bracket layer against the original Vector Neurons, in the point cloud classification \[3] and registration \[4] tasks. As expected, our performance is very similar to the baseline. This demonstrates that the proposed network generalizes Vector Neurons, while maintaining the performance on $SO(3)$ tasks.

****

|                |                            |                          |               |
| -------------- | -------------------------: | -----------------------: | ------------: |
|                | Point Cloud Classification (acc, \% ) |  |Point Cloud Registration      (angular error, deg)          |
|                |                                          |           Gaussian Noise $\sigma=0.01$  | Uniform Noise $scale=0.05$|
| Vector Neurons |                     73.70% |                   2.2263 |        1.6305 |
| Ours           |                     73.50% |                   2.5343 |        1.1324 |

****

\[3] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.

\[4] Zhu, Minghan, Maani Ghaffari, and Huei Peng. "Correspondence-free point cloud registration with SO (3)-equivariant implicit shape representations." _Conference on Robot Learning_. PMLR, 2022.

Reviewer comment (Evaluation):
I am surprised by the invariant error in Table 1 for the LN-LR model. This seems to be an order of magnitude beyond floating-point precision. Do you have any idea where this discrepancy originates?

Author response:
**5. I am surprised by the invariant error in Table 1 for the LN-LR model. This seems to be an order of magnitude beyond floating-point precision. Do you have any idea where this discrepancy originates?**

We’re happy to explain the discrepancy. Our network indeed operates on float32 in pytorch. However, during test time, we compute the average of the error over 10,000 testing points using Python’s default precision, which is float64. Because we have $10^5$ data points, if the cumulative error across data points is small, the computed average error can be lower than float32 precision.

Reviewer comment (Theory):
What is the insight behind LN-BRACKET, other than that it is an easily definable equivariant operation? It feels somewhat arbitrary, and no information is provided about the universality of a model containing this nonlinearity.

Author response:
**6. What is the insight behind LN-BRACKET, other than that it is an easily definable equivariant operation? It feels somewhat arbitrary, and no information is provided about the universality of a model containing this nonlinearity.**

This is a great question that, unfortunately, we omitted further motivation in the initial draft. We have two perspectives on this definition (besides its mathematical feasibility and natural existence in the current framework):
The residual layer with bracket has a simple explanation: it learns a delta (differential) term for the input via Lie derivative, which is the Lie bracket in the Lie algebra. In this sense, it increases the input order analogous to Taylor expansion via a first-order derivative. Higher order and nested bracket layers are also possible that we did not investigate in this work. We plan to study the universal approximation theory for arbitrary width and depth cases of the proposed network in the near future.
The second motivation comes from potential applications of the proposed network on physical and mechanical systems. The Euler-Poincaré equations of motion in the Lie algebra are derived as [9] 
		$$ I \dot{\xi} = [I\xi,\xi]^\vee,$$
where $I$ is the generalized inertial matrix and $\xi \in \mathfrak{g}$ is a Lie algebra element. These are equations of motion in the Lie algebra and are equivariant. The particular $SO(3)$ case, is the classical Euler equation in the body frame for a rotating rigid body. In the future, we would like to study the application of the developed framework in learning dynamics and control. $SL(3)$ can model volume-preserving deformable bodies, a generalization of rigid body $SO(3)$, which is an exciting problem to investigate. 

[9] Bloch, Anthony, P. S. Krishnaprasad, Jerrold E. Marsden, and Tudor S. Ratiu. "The Euler-Poincaré equations and double bracket dissipation." Communications in mathematical physics 175, no. 1 (1996): 1-42.

Reviewer comment (Experiments):
Related to the weaknesses, can you identify a practically useful application for this approach? Such an application should have interesting baselines for comparison.

Author response:
**7. Related to the weaknesses, can you identify a practically useful application for this approach? Such an application should have interesting baselines for comparison.**

We kindly refer the reviewer to our reponses in question 1 and 4.