Paper: A Lie Group Approach to Riemannian Batch Normalization (ICLR-2024)

Reviewer comment (Novelty):
Novelty of LieBN in Sec.4 is limited as it is rather straightforward from the prior works [Kobler+22a] and [Chakraborty+20].

Author response:
**1. Difference with the prior work [Kobler+22a] and [Chakraborty+20].**

The theory and implementation of our LieBN are different from [Kobler+22a] and more general than [Chakraborty+20]. Please refer to CQ#1 in the common response.

Reviewer comment (Reproducibility):
It is demanded to clarify computational details such as by showing back-props through comparison to the other RN approaches based on complicated manifold-based computation, which would significantly improve reproducibility.

Author response:
**2. Computational details on back-props**

As summarized in the following table, two types of matrix functions are involved in our LieBN on SPD manifolds: one is based on Eigendecomposition, and the other is the Cholesky decomposition. The Eigen-based matrix functions include matrix logarithm, exponential, and power. Their BP can be calculated by the  Daleckii-Krein formula [2 Thm. V.3.3]. The BP of the Cholesky decomposition has been well established in `torch.linalg.cholesky`. Therefore, we use `torch.linalg.cholesky` for the Cholesky decomposition. We have added a detailed discussion on matrix BP in the App. F in our revised manuscript.

Table A: Matrix functions involved in LieBN on SPD manifolds
|       Matrix Function      |          Type          |       Involvement       |
|:--------------------------:|:----------------------:|:-----------------------:|
|      Matrix Logarithm      |        Eigen-based       | LieBN-AIM and LieBN-LEM |
|     Matrix Exponential     |        Eigen-based       | LieBN-AIM and LieBN-LEM |
|     Matrix Square Root     |        Eigen-based       |        LieBN-AIM        |
| Matrix Inverse Square Root |        Eigen-based       |        LieBN-AIM        |
|        Matrix Power        |        Eigen-based       |        LieBN-AIM and power deformation        |
|   Cholesky Decomposition   | Cholesky Decomposition |        LieBN-LCM        |

Reviewer comment (Evaluation):
As to empirical performance results reported in the experiments, superiority of the method is less clear since the performance improvement is not significant due to large stds of performance scores.

Author response:
**3. Large std on EEG**

Recalling Tab. 4-5, it is only on the EEG datasets that the performance shows relatively large std, which should be attributed to the characteristics of the EEG dataset. EEG signals exhibit low signal-to-noise ratio (SNR), domain shifts, and low specificity. These challenges are more obvious in the inter-session and inter-subject scenarios. 

Despite the relatively large std, our LieBN outperforms the SPDDSMBN on the inter-subject classification by 3.87% (50.10 v.s. 53.97). Besides, our LieBN can show less std than the SPDDSMBN baseline on EEG datasets. For instance, for the inter-subject task (Tab. 5 (b)), the std of LCM-based LieBN is clearly less than SPDDSMBN ( 51.53±4.96 v.s. 50.10±8.08).

Reviewer comment (Evaluation):
Based on the experimental results, one cannot identify the best SPD-BN method that outperforms the others consistently. Although the authors insist such an inconsistency shows generality of the approach, it is less understandable and unfavorable from a practical viewpoint. In this case, provide some discussion and/or analysis about connection between types of metrics and tasks (or network architectures) for rendering insights into the SPD metrics.

Author response:
**4.  Discussion and analysis about the choice of metrics.**

The foremost challenge when building Riemannian learning algorithms is choosing an appropriate Riemannian metric. If the framework is general, there would be many metrics to select from, and the model would be more likely to perform better. This is why we view the generality of our framework as an advantage.

Generally speaking, AIM is the best candidate metric on SPD manifolds. The most significant reason is the property of affine invariance, which is a natural characteristic of describing covariance matrices. In our experiments, the LieBN-AIM generally achieves the best performance. In some scenarios, there are better choices than AIM. As shown in Tab. 4 (b), the best result on the HDM05 dataset is achieved by LCM-based LieBN, which improves the vanilla SPDNet by 11.71%. Therefore, when choosing Riemannian metrics on SPD manifolds, a safe choice would start with AIM and extend to other metrics.

Reviewer comment (Evaluation):
Analysis about the SPD-BN methods in Table 3 from computational viewpoint in comparison to the other RNs.

Author response:
**5. Computational analysis and comparison.**

Our paper compares our LieBN against two existing SPDBNs, including SPDNetBN and SPDDSMBN. As SPDDSMBN is the improved version of SPDNetBN, let us focus on comparing our LieBN and SPDDSMBN. For simplicity, we focus on LieBN induced by the standard ($\theta=1$) AIM, LEM, and LCM.

**Computational comparison of LieBNs under different metrics**
Recalling Tab. 3, the LCM-based LieBN only contains two Cholesky decompositions, while LEM-based LieBN contains two eigendecompositions. Note that Cholesky decomposition is more efficient than eigendecomposition. On the other hand, the AIM-based LieBN involves several more matrix decompositions. Therefore, our LieBN's efficiency order should be LieBN-LCM > LieBN-LEM > LieBN-AIM.

**Computational comparison of LieBNs against SPDDSMBN**

As SPDDSMBN is based on AIM, it should be less efficient than our LieBN-LCM and LieBN-LCM. Besides, the major difference between our LieBN-AIM and SPDDSMBN lies in biasing and centering. Our LieBN-AIM uses Cholesky decomposition for biasing and centering, while SPDDSMBN adopts the matrix square root. As the Cholesky decomposition is more efficient than the matrix square root, we could expect our LieBN-AIM to be more efficient than SPDDSMBN. 

**Empirical training cost**
The general order of efficiency should be LieBN-LCM > LieBN-LEM > LieBN-AIM>SPDDSMBN. This can be demonstrated by the average training time per epoch presented in Tab. 4 - Tab.5. Tab. 5 (b) shows that our LieBN is generally much more efficient than SPDDSMBN. In detail, LieBN-LCM only takes half of the training time of SPDDSMBN (7.74 v.s. 3.59). LieBN-LEM is also much more efficient than SPDDSMBN (7.74 v.s. 4.71). Besides, LieBN-AIM also achieves better efficiency than SPDDSMBN (7.74 v.s. 6.94).

Reviewer comment (Evaluation):
Empirical comparison regarding computation cost.

Author response:
**5. Computational analysis and comparison.**

Our paper compares our LieBN against two existing SPDBNs, including SPDNetBN and SPDDSMBN. As SPDDSMBN is the improved version of SPDNetBN, let us focus on comparing our LieBN and SPDDSMBN. For simplicity, we focus on LieBN induced by the standard ($\theta=1$) AIM, LEM, and LCM.

**Computational comparison of LieBNs under different metrics**
Recalling Tab. 3, the LCM-based LieBN only contains two Cholesky decompositions, while LEM-based LieBN contains two eigendecompositions. Note that Cholesky decomposition is more efficient than eigendecomposition. On the other hand, the AIM-based LieBN involves several more matrix decompositions. Therefore, our LieBN's efficiency order should be LieBN-LCM > LieBN-LEM > LieBN-AIM.

**Computational comparison of LieBNs against SPDDSMBN**

As SPDDSMBN is based on AIM, it should be less efficient than our LieBN-LCM and LieBN-LCM. Besides, the major difference between our LieBN-AIM and SPDDSMBN lies in biasing and centering. Our LieBN-AIM uses Cholesky decomposition for biasing and centering, while SPDDSMBN adopts the matrix square root. As the Cholesky decomposition is more efficient than the matrix square root, we could expect our LieBN-AIM to be more efficient than SPDDSMBN. 

**Empirical training cost**
The general order of efficiency should be LieBN-LCM > LieBN-LEM > LieBN-AIM>SPDDSMBN. This can be demonstrated by the average training time per epoch presented in Tab. 4 - Tab.5. Tab. 5 (b) shows that our LieBN is generally much more efficient than SPDDSMBN. In detail, LieBN-LCM only takes half of the training time of SPDDSMBN (7.74 v.s. 3.59). LieBN-LEM is also much more efficient than SPDDSMBN (7.74 v.s. 4.71). Besides, LieBN-AIM also achieves better efficiency than SPDDSMBN (7.74 v.s. 6.94).

Reviewer comment (Novelty):
I am unsure what is the contribution of the paper. As the authors state, the normalization scheme they propose has been used in previous work. There are some claims like "In contrast, our work provides a more extensive examination, encompassing both population and sample properties of our LieBN in a general manner. Besides, all the discussion about our LieBN can be readily transferred to right-invariant metrics." but I was not able to find out what specifically these differences are. The approach seems to be almost exactly the same when I look up in the cited papers where it is applied to Lie groups as well.

Author response:
We thank reviewer $\textcolor{green}{LaGM}$ ($\textcolor{green}{R3}$) for the valuable comment. In the following, we respond to the concerns in detail.
***

**1. Difference from the previous RBN methods.** 

The RBNs in these three works are **_either designed for a specific manifold or metric_**, or fail to control mean and variance, while our LieBN fulfills normalization on general Lie groups. We have summarized their limitations in Tab. 2 and discussed the difference in Sec. 3.2. Here we clarify the difference between our LieBN and their RBN in details.

**- Difference with the RBN proposed in [1]**

**_The RBN in [1] is confined within the standard Affine-Invariant Metric (AIM)_**. In contrast, we implement our LieBN under three invariant metrics: Log-Euclidean Metric (LEM), Log-Cholesky Metric (LCM), and AIM. Moreover, our LieBN further covers the deformed families of LEM, LCM, and AIM. 

**_For the specific AIM, our LieBN is also better than the RBN [1] in terms of computational efficiency._** The RBN in [1] is based on parallel transportation, Riemannian exp and log. The specific equation is calculated by the matrix power function (Eq. (10) in [1]). In contrast, our LieBN is based on Lie group left translation, which is fulfilled by Cholesky decomposition (Tab. 3). Note that Cholesky decomposition is more efficient than the matrix power function. As shown in Tab. 5 (b), DSMLieBN-AIM-(1) achieves similar performance to SPDDSMBN with less training time (6.94 v.s. 7.74).

**_Besides, the RBN in [1] is problematic when extending to other manifolds or metrics._** As shown in Eq. (9), the core formulation of the RBN in [1] can be described as
$$
\Gamma _{P \rightarrow Q}(S)=\operatorname{Exp} _Q\left[\mathrm{PT} _{P \rightarrow Q}\left(\operatorname{Log} _P(S)\right)\right].
$$

Since $\Gamma_{P \rightarrow Q}$ becomes affine action under AIM, this formulation can coincidentally control mean under AIM on SPD manifolds. However, the resultant mean and variance are generally agnostic under general manifolds. Therefore, the RBN based on $\Gamma _{P \rightarrow Q}$ could not be extended into other metrics or manifolds. On the contrary, our Props. 4.1-4.2 guarantee the rationality of our LieBN on other Lie groups, such as SO(n). For experiments on SO(n), please refer to the 3rd response to $\textcolor{red}{R1}$.

**- Difference with the RBN proposed by [3]**

**_Similar to [1], the RBN in [3] generally fails to normalize mean or variance either_,** as this RBN is a variant of $\Gamma_{P \rightarrow Q}$. In contrast, our LieBN can always control both mean and variance on Lie groups.

**- Difference with the RBN proposed by [4]**

**_The RBN on matrix Lie groups in [4] is also confined within a specific metric and group structures (Sec. 3.2 in [4])_.** On the contrary, our LieBN is designed for general Lie groups. In this sense, the RBN on matrix Lie groups in [4] is a special case of our LieBN. Furthermore, as stated at the end of Sec. 4, our LieBN framework can be extended into right-invariant metrics.

Besides, as stated in Rmk. C.5 in the appendix, the proof presented in [4] is a bit problematic, as the author fails to consider Riemannian volume when dealing with the probability density function. In contrast, our theoretical results in Props. 4.1-4.2 are more solid and general.

**- Summary**

In summary, the RBNs in [1-3] are either confined within a specific manifold or metric, or fail to control mean and variance. In contrast, we propose a  framework of batch normalization over general Lie groups and showcase its generality over three families of Lie groups on SPD manifolds.

> [1] Kobler R, Hirayama J, Zhao Q, et al. SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG.
> 
> [2] Brooks D, Schwander O, Barbaresco F, et al. Riemannian batch normalization for SPD neural networks. 
> 
> [3] Lou A, Katsman I, Jiang Q, et al. Differentiating through the fréchet mean.
> 
> [4] Chakraborty R. Manifoldnorm: Extending normalizations on Riemannian manifolds.

Reviewer comment (Novelty):
Using the Riemannian or Lie group exp and log maps for batch normalization was a good idea the first time it was presented, but I don't see the value added with the current paper.

Author response:
We thank reviewer $\textcolor{green}{LaGM}$ ($\textcolor{green}{R3}$) for the valuable comment. In the following, we respond to the concerns in detail.
***

**1. Difference from the previous RBN methods.** 

The RBNs in these three works are **_either designed for a specific manifold or metric_**, or fail to control mean and variance, while our LieBN fulfills normalization on general Lie groups. We have summarized their limitations in Tab. 2 and discussed the difference in Sec. 3.2. Here we clarify the difference between our LieBN and their RBN in details.

**- Difference with the RBN proposed in [1]**

**_The RBN in [1] is confined within the standard Affine-Invariant Metric (AIM)_**. In contrast, we implement our LieBN under three invariant metrics: Log-Euclidean Metric (LEM), Log-Cholesky Metric (LCM), and AIM. Moreover, our LieBN further covers the deformed families of LEM, LCM, and AIM. 

**_For the specific AIM, our LieBN is also better than the RBN [1] in terms of computational efficiency._** The RBN in [1] is based on parallel transportation, Riemannian exp and log. The specific equation is calculated by the matrix power function (Eq. (10) in [1]). In contrast, our LieBN is based on Lie group left translation, which is fulfilled by Cholesky decomposition (Tab. 3). Note that Cholesky decomposition is more efficient than the matrix power function. As shown in Tab. 5 (b), DSMLieBN-AIM-(1) achieves similar performance to SPDDSMBN with less training time (6.94 v.s. 7.74).

**_Besides, the RBN in [1] is problematic when extending to other manifolds or metrics._** As shown in Eq. (9), the core formulation of the RBN in [1] can be described as
$$
\Gamma _{P \rightarrow Q}(S)=\operatorname{Exp} _Q\left[\mathrm{PT} _{P \rightarrow Q}\left(\operatorname{Log} _P(S)\right)\right].
$$

Since $\Gamma_{P \rightarrow Q}$ becomes affine action under AIM, this formulation can coincidentally control mean under AIM on SPD manifolds. However, the resultant mean and variance are generally agnostic under general manifolds. Therefore, the RBN based on $\Gamma _{P \rightarrow Q}$ could not be extended into other metrics or manifolds. On the contrary, our Props. 4.1-4.2 guarantee the rationality of our LieBN on other Lie groups, such as SO(n). For experiments on SO(n), please refer to the 3rd response to $\textcolor{red}{R1}$.

**- Difference with the RBN proposed by [3]**

**_Similar to [1], the RBN in [3] generally fails to normalize mean or variance either_,** as this RBN is a variant of $\Gamma_{P \rightarrow Q}$. In contrast, our LieBN can always control both mean and variance on Lie groups.

**- Difference with the RBN proposed by [4]**

**_The RBN on matrix Lie groups in [4] is also confined within a specific metric and group structures (Sec. 3.2 in [4])_.** On the contrary, our LieBN is designed for general Lie groups. In this sense, the RBN on matrix Lie groups in [4] is a special case of our LieBN. Furthermore, as stated at the end of Sec. 4, our LieBN framework can be extended into right-invariant metrics.

Besides, as stated in Rmk. C.5 in the appendix, the proof presented in [4] is a bit problematic, as the author fails to consider Riemannian volume when dealing with the probability density function. In contrast, our theoretical results in Props. 4.1-4.2 are more solid and general.

**- Summary**

In summary, the RBNs in [1-3] are either confined within a specific manifold or metric, or fail to control mean and variance. In contrast, we propose a  framework of batch normalization over general Lie groups and showcase its generality over three families of Lie groups on SPD manifolds.

> [1] Kobler R, Hirayama J, Zhao Q, et al. SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG.
> 
> [2] Brooks D, Schwander O, Barbaresco F, et al. Riemannian batch normalization for SPD neural networks. 
> 
> [3] Lou A, Katsman I, Jiang Q, et al. Differentiating through the fréchet mean.
> 
> [4] Chakraborty R. Manifoldnorm: Extending normalizations on Riemannian manifolds.

Reviewer comment (Novelty):
I believe the authors need to argue convincingly what is the contribution of the paper, and why the paper presents a significant contribution relative to the previously methods.

Author response:
Thanks for your feedback! Our last response clarified our contributions and the main difference with previous RBN methods. The paper has also been modified accordingly. From our perspective, your main question should have been (largely) solved. 

Could you elaborate more on your remaining concerns? We still have time to answer. 😄

Reviewer comment (Evaluation):
Experimental results are not convincing.

Author response:
This paper mainly focuses on building LieBN on general Lie groups. **_Since we only change the model by adding a single BN layer, this slight modification is not expected to outperform every baseline._** Nevertheless, our experiments on the SPD manifolds indicate that our LieBN can improve the performance of SPD neural networks. **On the Radar, HDM05, FPHA, and EEG datasets, our LieBN improves the SPD baselines by 2.22%, 11.71%, 4.8% and 3.87%, respectively**. This is convincing to claim that SPD neural networks can benefit from our LieBN. 

**_Besides, SPDDSMBN is one of the SOTA methods in EEG classification._** Tab. 5 indicates the superiority of our work against SPDDSMBN. Moreover, Tab. 5 shows that LCM- or LEM-based LieBN can achieve comparable performance against SPDDSMBN with less training time. For instance, for inter-subject classification, LCM-based LieBN reaches similar results to SPDDSMBN while costing only half of the training time. Besides, LCM-based LieBN tends to show smaller std than SPDDSMBN (4.96 v.s. 8.08 for the inter-subject task)

Reviewer comment (Novelty):
The proposed technique is a simple tweak of those from Kobler et al. (2022b), Lou et al. (2020), and Chakraborty (2020), with no new concepts or ideas developed with respect to these works.

Author response:
**1. The proposed technique is not a simple tweak of those from Kobler et al. (2022b), Lou et al. (2020), and Chakraborty (2020).**

As stated in CQ#1 of the common response, our work is entirely theoretically different from Kobler et al. (2022b) and Lou et al. (2020), and more general than Chakraborty (2020). Please refer to the common response for details.

Reviewer comment (Experiments):
The experimental results are not convincing since the proposed method is only compared with some SPD neural networks, and on human action recognition, the proposed method is outperformed by the method of Laraba et al. (2017) on the HDM05 dataset by a large margin (72.27% vs. 83.33%).

Author response:
This paper mainly focuses on building LieBN on general Lie groups. **_Since we only change the model by adding a single BN layer, this slight modification is not expected to outperform every baseline._** Nevertheless, our experiments on the SPD manifolds indicate that our LieBN can improve the performance of SPD neural networks. **On the Radar, HDM05, FPHA, and EEG datasets, our LieBN improves the SPD baselines by 2.22%, 11.71%, 4.8% and 3.87%, respectively**. This is convincing to claim that SPD neural networks can benefit from our LieBN. 

**_Besides, SPDDSMBN is one of the SOTA methods in EEG classification._** Tab. 5 indicates the superiority of our work against SPDDSMBN. Moreover, Tab. 5 shows that LCM- or LEM-based LieBN can achieve comparable performance against SPDDSMBN with less training time. For instance, for inter-subject classification, LCM-based LieBN reaches similar results to SPDDSMBN while costing only half of the training time. Besides, LCM-based LieBN tends to show smaller std than SPDDSMBN (4.96 v.s. 8.08 for the inter-subject task)

Reviewer comment (Evaluation):
The results suggest that the proposed technique is probably not effective compared to other learning techniques designed in Euclidean space.

Author response:
This paper mainly focuses on building LieBN on general Lie groups. **_Since we only change the model by adding a single BN layer, this slight modification is not expected to outperform every baseline._** Nevertheless, our experiments on the SPD manifolds indicate that our LieBN can improve the performance of SPD neural networks. **On the Radar, HDM05, FPHA, and EEG datasets, our LieBN improves the SPD baselines by 2.22%, 11.71%, 4.8% and 3.87%, respectively**. This is convincing to claim that SPD neural networks can benefit from our LieBN. 

**_Besides, SPDDSMBN is one of the SOTA methods in EEG classification._** Tab. 5 indicates the superiority of our work against SPDDSMBN. Moreover, Tab. 5 shows that LCM- or LEM-based LieBN can achieve comparable performance against SPDDSMBN with less training time. For instance, for inter-subject classification, LCM-based LieBN reaches similar results to SPDDSMBN while costing only half of the training time. Besides, LCM-based LieBN tends to show smaller std than SPDDSMBN (4.96 v.s. 8.08 for the inter-subject task)

Reviewer comment (Experiments):
How does the proposed method perform on another Lie group, e.g., when being used in LieNet (Huang et al., 2017)?

Author response:
**3 LieBN on LieNet (Huang et al., 2017)**

The Lie group in LieNet [1] is SO(3). Our LieBN can also be implemented in this Lie group. However, we only implement centering and biasing operations for now due to limited time. 

**Theoretical ingredient**

Since we only implement the centering and biasing of LieBN, the core operation of LieBN becomes
$$
L _{B} \circ L _{M _{\odot} ^{-1}}(P _i) = B M ^{-1} P _i
$$
where $\{P _i\}$ is a batch of SO(3), $M$ is the SO(3) batch mean，and $B$ is an SO(3) biasing parameters. The batch mean can be obtained by [2, Alg. 1], while the running mean can be updated by the geodesic connecting the running mean and batch mean.  The calculation of batch mean [2, Alg. 1] requires Lie group exp & log on SO(3). We present all the necessary ingredients in Tab. 1. For more detail, please refer to [2-3].


Table 1: Riemannian operators required in LieBN on SO(3)
|       Operator      |          Expression          |     
|:--------------------------:|:----------------------:|
|Lie group exponentiation |$\exp(V)$| 
|Lie group logarithm |$\log(V)$| )\log(V)$|  
|Geodesic connecting $S$ and $R$|$\gamma (t;S,R)= S\exp(t\log(S ^\top R))$|
|Weighted Fréchet mean| [2, Alg. 1]|

**Implementation details and dataset**

As LieNet was originally implemented by Matlab, we use the open-sourced Pytorch code [4] to reimplement LieNet by torch. We found that the matrix logarithm in the Pytorch code [4] fails to deal with singular cases well. Hence, we use Pytorch3D [5] to calculate the matrix logarithm, i.e., `pytorch3d.transforms.matrix_to_axis_angle`. Besides, we implement matrix exponentiation by Rodrigues' formula [6, Eq. (2)].

Following LieNet, we adopt the G3D-Gaming dataset [7], and follow all the learning settings as in the original paper of LieNet. We apply the LieBN layer before each pooling layer. As the RotMap layer in the original LieNet is actually based on left translation as well, we use the LieBN layer to substitute the RotMap layer. Following LieNet, we adopt the architecture of three pooling layers, which can be illustrated as
$$
f _{\mathrm{LieBN}} \rightarrow f _{\mathrm{RotPooling}} \rightarrow f _{\mathrm{LieBN}} \rightarrow f _{\mathrm{RotPooling}} \rightarrow f _{\mathrm{LieBN}} \rightarrow f _{\mathrm{RotPooling}} \rightarrow f _{\log} \rightarrow f _{\mathrm{FC}}.
$$
where $f _{\log}$ and $f _{\mathrm{FC}}$ denote the matrix logarithm and FC layer. Similar to LieNet, we train the network by the standard cross-entropy loss.

**Results**

The results are presented in the following table. Due to different software, our reimplemented LieNet is slightly worse than the performance reported in [1]. However, we still can observe a clear improvement of LieNetLieBN over LieNet.

Table 2: Results on LieNet with or without LieBN
|       Methods      |          Result          |     
|:--------------------------:|:----------------------:|
|LieNet |86.06| 
|LieNetLieBN |88.18|

Reviewer comment (Presentation):
Nearly all tables are barely readables.

Author response:
**1. Table readability.**

Thanks for the suggestive comments. Due to the page limits, we presented the tables concisely. After our work gets accepted, we will move some experimental settings into the appendix and allocate more space for the tables (possibly allowing for two lines or a large figure).

Reviewer comment (Novelty):
Novelty of the work should emphasized more. LieBN provides more guaranties and a more sound approach. But in the writing, it is not completely clear of what is a full novelty over previous methods and what is a generalization.

Author response:
**2. The novelty of the work should emphasized more.**

Thanks for your constructive suggestion! We emphasize our novelty in the introduction in the revised paper (Page 2), highlighting our contributions that our work can theoretically normalize mean and variance and apply to diverse Lie groups.

Reviewer comment (Experiments):
A broad zoo of choices (AIM, LEM, LCM and alpha beta variants), but no clue on choosing. It's an usual question with this type of methods, and it always a little bit disapointing to simply benchmark over all the possible choices.

Author response:
**3. How to choose hyper-parameters in LieBN.**

Our SPD LieBN has at most three types of hyper-parameters: Riemannian metric, deformation factor $\theta$, and $\mathrm{O}(n)$-invariance parameters $(\alpha,\beta)$. The general order of importance should be Riemannian metric $>$ $\theta$ $>$ $(\alpha,\beta)$.

The most significant parameter is the choice of the Riemannian metric, as all the geometric properties are sourced from the metric. A safe choice would start with AIM, and then decide whether to explore other metrics further. The most important reason is the property of affine invariance of AIM, which is a natural characteristic of covariance matrices. In our experiments, the LieBN-AIM generally achieves the best performance. However, AIM is not always the best metric. As shown in Tab. 4 (b), the best result on the HDM05 dataset is achieved by LCM-based LieBN, which improves the vanilla SPDNet by 11.71\%. Therefore, when choosing Riemannian metrics on SPD manifolds, an appropriate choice would start with AIM and extend to other metrics. Besides, if efficiency is an important factor, one should first consider LCM as it is the most efficient one.

The second one is the deformation factor $\theta$. As we discussed in Sec. 5.1, $\theta$ interpolates between different types of metrics ($\theta=1$ and $\theta \rightarrow0$). Inspired by this, we select $\theta$ around its deformation boundaries (1 and 0). In this paper, we roughly select $\theta$ from $\{ \pm 0.5, \pm 1,\pm 1.5 \}$

The less important parameters are $(\alpha,\beta)$. Recalling Tab. 1, $(\alpha, \beta)$ only affects the Riemannian metric tensor and geodesic distance. For our specific SPD LieBN, they only affect the calculation of variance, which should have fewer effects than the above two parameters. Therefore, we simply set $(\alpha,\beta)=(1,0)$ during experiments.

We have added the above discussion to the App. G.

Reviewer comment (Novelty):
What is the interest of the (alpha, beta) generalization? In particular in context of neural networks? And what are the value used in the experiments?

Author response:
**4. $\mathrm{O}(n)$-invariance hyper-parameters $(\alpha, \beta)$**

As we stated in the last question above, $(\alpha, \beta)$ is less important than Riemannian metrics and deformation factor $\theta$. Nevertheless, we also conduct experiments on the effect of $(\alpha, \beta)$ here. As $\alpha$ is less important as a scaling factor [1], we set $\alpha=1$ and change the value of $\beta$. 

Recalling Eq. (3), $\beta$ controls the relative importance of the trace part against the inner product. Therefore, we set the candidate values of $\beta$ as $\{1,\frac{1}{n},\frac{1}{n^2}, 0, -\frac{1}{n} + \epsilon,-\frac{1}{n^2}\}$, where $n$ is the input dimension of LieBN, and $\epsilon$ is a small positive scalar to ensure $\mathrm{O}(n)$-invariance, *i.e.,* $\min (\alpha, \alpha+n \beta)>0$. $\frac{1}{n^2}$ and $\frac{1}{n}$ means averaging the trace in Eq. (3), while the sign of $\beta$ denotes suppressing (-), enhancing (+), or neutralizing (0) the trace. 

We focus on AIM-based LieBN on the HDM05 dataset. We set $\theta=1.5$, as it is the best deformation factor under this scenario. Other network settings remain the same as the main paper. The 10-fold average results are presented in the following table. Note that the dimension $n$ of the input feature of the LieBN layer is 30 in this setting. As expected, $\beta$ has minor effects on our LieBN. We also add this discussion in App. G.1 in our revised paper.

Table 1: The effect of different $\beta$ for AIM-based LieBN on the HDM05 dataset.
| $\beta$ | -0.0011 | -0.03 | 0.0011 | 0.0333 | 1 | 0 |
|:-------:|:-------:|:----:|:------:|:------:|:-:|:-:|
| Mean±STD | 68.18±0.86 | 68.12±0.74 | 68.20±0.85 | 68.18±0.85 | 68.16±0.80 | 68.16±0.68 |

Reviewer comment (Experiments):
In the article about RBN, Brooks et al discuss about the amount of data need to achieve good performance. Any insight about this for your layer?

Author response:
**5. Effect of the reduced amount of data.**

Following [2], we use 10% of training data to train SPDNet with and without LieBN on the Radar dataset. Following the main paper, we set $\theta=1, 1, -0.5$ and $(\alpha,\beta)=(1,0)$ for $(\theta,\alpha,\beta)$-AIM, $(\theta,\alpha,\beta)$-LEM, and $(\theta)$-LCM. The 10-fold average results are presented in the table below. We could observe a clear improvement of our LieBN to SPDNet under the limited data availability scenarios.

Table 2: Robustness to lack of data of SPDNet with and without LieBN on the Radar dataset.
| SPDNet | SPDNetLieBN-AIM | SPDNetLieBN-LCM | SPDNetLieBN-LEM |
|:------:|:---------------:|:--------------:|:--------------:|
| 84.17 ± 1.66 | **87.63 ± 1.72** | 86.65 ± 1.48 | 87.60 ± 1.31 |

Reviewer comment (Theory):
Is there a link between Frechet variance and the variance of the Gaussian used for normalization?

Author response:
**6. Links between the Fréchet variance and the variance of the Gaussian.**

There are some theoretical links between the Fréchet variance and the variance in the Gaussian distribution in Eq. (12).

Recalling the Gaussian distribution (Eq. (12)) on manifold $\mathcal{M}$
$$
p\left(X \mid M, \sigma^2\right)=k(\sigma) \exp \left(-\frac{\mathrm{d}(X, M)^2}{2 \sigma^2}\right).
$$
As shown in [3, Props. 4.7], $\mathrm{Var}(X)=\mathbb{E} _{\mathbf{X}}\left[\mathrm{d}^2(X, M)\right]=\sigma^2$. Therefore, $\sigma^2$ is the population variance. The empirical counterpart of population variance is the Fréchet variance [4, P12]. As shown by our Props. 4.2, our LieBN can control the sample variance.

The Maximum Likelihood Estimation (MLE) of $\sigma^2$ does not have a general solution, as $k(\sigma)$ depends on the specific metrics. However, the relation is very clear for the invariant metrics on the Lie groups of SPD manifolds. As shown in [5, Props. 7], under AIM, the Fréchet variance $v ^2$ satisfying $\sigma ^2=\phi(v ^2)$ with $\phi$ strictly increasing. For AIM, our LieBN implicitly controls the population variance through the explicit control of sample variance. For the LEM and LCM, our LieBN can directly transfer the latent Gaussian distribution:
$$
\mathcal{N}(M,\sigma^2) \xrightarrow{\text{Centering to }E} \mathcal{N}(E,\sigma^2) \xrightarrow{\text{
Scaling the variance}} \mathcal{N}(E, s^2) \xrightarrow{\text{Biasing to }B} \mathcal{N}(B, s^2).
$$
This result is obtained by Cor. C.4 in App. C and has been discussed in detail in App. C (Page 17).

Reviewer comment (Theory):
I guess it should be "neutral element" instead of identity in Eq 13?

Author response:
**7. It should be "neutral element" instead of identity in Eq. (13).**

Thanks for pointing this out! All the "identity" or "identity element" in the paper means neutral element. We agree that the current usage of "identity element" and "identity matrix" might lead to potential confusion. In the original submission, we clarified at the beginning of Sec. 4 that an identity element may not necessarily be an identity matrix. Now we have changed the "identity" element to the "neutral" element throughout the manuscript for better clarity.