Paper: Intrinsic Riemannian Classifiers on the Deformed SPD Manifolds: A Unified Framework (ICLR-2024)

Reviewer comment (Theory):
The authors claim their approach is general in terms of classifiers and Riemannian manifolds; however, they only derive results for multinomial logistic regression on the SPD manifold.

Author response:
**1. Results of Riemannian multinomial logistic regression (RMLR) on other manifolds.**

Recalling our RMLR in Eq. (14), three operators are involved: Riemannian metric, logarithm, and parallel transportation (or left translation). For specific manifolds, one only needs to put these three operators into Eq. (14). We further implement our RMLR on $\mathrm{SO}(n)$. Please refer to CQ#2 in the common response.

Reviewer comment (Novelty):
The contributions are not very clear. For example, the derivation of Theorem 3.4 has already been done in equation 17 of "Riemannian Multiclass Logistics Regression for SPD Neural Networks" from Chen et al. Furthermore, it can be directly derived from equation (3) by parametrizing \(b_k\) as \(\langle p_k, x \rangle\) and then interpreting the subtraction as a Riemannian logarithm.

Author response:
**3. It can be directly derived from Eq (3) by parametrizing $b _k$ as $\langle p_k, x \rangle$ and then interpreting the subtraction as a Riemannian logarithm.**

Thanks for your insightful comment. We can indeed directly interpret subtraction as the Riemannian logarithm and obtain the RMLR in Eq. (14). However, this direct re-interpretation is only an intuitive re-interpretation and has little theoretical support, as it involves no margin distance or margin hyperplane. However, the margin distance and hyperplane play an important role in designing RMLR, such as the ones on hyperbolic[1] and spherical manifolds [2]. 

Our insight is that RMLR in Eq. (14) is derived from the Riemannian margin distance (Def. 3.3) and hyperplane (Eq. (7)). In this way, it is precise to call Eq. (14) as RMLR. 

We assume your concerns might come from our original structure of Sec. 3.2. To obtain RMLR, the margin distance in Def. 3.3 should be first solved. In the original submission, we put the results on margin distance in the appendix and directly presented our RMLR in the main paper. We have added a theorem about the solution of margin distance (Thm. 3.4) between Def. 3.3 and RMLR (Thm. 3.5) in the revised paper.

Reviewer comment (Presentation):
The distance \(d(S, \tilde{H}_{A, P})\) is defined twice: in equation (8) and equation (11). One should be presented as a proposition and the other as a definition.

Author response:
**2. The distance is defined twice: in Eq (8) and Eq (11).**

The distance in Eq. (11) is different from Eq. (8). The distance in Eq. (11) is the one used in this paper. For better clarity, we have change the $d(\cdot)$ in Eq. (8) as $\tilde{d}(\cdot)$.

Reviewer comment (Theory):
An example of how to apply Theorem 4.2 and Lemma 4.3 to a Riemannian metric should be added to better understand their implications.

Author response:
**6. How to apply Thm. 4.2 and Lem. 4.3.**

**Thm. 4.2 helps to tune the hyper-parameter $\theta$**

The significance of deformed metrics lies in their interpolation between different metrics. Therefore, understanding the limiting cases could help us to tune the hyper-parameter $\theta$. In detail, one can decide $\theta$ for a specific deformed metric around its deformation boundary. This is how we select $\theta$, as shown in App. G.1.

For the specific Thm. 4.2, it describes the limiting cases ($\theta=1$ and $\theta \rightarrow 0$) of the deformed metrics of LEM, EM, and LCM. For example, for LEM, the candidate values of $\theta$ \{0.5, 1, 1.5\} are around the deformation boundaries of $\theta=0$ and $\theta=1$. We would not consider the values far from the deformation boundaries, such as $\theta=5$.

**Lem. 4.3 provides a succinct proof of deformed SPD MLRs**

Sec. 4 aims to present specific SPD MLRs under five families of deformed metrics. Note that all five deformed metrics are pullback metrics, which can be uniformly processed. Therefore, we present Lem. 4.3 to generally solve the RMLR under deformed metric, circumventing the lengthy ad hoc proofs.

Our specific SPD MLRs in Tab. 2 are all derived by Lem. 4.3. In detail, the SPD MLR under $2\theta$-BWM is derived by Eqs. (20-21), while other SPD MLRs are derived by Eqs. (18-19). Please refer to App. F.3 for detailed application of Lem. 4.3 for each metric. Perhaps we did not clarify this relation clearly in the original paper. We have highlighted this relation in our revised paper (P6).

Reviewer comment (Writing):
Can you explain more precisely the contributions of the paper? The ones presented in the introduction are too broad.

Author response:
**5. More precise contributions of the paper.**

This work presents a general framework for building Riemannian classifiers on manifolds and specifically showcases our framework on the deformed SPD manifolds. Please refer to CQ#1 in the common response for a detailed explanation.

Reviewer comment (Theory):
Can you provide an example of how to apply Theorem 4.2 and Lemma 4.3?

Author response:
**6. How to apply Thm. 4.2 and Lem. 4.3.**

**Thm. 4.2 helps to tune the hyper-parameter $\theta$**

The significance of deformed metrics lies in their interpolation between different metrics. Therefore, understanding the limiting cases could help us to tune the hyper-parameter $\theta$. In detail, one can decide $\theta$ for a specific deformed metric around its deformation boundary. This is how we select $\theta$, as shown in App. G.1.

For the specific Thm. 4.2, it describes the limiting cases ($\theta=1$ and $\theta \rightarrow 0$) of the deformed metrics of LEM, EM, and LCM. For example, for LEM, the candidate values of $\theta$ \{0.5, 1, 1.5\} are around the deformation boundaries of $\theta=0$ and $\theta=1$. We would not consider the values far from the deformation boundaries, such as $\theta=5$.

**Lem. 4.3 provides a succinct proof of deformed SPD MLRs**

Sec. 4 aims to present specific SPD MLRs under five families of deformed metrics. Note that all five deformed metrics are pullback metrics, which can be uniformly processed. Therefore, we present Lem. 4.3 to generally solve the RMLR under deformed metric, circumventing the lengthy ad hoc proofs.

Our specific SPD MLRs in Tab. 2 are all derived by Lem. 4.3. In detail, the SPD MLR under $2\theta$-BWM is derived by Eqs. (20-21), while other SPD MLRs are derived by Eqs. (18-19). Please refer to App. F.3 for detailed application of Lem. 4.3 for each metric. Perhaps we did not clarify this relation clearly in the original paper. We have highlighted this relation in our revised paper (P6).

Reviewer comment (Novelty):
The proposal on the “unified framework” is an overclaim. The paper provides nice results and detailed theoretical discussions for different metrics. However, there is still room for exploration to develop a “unified framework” such as extending the work for SPD manifolds with different structures, transformations, and classifiers. Therefore, it is suggested that the authors revise their claim considering the concrete results given in the paper, i.e., employment of 5 additional metrics for classification on SPD manifolds.

Author response:
**1.  “unified framework” --> "general framework".**

Thanks for your suggestive comments. Our RMLR in Eq. (14) is a general framework, as it only requires the existence of Riemannian logarithm $\mathrm{Log} _P Q$ for any pair $P, Q$ in the manifold $\mathcal{M}$. We call this property geodesic connectedness, and diverse manifolds in machine learning satisfy this property. Therefore, our framework can be implemented beyond SPD manifolds. We have extended our framework into the Lie group. Please refer to CQ#2 in the common response for more details.

Although most of the existing manifolds in machine learning are geodesic connectedness, we do not know if, in the future, there would be a powerful metric with the unknown Riemannian logarithm. Therefore, as you said, "unified" might be a bit overly absolute. We have changed the word "unified" to "general".

Reviewer comment (Theory):
Although theoretical discussions on different formulations of the metrics are nice, they should be extended by considering their complexity and equivalence properties. In addition, experimental analyses should be extended with additional datasets and backbones.

Author response:
**2. Experimental analyses should be extended with additional datasets and backbones.**

Our RMLR in Eq. (14) can readily apply to other manifolds or metrics. Please refer to CQ#2 in the common response for the experiments on $\mathrm{SO(n)}$.

Reviewer comment (Experiments):
Can you provide a comparative analysis of complexity (memory and running time footprints) of different metrics, both theoretically and experimentally (e.g., even for one task)?

Author response:
**3. Comparative analysis of complexity (memory and running time).**

**Memory cost:** Recalling Eq. (14) and Tab. 2, each class $k$ requires an SPD parameter $P _k$ and Euclidean parameter $A _k$ in SPD MLR. Therefore, the memory cost of the SPD MLR under each metric is the same.

**Computational cost:** The key factor of the computational complexity of SPD MLRs under different metrics lies in the number of matrix functions, such as matrix power, logarithm, Lyapunov operator, and Cholesky decomposition. These matrix functions are divided into two categories: one is based on eigendecomposition, and the other is the Cholesky decomposition. The following table summarizes the number of matrix functions in SPD MLR. Therefore, the general efficiency of SPD MLR should be EM>LEM>LCM>AIM>BWM. Note that Cholesky decomposition is more efficient than eigendecomposition. Without deformation, the order should be EM>LCM>LEM>AIM>BWM. We report the average training time per epoch in Tab. 10 in App. G.3. As shown in Tab. 10, EM-, LEM-and LCM-based SPD MLR is more efficient than AIM- and BWM-based MLR.

Table A: Number of matrix functions for each class $k$ in different SPD MLRs. a (b) means the number of matrix functions in the SPD MLR under the deformed (standard) metric.
| Metric | Eig-based Matrix Functions | Cholesky Decomposition | In Total |
|:------:|:--------------------------:|:----------------------:|:-----:|
| LEM    |             1 (1)            |           0 (0)           |  1 (1) |
| AIM    |             3 (2)            |           0 (0)           |  3 (2) |
| EM     |             1 (0)            |           0 (0)           |  1 (0) |
| LCM    |             1 (0)            |           1 (0)           |  2 (1) |
| BWM    |             5 (4)            |           1 (1)           |  6 (5) |

Reviewer comment (Experiments):
The accuracy of models is sensitive to hyperparameters of the metrics. How can a researcher estimate these hyperparameters in practice?

Author response:
**4. How can researchers estimate these hyper-parameters in practice?**

We have briefly discussed the hyper-parameters in Page 8 and App. G.1. Let us present a more detailed discussion. In fact, for a specific SPD MLR, there are at most three kinds of hyperparameters: Riemannian metric, deformation factor $\theta$, and $(\alpha, \beta)$. 

The most significant parameter is the choice of Riemannian metric, as all the geometric properties are sourced from a metric. A safe choice would start with AIM or BWM, and then decide whether to explore other metrics further. In detail, as indicated in Tab. 3-6, AIM and BWM generally achieve more robust performance than EM, LCM, and LEM. This could be attributed to the nice property of AIM [1] and BWM [2]. Besides, we also observe the promising performance of MLR under the deformed EM (Power Euclidean Metric (PEM)). Therefore, if efficiency is essential, PEM could be an excellent candidate metric.

For the defromation factor $\theta$, one can select its value around the deformation boudary, *i.e.,* \{0.5, 1, 1.5\} for $\theta$ and $\{0.25, 0.5, 0.75\}$ for $2\theta$. The deformation boundary has been systematically presented in Fig. 1. Let us take EM as an example. The $(\theta,\alpha,\beta)$-EM interpolates between $(\alpha,\beta)$-EM ($\theta=1$) and $(\alpha,\beta)$-LEM ($\theta \rightarrow 0$). Therefore, there are roughly three candidate values, $\theta=0.5, 1, 1.5$. Other metrics are similar. We made a more refined selection in our work for an extensive examination. Nevertherless, \{0.5, 1, 1.5\} for $\theta$ and \{0.25, 0.5, 0.75\} for $2\theta$ are enough as candidate values.

$(\alpha, \beta)$ can be set as $(1,0)$, as they are less important than the above two hyper-parameters. In detail, as $\alpha$ is less important as a scaling factor [1], one can set $\alpha=1$. $\beta$ only affects the trace part of the Riemannian metric (Eq. (1)), and does not affect Riemannian operators like Riemannian logarithm, exponentiation, and parallel transportation. Therefore, $\beta$ can also be set as 0. In our main paper, to extensively present the effect of each parameter, we also validate the effect of $\beta$. If one wants to do a more refined selection of $\beta$, one can follow the suggestion presented in P8 to tune $\beta$.

>[1] Thanwerdas Y, Pennec X. Is affine-invariance well defined on SPD matrices? A principled continuum of metrics.
>
>[2] Bhatia R, Jain T, Lim Y. On the Bures–Wasserstein distance between positive definite matrices.

Reviewer comment (Novelty):
The contribution is incremental as it is heavly based on the works of Nguyen & Yang (2023) and Thanwerdas & Pennec (2019a; 2022a).

Author response:
**1. Our work and contributions are significantly different from Nguyen & Yang (2023) [1] and Thanwerdas & Pennec (2019a; 2022a) [2-3].**

We have briefly discussed the difference in Rmks. 3.6 and 4.4. Here, we present a more detailed explanation.

**Difference with gyro SPD MLR [1]**

Our work differs from gyro SPD MLR [1] both in ***theory*** and ***practice***. 

***Theory:*** Our framework uses no gyro structure and is more general than gyro SPD MLR.

Firstly, the gyro MLR is based on gyro structures. However, not all metrics can induce gyro structures. In contrast, our method only requires Riemannian logarithm and parallel transportation (or group translation), which usually exists in manifolds in machine learning. Our method is thus more general and can be applied to more metrics.

Secondly, even if the gyro structure exists, the margin distance to the hyperplane in Gyro MLR is based on gyro distance and gyro trigonometry [1, Def. 2.22]. Note that gyro distance is defined by the tangent space at the identity [1, Def. 2.15]. However, the most natural counterparts on manifolds are geodesic distance and Riemannian trigonometry (Eq. 12). Therefore, we directly use geodesic distance and Riemannian trigonometry to obtain the margin distance Eq. (13).

Thirdly, Gyro MLR is obtained case-by-case for a specific gyro space, since the pseudo-gyrodistance [1, Def. 2.22] should be specifically solved for each gyro space. On the contrary, our work solves the margin distance in a general form (Thm. 3.4), and presents a general framework in Thm. 3.5. For a specific metric, one only needs to put the Riemannian logarithm and parallel transport into Eqs. (14-15). These operators exist in most of the manifolds in machine learning. 

***Practice:*** Our work implements five families of deformed SPD MLRs (Tab. 2), while gyro SPD MLR only implements gyro SPD MLRs under the standard AIM, LEM, and LCM. Besides, as we discussed in Rmk. 4.4, our SPD MLRs incorporate the results presented in [1]. Moreover, the BWM and EM are geodesically incomplete, while gyro operations require geodesic completeness [1, Eqs. (1-2)]. Therefore, EM and BWM could be problematic to induce gyro structures, as could gyro SPD MLRs. On the contrary, SPD MLRs can be easily obtained by our Thm. 3.5 for these two metrics. The following table briefly summarizes the difference between our SPD MLRs and gyro SPD MLRs. In addition, our RMLR in Thm. 3.5 can also be implemented into other manifolds, such as $\mathrm{SO}(n)$. Please refer to CQ#2 in the common response for our MLR in Lie groups.

Tab. 1 Difference between our SPD MLRs and the gyro SPD MLRs.
|       SPD MLR      | Metrics Involved | Theoretical Foundations | Geometric Requirement |
|:------------------:|:----------------:|:-----------------------:|-----------------------|
|  SPD MLRs  | The standard AIM, LEM, and LCM | Gyro-structures induced by the standard AIM, LEM, and LCM | Gyro-structures, geodesic completeness|
|  Ours  | $(\theta,\alpha,\beta)$-AIM, $(\theta,\alpha,\beta)$-LEM, $(\theta,\alpha,\beta)$-EM, $\theta$-LCM, and $2\theta$-BWM | Riemannian geometry | Geodesic connectedness|




**Difference with [2-3]**

Although power deformation is discussed in [2-3], they only cover the deformation of $(\alpha,\beta)$-AIM, BWM, and the standard EM. Our paper further generalized $(\alpha,\beta)$-LEM, LCM, and $(\alpha,\beta)$-EM into deformed metrics (see Sec. 4.1). Besides, we not only discuss the properties of all the deformed metrics in Fig. 1 and Tab. 1, but also validate the efficacy of deformation. Experimental results shown in Tabs. 3-6 indicates the effectiveness of the deformed metrics.

Reviewer comment (Presentation):
Experimental results are poorly presented (the text size in some tables, e.g. Tabs. 3 et 6 is too small to read).

Author response:
**2. Tabs. 3-6 is too small to read**

Thanks for the suggestion. After our work gets accepted, we will divide each table into two lines and move some stuff to supplementary.

Reviewer comment (Presentation):
Limitations are not discussed.

Author response:
**3. Limitations are not discussed.**

The main limitation of our work is the over-parameterized $P _k$ in our RMLR. Recalling our RMLR in Eq. (14), each class would require an SPD parameter $P _k$ and a Euclidean parameter $A _k$. Consequently, as the number of classes grows, the classification layer would become burdened with excessive parameters. We have added a discussion on this limitation in App. A and will address this problem in our future work.

Reviewer comment (Theory):
Clarification is needed on whether "geodesic connectedness" means "there exists a unique geodesic line connecting any two points," since existing literature indicates that a Riemannian manifold is connected if any two points can be joined by a broken geodesic.

Author response:
**4. Geodesic connectedness and connectedness in Props. 12.10 in [4]**

Thanks for your valuable comment. The geodesic connectedness in Def. 3.1 is a bit different from the connectedness in Props. 12.10 [4]. In fact, the connectedness mentioned in Props. 12.10 is a standard term derived from topology. Consequently, Props. 12.10 is presented as a proposition rather than a definition. Intuitively speaking, connectedness characterizes whether the space can be described as a whole. Please refer to [5, Ch. 3.5] for more details on connectedness. In contrast, our geodesic connectedness only describes whether two points can be connected by a geodesic, and we do not think of broken geodesics. Of course, geodesic connectedness implies connectedness, but connectedness does not infer geodesic connectedness.

Whether Def. 3.1 is strong depends on the context. For example, on general manifolds, even the geodesic line could be less meaningful as a measurement, as the geodesic distance could be independent of the geodesic line [6, P146]. However, this paper focuses on the Riemannian manifolds in machine learning. In this context, Def. 3.1 can be satisfied by many manifolds and metrics, such as SPD manifolds, Grassmannian manifolds, Stiefel manifolds, and various kinds of Lie groups. To the best of our knowledge, we have not met the metrics that can be used in machine learning but do not satisfy Def. 3.1. We therefore think this requirement is weak in the context of machine learning.

Besides, the proof of Thms 3.4-3.5 only requires the existence of Riemannian logarithm $\mathrm{Log} _P Q$ for any pair $P, Q \in \mathcal{M}$. However, we do not find a term in standard Riemannian geometry to characterize this property. As the existence of $\mathrm{Log} _P Q$ literally means geodesic connectedness, we define this property as geodesic connectedness. If you have any better suggestions to characterize this property, we are willing to make changes accordingly.

Reviewer comment (Theory):
Clarification is requested on the requirements for the proposed approach to be applicable. If a unique geodesic between any two points is required, the range of applicability might be as limited as Nguyen & Yang (2023).

Author response:
**5. Whether the range of applicability of the proposed approach is as limited as Nguyen & Yang (2023) [1]**

Our applicability is broader than gyro MLR. For the gyro MLR, it has two strong requirements: geodesic completeness and the existence of gyro structures. Firstly, the definition of gyro operations [1, Eqs. (1-2)] implicityly require the geodesic completeness. Many incomplete metrics exist, such as BWM, EM, and generalized BWM [8] on SPD manifolds. Secondly, even if a metric is complete, Eqs. (1-2) in [1] should further satisfy the axioms of gyro space [7, Def. 2.3]. On the contrary, our RMLR (Eq. (14)) merely requires the Riemannian logarithm, which is satisfied by various manifolds or metrics in machine learning.

Besides, even if a metric admits gyro structure, to obtain gyro MLR, one still needs to obtain the pseudo-gyrodistance to a hyperplane. In contrast, we have finished all the derivations in a general form. Therefore, one only needs to put Riemannian operators in Eq. (14) and then obtain the final formulation. Moreover, as discussed in Rmk. 4.4, our results on the specific SPD manifolds cover the results presented in [1].

Reviewer comment (Novelty):
Doubts about the statement at the end of Section 4.2 that "our work is the first to apply EM and BWM to establish Riemannian neural networks," since Han et al. (2021) has already studied Bures-Wasserstein geometry for Riemannian optimization on SPD manifolds with various machine learning applications.

Author response:
**6. Doubts on the statement "our work is the first to apply EM and BWM to establish Riemannian neural networks**

Although both [9] and our work use BWM, our work differs from [9] in the application of BWM. [9] applies BWM to optimize the SPD parameters in machine learning algorithms. In contrast, we use BWM to directly build Riemannian networks. Constructing optimization algorithms differs from building networks for a specific metric in considering backpropagation (BP). Optimization algorithms do not need to implement BP. However, building Riemannian networks needs to pay special attention to BP, as Riemannian computation usually requires several types of matrix decompositions. For example, in App. F.2.1, we discuss the instability in the BP of parallel transportation of BWM. Besides, we design the numerical stable BP for the Lyapunov operator in BWM in App. F.2.2. Considering the additional efforts to complete the BP and different usages, we therefore think it is safe to claim that we are the first to use BWM for building networks.

Reviewer comment (Theory):
In Remark 3.2, it is unclear if item (a) is the authors' original observation or a known result. If original, a brief proof should be provided; if known, it should be properly cited.

Author response:
**9. Item (a) in Remark 3.2**

This can be derived by definition. Let us formalize this claim and present the proof. Besides, we have added the following lemma and proof in App. C.

**Lemma 1.** A manifold $\mathcal{M}$ is geodesically connected iff $\mathrm{Log} _P Q$ exists for any pair of $P, Q \in \mathcal{M}$.


*Proof:* $\Leftarrow:$ This can be directly obtained by the definition of Riemannian logarithm. The existence of $\mathrm{Log} _P Q$ means that there is a geodesic $\gamma$ connecting $P$ and $Q$ with an initial velocity of $\mathrm{Log} _P Q \in T _P \mathcal{M}$.

$\Rightarrow:$ Two points $P,Q \in \mathcal{M}$ are connected by a geodesic $\gamma(t;P,V)$ starting from $P$ with the initial velocity $V \in T _P\mathcal{M}$. In other words, there is a $t _Q$ such that $Q = \gamma(t _Q;P,V)$. By the homogeneity of $\gamma$, we have $Q = \gamma(t _Q;P,V) = \gamma(1;P,t _Q V)$. As the inverse of Riemannian exponential map $\mathrm{Exp} _P (t _Q V) = \gamma(1;P,t _Q V)$, the Riemannian logarithm is $\mathrm{Log} _P (Q)= t _Q V$. 

Therefore, our Def. 3.1 is equivalent to the existence of $\mathrm{Log} _P (Q)$ for any pair of $P, Q \in \mathcal{M}$.

Reviewer comment (Novelty):
The novelty is not clear. Using SPD matrices for human action recognition and EEG is not new.

Author response:
This paper presents a general framework for building Riemannian Multinomial Logistic Regression (RMLR) on manifolds, and specifically showcases five implementations on SPD manifolds. Please refer to CQ#1 in the common response for detailed discussions about our contributions and novelty.

Reviewer comment (Novelty):
The authors should clarify the novelty of the proposed approach and reorganize the paper.

Author response:
This paper presents a general framework for building Riemannian Multinomial Logistic Regression (RMLR) on manifolds, and specifically showcases five implementations on SPD manifolds. Please refer to CQ#1 in the common response for detailed discussions about our contributions and novelty.