Paper: Learning Unorthogonalized Matrices for Rotation Estimation (ICLR-2024)

Reviewer comment (Theory):
Lack of discussion about other possible ways to constrain orthonormality in learning problems. While the problems of orthogonalization in the learning problem may bring issues as described in the paper, there are other possible solutions that the paper does not discuss. For example: (i) In 3D computer vision, some camera-pose estimation solvers constrain the rotation with \(I - M^{\intercal}M\) and \(\text{det}(M)=1\) pose problems and show good estimation results (see Reference [A]). Why couldn't this trick replace the SVD or Gram–Schmidt orthogonalization procedures? Although the paper does not discuss a learning approach, the trick is applicable to any learning problem. Unfortunately, the paper lacks discussion about these tricks and whether they also present problems when learning.

Author response:
**Q1: Contrain orthonormality with $I-M^{\intercal}M$?**

A1: Thank you for your suggestion. We compare our method with adding the rotation constraint loss, which is implemented by ProHMR [1]. The PA-MPJPE is 55.3 vs. 54.8 (PRoM). The experimental setting is the same as the ablation study. Adding this loss makes the learning in between PRoM and previous orthogonalization incorporated methods. The results are also in between.

[1] Kolotouros, Nikos, et al. Probabilistic modeling for human mesh recovery. CVPR'21

Reviewer comment (Experiments):
An alternative to the pipeline shown in Figure 1 is not discussed. Learning rotation with neural networks does not need to follow that pipeline. One can easily use a quaternion parametrization, remove \(\mathcal{L}_{\theta}\) and thus remove the orthogonalization, and simply add a regularizer term that enforces the norm of the quaternion to be one.

Author response:
**Q2: Quaternions are also used in other works; Continuity lacks solid theorems**

A2: In our work, we focus on whether to use orthogonalizations during training with the matrix representation. 6D is one of the representative works that incorporate orthogonalizations. 6D is also frequently used in pose estimation tasks. Our work mainly improves upon 6D representation, which will have an impact on the understanding of learning rotation matrices and the further learning paradigm of pose estimation. We acknowledge that other rotation representations, e.g., quaternions, are used in other works, comparing 6D with quaternion with solid theorems is out of the scope of our paper. 

However, we still provide a more detailed empirical comparison among all rotation representations. The experimental setting is the same as the ablation studies, e.g., Table 3 (a). The results are shown below.

| Representation | Quaternion | axis-angle | Euler | 6D   | SVD  | PRoM |
|----------------|------|------------|-------|------|------|------|
| PA-MPJPE       | 62.8 | 64.3       | 66.8  | 57.5 | 56.7 | **54.8** |

Reviewer comment (Theory):
The continuity issues discussed in the submission and used to motivate the problem are questionable. One of the main claims of the paper in Learning for Rotations in Section 2 states that the parametrization of 3D rotations with four or fewer dimensions is discontinuous and non-ideal for learning (according to Zhou et al. 2019). However, Zhou et al. 2019 defines the continuity concept from an intuitive example and lacks solid theorems showing that these definitions and observations strictly apply to learning rotations. Subsequent works using quaternions show that they work well with neural networks (see Reference [B] below). Reference [B] conflicts with the observations of Zhou et al., which form the foundation of this work and explore pseudo-rotation matrices. Thus, this puts into question the main assumptions stated in Section 2.

Author response:
**Q2: Quaternions are also used in other works; Continuity lacks solid theorems**

A2: In our work, we focus on whether to use orthogonalizations during training with the matrix representation. 6D is one of the representative works that incorporate orthogonalizations. 6D is also frequently used in pose estimation tasks. Our work mainly improves upon 6D representation, which will have an impact on the understanding of learning rotation matrices and the further learning paradigm of pose estimation. We acknowledge that other rotation representations, e.g., quaternions, are used in other works, comparing 6D with quaternion with solid theorems is out of the scope of our paper. 

However, we still provide a more detailed empirical comparison among all rotation representations. The experimental setting is the same as the ablation studies, e.g., Table 3 (a). The results are shown below.

| Representation | Quaternion | axis-angle | Euler | 6D   | SVD  | PRoM |
|----------------|------|------------|-------|------|------|------|
| PA-MPJPE       | 62.8 | 64.3       | 66.8  | 57.5 | 56.7 | **54.8** |

Reviewer comment (Experiments):
The evaluations consider two baselines (Zhou et al., 2019; Levinson et al., 2020). Both methods use 6D representations with differences in normalization. A comparison to more recent methods [1,2] should be more convincing.

Author response:
A1: Thank you for pointing this out. Considering [B] didn’t release their code, we reproduce the results of [A] with our ablation study setting. With the only change of rotation representation from 6D to RPMG-6D, [A] achieves the PA-MPJPE of  55.8, whereas PRoM has a lower error of 54.8. We want to highlight that one of the main contributions of our paper is that we achieve much better results on the challenging real-world human pose datasets, e.g., 3DPW and Human3.6M.

Reviewer comment (Novelty):
The proposed method chose to base on CLIFF (Li et al., 2022). Since the proposed method is not a novel framework, it is generally applicable to any existing methods for pose estimation. Hence, I assume that PROM could be used together with baselines to also improve learning.

Author response:
A2: Yes, PRoM will improve other baselines. We have already included these experiments in the last paragraph of Sec. 5.3. The detailed numbers are presented in Table 4 in the Appendix due to the space limit.

Reviewer comment (Novelty):
In the area of camera pose regression, it has been shown that predicting pseudo rotation matrix during training works in [3] and quantanion without orthogonalization is also possible [4]. As a result, the novelty of this work might be reduced to introducing the simple idea to other tasks.

Author response:
A3: Yes, also as reviewer h3do commented, direct regression of rotation matrices is intuitive and has been used in the past literature. However, our contribution is mainly the gradient and optimization analysis, which shows that incorporating orthogonalization in the training is problematic. Our work gives theoretical support for why using unorthogonalized matrices is more beneficial.

Reviewer comment (Novelty):
Most claims in the paper are either vague or not surprising (not showing what the authors intended to show). Most importantly, the proposed method (using a plain 3x3 matrix without any orthogonalization and instead guiding it to a rotation matrix using an additional loss, i.e., soft constraint) is a somewhat basic technique that has been frequently used in the literature. Accordingly, I'm afraid the paper has little merit to the research community.

Author response:
**Q1: The proposed method is somewhat basic and is frequently used in the literature.**

A1: PRoM is just a simple yet effective solution according to our optimization analysis, which is the main contribution of our paper. Most works still incorporate orthogonalizations, e.g., in the fields of point cloud pose estimation, human pose and shape estimation tasks. People tend to apply orthogonalizations on rotation matrices to ensure it’s valid to rotate the downstream inputs. It’s a tradeoff between the validity of rotation matrices and optimization. Our results show that the optimization matters more.

Reviewer comment (Theory):
Section 3.4.1 (and Appendix A.3) basically says that (5) has two more terms other than the first term, so the gradient can be scrambled. The claim itself is vague, but before that, what the authors need to show is not that there are two more terms, but that these two terms do not die out (and actually steer the overall gradient towards funny directions) in the majority of the input space. In (12)-(16), we can actually see that there are many duplicating expressions, as well as many terms related to cross-product, inside those three terms. This suggests that the two terms (related to \( r_2 \) and \( r_3 \)) might actually be canceled, either by themselves or by combining the two. I actually showed this in my previous review for a specific case provided in the previous manuscript. This might hold for larger regions in the input space if we do the manipulation. The authors must explicitly derive against this to justify their claim, not suggesting them as in the current paper.

Author response:
**Q2: Eq. (5) is vague and need to explicitly derive the two terms will not die out?  
Q4: Fig. 3 is vague? Incorporating orthogonalization is obviously more complex.**

A2 & A4: We answer the concerns together since they are related. For the ambiguous gradient, we only give some intuitive derivation (Eq. (5)) and Fig. 2 (a) is the exact empirical verification. Fig. 3 serves as a supplement to Fig. 2 (a). We leave explicit theoretical support for this in Sec. 4.2. We answer the questions separately below in detail. 

A2: Eq. (5) is an observation of the gradients from the orthogonalizations and serves as the motivation of PRoM. The explicit influence of these conflicting terms on the optimization is shown in Sec. 4.2. We leave strong theoretical proof for the optimization analysis instead of gradient because the optimization analysis is more established in the convergence and optimization papers (see references below) Therefore, we think it’s more convincing to demonstrate the negative impact of the orthogonalizations in the optimization analysis. Also, the two terms will not die out as shown in Fig. 2 (a).

Reviewer comment (Theory):
Theorems in Section 4.2 are not surprising, and most importantly, they do not prove anything. What they are showing are (specific) upper bounds of losses for orthogonalized and unorthogonalized cases. This does not guarantee anything. They do not guarantee that the actual loss will be lower for the unorthogonalized case, and moreover, there is no proof that they are the best upper bounds we can have (so it is not correct to say that the unorthogonalized case has a better chance of a lower loss).

Author response:
**Q3: better upper bound does not guarantee anything?**

A3: The upper bound analysis of loss is widely adopted both in machine learning theory and its application in vision [1,2,3,4,5,6]. It both considers the convergence rate and irreducible error. Our work follows the convention of analyzing the different upper bounds for orthogonalization incorporated methods and PRoM. For the whole term, the only difference is caused by the orthogonalization function, which gives non-negligible $\psi(B_i)$ and further leads to a slower convergence rate and worse upper bound. Additionally, we show by experiments that PRoM converges much faster than orthogonalization incorporated methods (Fig. 3 (a) and (b)).

[1] Cohen, Jeremy M., et al. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR’21

[2] Ahn, Kwangjun, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. ICML’22

[3] Fehrman, Benjamin, Benjamin Gess, and Arnulf Jentzen. Convergence rates for the stochastic gradient descent method for non-convex objective functions. JMLR’20

[4] Mertikopoulos, Panayotis, et al. On the almost sure convergence of stochastic gradient descent in non-convex problems. Neurips’20

[5] Lei, Yunwen, et al. Stochastic gradient descent for nonconvex learning without bounded gradient assumptions. TNNLS’22

[6] Chen, Joya, et al. DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training. ICLR’23

Reviewer comment (Evaluation):
The meaning of Fig. 3 is vague. It simply shows that the correlation between the gradient and \( t_{11} \) is not apparent in Euclidean space. This has nothing to do with the stability or ambiguity of the gradient. More importantly, it is obvious that the relation between the gradient and the parameter is more complex for orthogonalization than for plain identity mapping.

Author response:
**Q2: Eq. (5) is vague and need to explicitly derive the two terms will not die out?  
Q4: Fig. 3 is vague? Incorporating orthogonalization is obviously more complex.**

A2 & A4: We answer the concerns together since they are related. For the ambiguous gradient, we only give some intuitive derivation (Eq. (5)) and Fig. 2 (a) is the exact empirical verification. Fig. 3 serves as a supplement to Fig. 2 (a). We leave explicit theoretical support for this in Sec. 4.2. We answer the questions separately below in detail. 

A4: Fig. 3 is an empirical study with different $\sigma$s for comprehensive understanding. The focus is that the larger $\sigma$ is, the more diversity of gradients will be. It serves as a supplement of Fig. 2 (a). As stated in the review, it is indeed obvious that the gradient is more complex for orthogonalization. We are actually targeting the complex gradient and demonstrating both the theoretical and empirical benefits of removing this complex gradient during the training.

Reviewer comment (Experiments):
What if we do not have ground truth rotations? Without \( L_\theta \), it is possible that PRoM deviates too far from the rotation space, and this can be detrimental to having a correct solution. What should we do then?

Author response:
**Q5: What if we do not have ground truth rotations?**

A5: Our analysis still applies when ground truth rotations are not available. In Theorem 1, we do not need the presence of the ground truth $R$, but only the predicted rotation matrices $\hat{R}$. Also, empirically, in sec. 5.2.3, we show that when the whole task only has the downstream supervision without ground truth rotation, it can still perform better than orthogonalization incorporated methods.