From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper got one 10 (strong accept), two 6s (marginally above threshold) and one 5 (marginally below threshold). The major weakness of the paper includes that the convergence property was not made clear, missing some references, and the experiments are a bit weak. The authors mostly addressed the reviewers' concerns. By the overall scores and the novelty of the paper, the AC recommended acceptance.
justification_for_why_not_higher_score: Three reviewers did not give strong support to the paper. They only gave borderline scores.
justification_for_why_not_lower_score: This paper got the highest average scores among the papers I handled. I think the idea of structure preserving discretization and momentum is good.
summary_of_AC-reviewer_meeting: N/A
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper5483/Authors
title: Thank you
comment: Thanks for your reply. Indeed, one remarkable fact about Algorithm 3, thanks to Prof. Higham, is its quadratic convergence without even requiring any matrix inversion. The book you mentioned seems to be another very educational reference, and we sincerely appreciate it!

From: ICLR.cc/2023/Conference/Paper5483/Area_Chair_CatB
title: Thanks for clarification!
comment: Your clarification is fairly enough. What I wrote "Higham 1997" actually referred to his book: Functions of Matrices: Theory and Computation (my fault), where on page 153 it was written that m=0, in which m is the order of convergence stated in Theorem 6.11. Usually, quadratic convergence requires inversion, but there is no inversion in Algorithm 3. That was why I doubted.

From: ICLR.cc/2023/Conference/Paper5483/Authors
title: Convergence speed of Algorithm 3
comment: Dear Area Chair CatB,

Thank you very much for the question and an opportunity of clarification.

The Higham 1997 paper that we referred to is the one published in Numerical Algorithms 15 (1997) from page 227 to page 242. We were unable to find page 153 where the convergence rate was stated to be of order 0. Higham's eq(2.6b) on page 231 was what we used in Algorithm 3, and right below eq(2.6b) it was written "we have **quadratic** convergence of $Y_k$ to $A^{1/2}$ and $Z_k$ to $A^{-1/2}$". The same statement was repeated several times throughout this reference.

This quadratic convergence was also consistent with our numerical experiments, such as those in Fig.5c.

A side remark is, Algorithm 3 indeed helps us improve the computational efficiency, but it is not the sole reason.  Our speed mainly comes from the automatic structure-preservation of momentum, and quantitative evidence was provided in Appendix C.

Thanks again!

Best wishes,

Paper5483 Authors

From: ICLR.cc/2023/Conference/Paper5483/Area_Chair_CatB
title: About Algorithm 3
comment: Dear Authors,
You wrote above Algorithm 3 that the Newton-Schur iteration presented in Algorithm 3 is quadratically convergent. However, I am sketptic about this as it is written on page 153 of (Higham 1997) that the convergence rate is of order 0, which means that it can only be sublinear at the most. Since Algorithm 3 is critical for reducing the computation cost of your algorithm, could you elaborate on this issue? Thanks!

AC

From: ICLR.cc/2023/Conference/Paper5483/Authors
title: Thank you
comment: We sincerely thank the area chair, the reviewers, and everyone involved, for working on and helping improve our paper, as well as your precious time.

From: ICLR.cc/2023/Conference/Paper5483/Area_Chair_CatB
title: Please respond to author rebuttals
comment: Dear Reviewers,

The authors have submitted their rebuttals. Please have a look and respond to their efforts. This will be a respect to their hard work. Many thanks!

Area Chair

From: ICLR.cc/2023/Conference/Paper5483/Reviewer_AYdG
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper develops a gradient-based solution for optimizations if objective functions on a Stiefel manifold. It uses the geometry of Sitefel manifolds along with momentum-based ideas for speeding up gradient searches to develop an intrinsic solution for the optimization. It is quite elegant in its use of geometry and in ensuring that the iterative updates stay on the required spaces (manifold and the tangent bundle). I did not follow all the details in the construction but trust that the paper is solid in its development. From an application point of view, the focus is on training neural networks where orthogonality is important. The experimental results involve training ViT from scratch using different orthogonality constraints and the proposed method is clearly found to be better. 
strength_and_weaknesses: Strengths: 

This is a strong paper on exploiting the differental geometry of Stiefel manifolds and bringing some of the momentum-based improvements from Euclidean domains to this manifold. The experimental results are quite satisfactory. 
clarity,_quality,_novelty_and_reproducibility: 
It is a well written paper. It is naturally heavy on differential geometry of Sitefel manifolds and the formulations of solutions as dynamical systems. Perhaps some of this can be postponed to the appendix. 

Also perhaps some early research in computer vision on optimization on Sitefel manifolds can also be cited, especially for comparing with Stochastic Gradient Descent and simulated annealing techniques. See, for example, 
Xiuwen Liu, A. Srivastava and K. Gallivan, "Optimal linear representations of images for object recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 5, pp. 662-666, May 2004, doi: 10.1109/TPAMI.2004.1273986.

summary_of_the_review: 
A strong paper on solving general optimization problems on Stiefel manifolds. Impressive level of details in 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 10: strong accept, should be highlighted at the conference

From: ICLR.cc/2023/Conference/Paper5483/Reviewer_RJLh
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors considered the problem of optimization on Stiefel manifold. Heuristically, the proposed method can be divided into the following steps: 1) formulate a variational principle which naturally constraints the trajectory on a Stiefel manifold, hence it exactly preserves the manifold structure without extra operations such as projection / retraction 2) With a smart discretization in time of the derived ODEs, the authors came to an discrete-time optimization scheme.

The provided thorough theoretical analysis, including: 1) detailed derivation of the variational principle 2) proof of the structure preserving property 3) discretization scheme

Finally, the authors studied the benefits of orthogonality, including orthogonality at initialization, within an attention head, and across attention heads. The author validated the proposed method in experiments.
strength_and_weaknesses: Strengths:
1) The proposed method looks pretty interesting and novel to me.The authors had a clear presentation from the general idea of constraint optimization on the manifold as a continuous-time process with continuous-time constraints, then to a proper discretization scheme.
2) The problem to be studied is important. The authors have clearly demonstrated that orthogonality helps with the generalization performance of machine learning models, and have demonstrated the benefits of the proposed method with extensive studies.
3) The theoretical analysis and proofs are solid. I only loosely checked the theoretical analysis and it looks pretty convincing to me.

Weakness:
1) It might be better to start with toy examples to introduce the idea of Stiefel optimization, especially for readers like me who are not familiar with this specific field.
2) I would suggest the authors conduct experiments on more challenging datasets rather than Cifar.
clarity,_quality,_novelty_and_reproducibility: Quality:
Good quality in terms of novel and solidness

Clarity:
Good presentation

Originality:
Novel
summary_of_the_review: Overall the paper proposed a novel method to solve the Stiefel optimization problem without extra steps like projection / retraction. The idea is novel and well validated.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5483/Reviewer_TSJE
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This papers considers the problem of optimizing a function on the Stiefel manifold. The main objective of the paper is to build algorithms with momentum, which is often difficult / expensive in a Riemannian context. The key idea of the authors is to consider an ODE with momentum that converges to the solutions of the problem, and then to have a judicious discretization process that produces feasible iterates, while maintaining a relatively low cost (only one projection on the manifold should be computed at each iteration). The authors also extend the celebrated adam algorithm in this setting.

The authors then demonstrate the promises of their approach to train vision transformers with orthogonal attention heads and for computing projection robust wasserstein distances.
strength_and_weaknesses: # Strengths 
- This paper considers a timely problem : optimization on the stiefel manifold arises naturally in many machine learning problems and deep learning problems, and considering the success of momentum-based techniques  / adam in the Euclidean case, it is important to have extensions of these methods to the stiefel manifold

- the proposed algorithm seems sound, and is computationally quite cheap (on par with classical Riemannian SGD since we have 1 projection per iteration).

- The numerical evaluation is satisfactory, the advantage of the proposed algorithm is clearly illustrated.

# Weakness

- The striking weakness of this paper is the lack of any theoretical result regarding the actual convergence of the proposed method. This is an optimization paper and yet there is no convergence result (while the method seems to work well in practice). This is a clear hole in the plot of the paper: what can we theoretically expect from this method ? Do the iterates of algorithm 1 decrease the loss function?
clarity,_quality,_novelty_and_reproducibility: - The paper is very clearly written and easy to read

- The paper is of good quality, the experimental validation is great and convincing. Still, the theoretical analysis is missing.

- The paper proposes an original method to incorporate momentum for optimization on the stiefel manifold.

summary_of_the_review: This is a paper that this easy and pleasant to read, that proposes a new method for an important problem in ML and that convincingly demonstrates its utility in practice. However, the lack of any theoretical result is problematic to me.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5483/Reviewer_y3Tv
confidence: 1: You are unable to assess this paper and have alerted the ACs to seek an opinion from different reviewers.
summary_of_the_paper: This paper presents a Momentum Stiefel Optimizer with several pleasant properties. 
strength_and_weaknesses: The paper is well structured and well written; Since I was not familiar with this field and not able to confirm every single proof/algorithm, I am unable to assess this paper.
clarity,_quality,_novelty_and_reproducibility: The paper is well structured and well written; The authors provide the code to reproduce the results.
summary_of_the_review: The paper is well structured and well written; Since I was not familiar with this field and not able to confirm every single proof/algorithm, I am unable to assess this paper.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold