Paper: Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold (ICLR-2024)

Reviewer comment (Theory):
The claim 2 in the introduction as well as the section 4.1 are misleading. Indeed, the authors mention they don't use retraction or vector transport to reduce the computation. However, they use the orthogonal projection onto the Stiefel manifold, which is a retraction, and the orthogonal projection onto the tangent space, which is a vector transport (see "Optimization on matrix manifolds" from Absil et al. 2008). This claim should be removed.

Author response:
[This claim 2 should be removed] Indeed, we don't use retraction directly. However, we still need to bound the projection operator using retraction and vector transport based on Eq.19 and Eq.20. Thus, we must give the complete claim of retraction and vector transport.

Reviewer comment (Theory):
The paper lacks an overview of the proof to get the global convergence. The different proofs are long and technical and an overview would help the reader.

Author response:
[The paper lacks an overview of the poof] Thanks for your suggestion. We have added an overview of the proofs in Figure 5 in the modified version.

Reviewer comment (Writing):
Section 4.1: "The Riemannian gradient step with a unit step size, i.e., ..." — is it a unit step size or a null/zero step size?

Author response:
[Is it a unit step size] It follows from the setting of DRDGD and DPRGD that a unit step size means the coefficient before $\sum_{j=1}^n W_{i j}^t x_{j,k}$ is 1.

Reviewer comment (Theory):
Lemma 1, eq (24): how do you get the second and third inequalities? There appears to be something wrong here.

Author response:
[Lemma 1, eq (24): how do you get second and third inequalities] We miss a parenthesis in the second inequality, which has been corrected in the modified version. And $0 \leq \Vert x \Vert, \Vert y \Vert \leq 1$ is implicit in this proof.

Reviewer comment (Novelty):
Theorem 2 is independent from the conjugate gradient. Is it new or is it a known result from a different paper?

Author response:
[Is Theorem 2 new or is it a known result from a different paper] In fact, similar results have been given in [1], which we extended from retraction to projection operator and made it compatible with the conjugate gradient.

Reviewer comment (Theory):
After eq (31), you mention that $x_{i,k_0+1} \to x_{i,k_0}$. This limit is not understood — is it a mistake?

Author response:
[$x_{i,k_0+1} \rightarrow x_{i,k_0}$] There is no problem. It means that $x_{i,k_0+1}$ will no longer be updated and will remain at the previous point $x_{i,k_0}$ as the consensus error converges and the gradient approaches 0. A similar statement can be found in Theorem 4.2 based on [2].

Reviewer comment (Theory):
Second inequality in eq (36) — can you explain how you get it?

Author response:
[Second inequality in eq (36)] First, the second part comes from Eq. 35. Then, the third part is $$ \Vert \mathcal{P}\_{T\_{x\_{i,k}}\mathcal{M}}\left(\sum\_{j=1}\^n W\_{i j}\^t \eta\_{j,k-1}\right) \Vert \_{x\_{i,k}}\^2  \leq \Vert \sum\_{j=1}\^n W\_{i j}\^t \eta\_{j,k-1} \Vert \_{x\_{i,k}}\^2  \leq \Vert  \eta\_{i,k-1} \Vert \_{x\_{i,k-1}}\^2$$, where $ \Vert \sum\_{j=1}\^n W\_{i j}\^t \eta\_{j,k-1} \Vert \_{x\_{i,k}}\^2  \leq \Vert  \eta\_{i,k-1} \Vert \_{x\_{i,k-1}}\^2$ is a similar assumption followed with formula (4.28) in [2].

Reviewer comment (Theory):
Form of convergence guarantees is not exhaustive, because the rate of the convergence is not established.

Author response:
Thank you for your efforts in improving the quality of our paper. In fact, in the framework of conjugate gradient, the convergence rate has not been established so far. Naturally, the decentralized version we derive from this framework can not provide the convergence rate as well.

Reviewer comment (Novelty):
Theoretical framework is mostly inherited from Zhu's original papers, but that analysis does not allow providing guarantees on convergence rate, so does not this paper, which means that there were no significant extending of that framework.

Author response:
Thank you for your efforts in improving the quality of our paper. In fact, in the framework of conjugate gradient, the convergence rate has not been established so far. Naturally, the decentralized version we derive from this framework can not provide the convergence rate as well. We guess the view inherited from Zhu's original papers is caused by the framework of the Riemannian conjugate gradient. Different studies revolve around geometric tools without affecting the theoretical framework of conjugate gradient, e.g., inverse retraction [1] and vector transport [2]. Specifically, we present a quite simple geometric tool (projection operator), and extend it to decentralized scenarios, which obviously makes a significant extension to unify the framework of conjugate gradient and decentralized optimization.

Reviewer comment (Writing):
Typo in "Lemma 3 In Alogrithm"

Author response:
1. We have modified the corresponding typo in the modified version.

Reviewer comment (Experiments):
What about time-varying case? Conidering the case of time-varying graph would be important for all-around extending CG for decentralised setting.

Author response:
2. This is a very meaningful and interesting idea for the time-varying graph in decentralized CG. We also notice that there have been some recent works focusing on time-varying graphs, e.g., [3], [4], [5]. By reviewing these works, we find that this paper can be extended to time-varying cases because we have no additional requirements for decentralized settings. Of course, there are a large number of proofs to be considered that would go well beyond the scope of this paper and would be sufficient to form a new paper. We will consider this exciting setting in future work.

Reviewer comment (Theory):
There is still room for improvement in the clarity of the proof. Some symbols that appear in the convergence analysis section of the text, such as $g_{i,k+1}$, $\mathcal{N}$, and $C$, are not defined in the text.

Author response:
Thank you for your valuable suggestions, and we have added the definitions of the corresponding symbols in the modified version.

Reviewer comment (Theory):
It seems that there are some assumptions about the step size $\alpha_{k}$ that are not mentioned in Assumption 3 about $\alpha_{k}$ in the body of the proof, such as the assumption about $\alpha_{k}$ in Lemma 2. This leads to unclear assumptions about $\alpha_{k}$.

Author response:
Indeed, the step size $\alpha_k$ should be bounded as the description of Lemma 2, and we have added this part to Assumption 3.

Reviewer comment (Evaluation):
The measures mentioned in the simulation should converge towards 0, which is not well demonstrated in the experimental results. For example, the measures in Figure 3 tend to be constant after it drops to a certain level. This does not support the theoretical results very well.

Author response:
In theoretical analysis, we require the step size to be infinitely close to 0. However, for a fair comparison with different methods in the experiment, we employ fixed step sizes, i.e., $\alpha_k=\frac{\hat{\alpha}}{\sqrt{K}}$ with $K$ being the maximal number of iterations. Obviously, this causes the measures only drop to a certain level.

Reviewer comment (Theory):
The definition of doubly stochastic matrix seems to be $\sum_{i} x_{ij}=\sum_{i} x_{ij}=1$, which is different from the definition in Assumption 1.

Author response:
In our opinion, the definition in Assumption 1 puts forward higher requirements for a doubly stochastic matrix, including symmetry and eigenvalues, but it still meets the properties of a doubly stochastic matrix. Actually, the completely consistent definition is also given in Assumption 1 based on [1].

Reviewer comment (Theory):
It seems like there are many assumptions of the parameters such as $\alpha_{k}$. Are these assumptions easy to satisfy?

Author response:
Many assumptions are built into conjugate gradient, e.g., the Armijo condition and strong Wolfe condition, and their purpose is to theoretically prove the convergence of conjugate gradient. All assumptions only ensure that the algorithm converges in the worst case, and they can be satisfied from the view of practical convergence.

Reviewer comment (Theory):
In this paper, the decreasing step size is used in the convergence proof, while the fixed step size is used in the simulation. Does this difference affect convergence?

Author response:
The fixed step size causes the measures approach to be constant (very close to 0) after it drops to a certain level, rather than continuously dropping as the decreasing step size.

Reviewer comment (Novelty):
What is the significance of the eigenvalue problem used in the simulation in real life?

Author response:
The eigenvalue problem is a fundamental concept in linear algebra with widespread applications in various fields, including physics, engineering, computer science, and statistics. In the context of simulations and real-life applications, the eigenvalue problem is particularly significant for several reasons: (1) Eigenvalues are used in control theory to analyze the stability of control systems. (2) Eigenvalue decomposition is utilized in various algorithms for dimensionality reduction and feature extraction in machine learning. (3) Eigenvalue problems are foundational in spectral methods used in numerical simulations.

Reviewer comment (Evaluation):
In general, since the problems solved are the same, the convergence result of the distributed algorithm should be independent of the structure of the graph if the assumptions about the graph are satisfied. In Figure 3 of the simulation, the same algorithm seems to converge to different solutions under different graphs. Why did this happen?

Author response:
In practice, the proposed method indeed achieves convergence under different graphs, which shows that their convergence is indeed independent of the structure of the graph. However, different graph structures will cause different efficiency of information transmission, thus affecting the convergence trend.