Paper: A primal-dual perspective for distributed TD-learning (ICLR-2024)

Reviewer comment (Writing):
There exist some grammar issues, such as "there exists \bar{h}_1 amd \bar{h}_2..." "the proposed distributed TD-learning do not require". Please check and polish the presentation.

Author response:
Thank you for the constructive comments. We will polish the presentation and check the manuscript carefully. Finally, as for clarification, we do not require doubly stochastic matrix and the additional details can be found within the general comments.

Reviewer comment (Presentation):
The paper is not organized very well: the author mentioned several literature many times throughout the whole paper, which feels very tedious; when reading section 3, I was confused since I don’t know the reason of introducing and proving those lemmas until I read section 4, also the notations in section 3 do not closely correspond to the notations used in the rest of the paper.

Author response:
Following the reviewer's comments, we have removed repeated literature citations throughout the manuscript. Moreover, new discussions have been added in Section 3 in order to inform that the results derived in Section 3 will serve as foundation tools for the results of Section 4. Furthermore, the notations in Section~3 have been modified to align more closely to the rest of the paper.

Reviewer comment (Experiments):
The simulation results are too few. In the paper they proved the convergence results for the constant and diminishing step size, but in the simulation section, both figures are for constant step size. The reviewer expected to see the diminishing step size case in the main body of the paper.

Author response:
In line with the reviewer's comments, we have newly added simulation results with the diminishing step-sizes in Figures (1b) and (2c) of the revised manuscript. Further details on the additional experiments can be found in the ``General Comments'' at the beginning of this document.

Reviewer comment (Novelty):
The analysis doesn't feel very original. The main difference from the cited existing papers, from the reviewer's understanding, is a linear mapping and a multiplication of LL^+. Also, the results rely on the relationship between the smallest and largest eigenvalue of the graph Laplacian, and it is unclear how many network graphs can meet those requirements.

Author response:
For the first comment, the main difference with other distributed TD-learning algorithms is that our algorithm is based on a primal-dual method, and it works in more relaxed and general scenarios, i.e., it does not require that the underlying communication network graph is characterized by a doubly stochastic matrix. We would like to note that the analysis of primal-dual method with null space constraints and Markovian observation model is not a trivial task, and it requires much more intricate analysis procedures compared to the existing analysis. Moreover, we believe that the analysis in this paper provides additional insights on utilizing the properties of Moore-pseudo inverse. 

For the second comment, the condition on the network graph structure is minimal in the context of the related literature (all the papers mentioned in Table 1) that it needs to be connected, undirected and without any loops. Moreover, the minimal condition on the proposed step-size to guarantee convergence only depends on the number of agents, $N$, rather than maximum and minimum eigenvalues of the graph Laplacian as a conservative choice. Therefore, many practical problems can meet these conditions. Further detailed discussion is given in the response to question 5.

Reviewer comment (Writing):
In the abstract, the authors mentioned their algorithm and method do not require the network structure characterized by a doubly stochastic matrix. But throughout the whole paper, there is no introduction to the doubly stochastic matrix and how it is related to communication networks. This is presented as an important contribution, but it is still unclear what kind of networks correspond to a doubly stochastic matrix and what kind of networks do not. Therefore, the significance of the contribution is unclear.

Author response:
A doubly stochastic matrix corresponding to the communication network can be constructed by assigning positive weights to each edge in the network. Additionally, every column sum and row sum of the doubly stochastic matrix needs to be one. Further details about doubly stochastic matrix can be found in page 8 and Appendix A.2 in the revised manuscript. The limitations of the existing distributed TD-learning algorithms requiring the doubly stochastic matrix mentioned in the General Comment, can be summarized as follows: 1) Extension to directed graph and time-varying graph scenario is challenging ; 2) The performance of the algorithm is sensitive to the choice of doubly stochastic matrix. The related discussions have been newly added in the revised version on page 8. For more detailed discussions on the doubly stochastic matrix, please refer to our General Comment at the beginning of this document.

Reviewer comment (Theory):
The paper only considers the average reward scenario. However, there can be another reward scenario (cooperative or competitive), can the result be extended to those setups?

Author response:
We believe that our analysis can be extended to other setups as long as the reward scenario can be achieved at the consensus of parameters of each agents. For instance, one can consider different scenarios (cooperative or competitive), where communications among the agents are allowed. In these scenarios, we can consider new distributed algorithms which apply the proposed strategies, which can be potential future topics.

Reviewer comment (Novelty):
There is already quite a bit of work on the multi-agent RL framework for the average-reward case. Please see [A1]. The authors should discuss both in terms of methodology and the results whether they are related or different. The above paper provides the sample complexity bound, and even consider general function approximation case.

Author response:
Thank you for pointing out the work, and we have added discussion about [A1] in the revised manuscript. To our best knowledge, the critic part (which corresponds to the TD-learning algorithm) in [A1], uses linear function approximation rather than general function approximation. The algorithm in [A1] uses mini-batch update, where the size of mini-batch depends on number of agent $N$. We believe that our algorithm can be also improved using such mini-batch style updates. Moreover, we can also derive the sample complexity $\mathcal{O}(1/\epsilon)$ or $\mathcal{O}(\log (1/\epsilon)/\epsilon)$ from the mean squared error bound in Theorem 4.2 and Theorem 4.3, respectively. Note that the sample complexity in [A1] is also derived from the mean squared error bound.

Reviewer comment (Novelty):
The exponential stability of primal-dual ODE dynamics is known in the literature when the constraint matrix is rank-deficient. The improvement is only some constant for a special case of objective and constraint functions, which might be not very important to the TD analysis.

Author response:
For the first comment, to the authors' knowledge, the exponential stability of primal-dual ODE dynamics when the constraint matrix is rank-deficient has not been fully studied so far. Moreover, as pointed out by the reivewer, while the improvement may be restricted to special scenarios, it offers different insight on utilizing the properties of Moore-pseudo inverse. Furthermore, it provides simplified tool for the analysis of the suggested distributed TD-learning.

Reviewer comment (Theory):
The conducted primal-dual ODE based analysis can only guarantee mean-path performance, which might be not very useful in practice due to large variance.

Author response:
We would like to note that the primal-dual ODE based analysis conducted in this paper not only guarantees the mean-path performance but also is used for the convergence properties of its stochastic versions (distributed TD-learning), which are given in the second part of the paper. The mean-path part can be seen as a contribution of our paper but it can be seen as a preliminary step for its stochastic version (distributed TD-learning).

Moreover, the majority of the theoretical works on analysis of TD-learning [R1,R2] including most of the distributed TD-learning which are listed in the Table 1 in the manuscript, also deal with the mean-path setting first and investigate its stochastic versions.

Reviewer comment (Experiments):
The provided example is artificial, and there is no comparison with existing distributed TD algorithms.

Author response:
According to the reviewer's comment, we have newly added experiments on comparisons with the existing distributed TD algorithms in Appendix A.13 in the revised version. Further details about the experiments are given in the General Comment.

Reviewer comment (Theory):
Is a missing $\mathbf{P}^\pi$ in projected Bellman equation?

Author response:
Thank you for pointing out the error, which has been corrected in the revised version.

Reviewer comment (Theory):
Can the authors explain more how the new TD learning algorithm is built on the result of Wang and Elia (2011)? When does strongly convexity hold?

Author response:
The original work of Wang-Elia studied distributed optimization algorithms in continuous-time domain with a general objective function $f(x)$ and its gradient $\nabla f(x)$. We replace $\nabla f(x)$ with stochastic TD-error $\phi_k (r_k+\gamma \phi_k^{\prime\top}\theta_k-\phi_k^{\top}\theta_k) $, which is, however, not a (stochastic) gradient of any objective function.


For the second question, we do not require the strong convexity. We only require the matrix $A$ to be negative definite, i.e., $A+A^{\top}\prec 0$ where $A$ can be an asymmetric matrix. TD-learning is a stochastic approximation scheme finding a fixed point of the equation $Ax=b$ rather than using a gradient of a certain objective function.