Paper: Improving Sample Efficiency of Model-Free Algorithms for Zero-Sum Markov Games (ICLR-2024)

Reviewer comment (Theory):
The description of technical innovations is not so clear. In the end of page 5, I cannot understand how V^ref helps. The authors shall elaborate why this overcomes the 1/H factor. (BTW, is the V bar in the last line V bar_h instead?)

Author response:
Q: The description of technical innovations is not so clear. 

A: Thanks for the question. We elaborate the novelty in the algorithm design and theoretical analysis in the general response, and we also incorporate it in our revised version. 

Q: In the end of page 5, I cannot understand how $V^{ref}$ helps. The authors shall elaborate why this overcomes the $1/H$ factor. 

A: Generally speaking, an accurate estimate $V^{ref}$ provides a better estimate of $V-V^{ref}$ in comparison to directly estimating $V$. First of all, we show that learning $\beta$-optimal ($\beta=O(1/H)$) reference value functions $V^{ref}$ requires $O(SABH^5/\beta^2)$ samples, which is independent of the number $K$ of episodes (Corollary 4.5). Then, with an access to such a $\beta$-optimal reference value functions $V^{ref}$, applying concentration inequality to the term $V-V^{ref}$ results in a better sample complexity as their difference is upper bounded by $\beta$, compared with applying the concentration directly to $V$ whose value is upper bounded by $H$. The above two facts altogether offset the weakness of using only $O(H)$ fraction of data. 

Q: BTW, is the $\bar{V}$ in the last line $\bar{V}_h$ instead?

A: We thank the reviewer for the careful review. The $\bar{V}$ in the last line is $\bar{V}_h$. We fixed the notation in our revision.

Reviewer comment (Evaluation):
The result still suffers from the curse of multiagents (ie AB instead of A+B). The authors discussed it in the end of Sec 4.1. While understanding it's technically hard, it is still a pity.

Author response:
Q: The result still suffers from the curse of multiagents (i.e. $AB$ instead of $A+B$). The authors discussed it in the end of Sec 4.1. While understanding it's technically hard, it is still a pity.

A: Thanks for the comment. We agree that it is very important to investigate the "curse of multi-agency", for example, in our context, to understand whether the reference-advantage decomposition technique can be used to further improve the sample complexity by designing a V-learning type algorithm (which has been used in the literature to deal with multi-agency). Below, we discuss our initial thoughts and potential technical challenges we expect along this direction. A main step is to first incorporate reference-advantage decomposition into the update of optimistic value functions, i.e., we replace the standard update rule (for both optimistic and pessimistic value functions) $V_h(s)=r(s)+P_h V_{h+1}(s)+bonus$ by the following reference advantage-based update rule $V_h(s)=r_h(s)+P_hV_{h+1}^{ref}+P_h(V_{h+1}-V_{h+1}^{ref})+bonus$. Moreover, reference value functions will be updated ($\beta$-approximate $V^{ref}$ are learned) once sufficient samples are collected (similar to lines 14-15 in our algorithm). Further, the success of V-learning relies on the adversarial bandit subroutine where the loss function is $l=(\beta+1-r-V+V^{ref})/(\beta+1)$. Main challenge here is to ensure that the term handled by such adversarial subroutine benefit from the reference-advantage decomposition. Another potential challenge is to incorporate variance reduction technique into adversarial bandit subroutine. We leave this interesting problem as our future work.

Reviewer comment (Writing):
The reviewer is giving a 5 because of being unable to understand the main technical contribution, and would be happy to reevaluate after the authors' feedback.

Author response:
We appreciate the effort and time of the reviewer. The main concern is that the main technical contribution is unclear to the reviewer. In our general response, we recap our novelty in algorithm design and theoretical analysis, and we have revised our paper to emphasize the technical contribution. We sincerely request the reviewer to re-evaluate our work based on the revision, and we are open to further discussions.

Reviewer comment (Novelty):
Since the single-agent work [1] is very related, it would be better if a short introduction about it is given.

Author response:
Thanks for the suggestion! In our revision, we briefly introduce reference-advantage decomposition in single-agent RL [1] to help the reader digest the technical contributions of our work.

[1] Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.

Reviewer comment (Writing):
Some intuitive explanations may be needed for readers to catch the general idea before section 3, which discusses the details of the algorithms directly. It would be even better if there were examples.

Author response:
Thanks for the comment. We are aware that the complicated algorithm (Algorithm 1) may be hard to follow and may not be the efficient way to convey the general idea. In our revision, We presented a schematic algorithm and defer the details of the algorithms to the appendix. Besides, in the revision, we give a short introduction to the reference-advantage decomposition in single-agent RL [1], elaborate the difficulty and our contribution of extending such a technique to zero-sum Markov game before introducing our main algorithm.

[1] Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.

Reviewer comment (Writing):
The main weakness of the paper is clarity. In my opinion, an accurate revision is essential to significantly enhance the quality of the paper, and high level ideas could be communicated more efficiently.

Author response:
We appreciate the effort and time of the reviewer! The major concern that affected the reviewer's evaluation is the clarity issue. We posted the revised paper to improve the clarity of the paper and sincerely hope that the reviewer can re-evaluate our work based on the lastest version. We are open to any more discussions.

Q: The main weakness of the paper is clarity. In my opinion, an accurate revision is essential to significantly enhance the quality of the paper, and high level ideas could be communicated more efficiently. In particular the paper assumes that the reader is familiar with [1] and [2], and without a deep understanding of these works the reader is confronted with high level ideas that are hard to grasp. This leaves the reader unsure on the technical contributions of the paper.

A: We thank the reviewer for bringing the clarity issue up! In our general response, we elaborate the technical novelties in algorithm design and analysis. We acknowledge that in the current version, the readers may have difficulty understanding the high level ideas and technical contributions of our work without knowing relevant works [1] and [3]. In our revision, we introduce in greater detail the reference-advantage decomposition in single-agent RL [1], and the difficulties when extending it to two-player zero-sum Markov games. 

[1] Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.

[2] Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Proceedings of Thirty Third Conference on Learning Theory, 2020.

[3] Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. In Advances in Neural Information Processing Systems, 2020.

Reviewer comment (Writing):
The paper assumes that the reader is familiar with [40] and [36], and without a deep understanding of these works the reader is confronted with high level ideas that are hard to grasp.

Author response:
We appreciate the effort and time of the reviewer! The major concern that affected the reviewer's evaluation is the clarity issue. We posted the revised paper to improve the clarity of the paper and sincerely hope that the reviewer can re-evaluate our work based on the lastest version. We are open to any more discussions.

Q: The main weakness of the paper is clarity. In my opinion, an accurate revision is essential to significantly enhance the quality of the paper, and high level ideas could be communicated more efficiently. In particular the paper assumes that the reader is familiar with [1] and [2], and without a deep understanding of these works the reader is confronted with high level ideas that are hard to grasp. This leaves the reader unsure on the technical contributions of the paper.

A: We thank the reviewer for bringing the clarity issue up! In our general response, we elaborate the technical novelties in algorithm design and analysis. We acknowledge that in the current version, the readers may have difficulty understanding the high level ideas and technical contributions of our work without knowing relevant works [1] and [3]. In our revision, we introduce in greater detail the reference-advantage decomposition in single-agent RL [1], and the difficulties when extending it to two-player zero-sum Markov games. 

[1] Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.

[2] Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Proceedings of Thirty Third Conference on Learning Theory, 2020.

[3] Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. In Advances in Neural Information Processing Systems, 2020.

Reviewer comment (Writing):
What does "min-gap" exactly refers to? It is never defined formally. Does it refers to $\Delta(s,h)$ which is the minimum gap over all episodes and this ensures monotonicity?

Author response:
Q: What does "min-gap" exactly refers to? It is never defined formally. Does it refers to $\Delta(s,h)$ which is the minimum gap over all episodes and this ensures monotonicity?

A: Your understanding is correct. In Section 3, we presented the min-gap based reference-advantage decomposition in detail and we elaborated the "min-gap" at the end of page 4. We further clarify this point in the revision.

Reviewer comment (Writing):
You say that using the techniques of [40] for single agent RL is challenging due to the fact the upper bounds are no longer monotonic. It is unclear way this is needed. More precisely, the last two paragraphs on page 4 are really hard to read and do not convey clearly neither the challenges nor the contributions.

Author response:
Q: You say that using the techniques of [40] for single agent RL is challenging due to the fact the upper bounds are no longer monotonic. It is unclear why this is needed. More precisely, the last two paragraphs on page 4 are really hard to read and do not convey clearly neither the challenges nor the contributions.

A: Thanks for the helpful comment! We would like to give a detailed explanation in our revision, and we briefly discuss it below.

In single agent RL, the V function preserves the monotonic structure if the corresponding Q function is non-increasing, which is crucial for the reference-advantage decomposition. In two-player zero-sum game, one naive thought is to obtain the non-increasing optimistic V function and non-decreasing pessimistic V function. However, this is no longer possible due to the policy updating by the CCE oracle. Compared to single agent RL where the policy is updated by selecting an greedy action, the policy updated by the CCE oracle does not preserve such a property. In other words, the optimistic and pessimistic V functions are not monotonic even if the corresponding optimistic and pessimistic Q functions are monotonic. To resolve the issue, we observe that if the gap between the pair of optimistic and pessimistic V value functions are monotonic, the reference-advantage decomposition still works. To this end, we propose min-gap based approach to keep track of the smallest gap between the pair of optimistic and pessimistic V functions in the history and record the corresponding pair of V functions. Upon receiving enough samples (Algorithm 1 line 22), the pair of reference V function is then updated as the recorded pair of V functions.

Reviewer comment (Presentation):
Algorithm 1 gives too many unnecessary details, and simplifying it would help the reader understand better its workings. In particular, I think lines 9 to 13 are never commented and could be written with $O$ notation or something alike.

Author response:
Q: Algorithm 1 gives too many unnecessary details, and simplifying it would help the reader understand better its workings. In particular, I think lines 9 to 13 are never commented and could be written with $O$ notation or something alike.

A: Thanks for the great suggestion! We agree that a schematic algorithm would be easier to follow. In the revision, we present the following simplified algorithm in the main paper and defer the detailed version to the appendix. In the simplified algorithm, we keep the stage-based update backbone, and for each stage, we only outline the main steps including sample collection and accumulator update, optimistic and pessimistic value function update and policy update, tracking the pair of min-gap value functions, and reference value function selection.

Reviewer comment (Theory):
In algorithm 1 the actions played come from a correlated strategy (line 6)? Can the learning be decoupled by marginalising $\pi$ or the marginalisation has to happen only at the end of the learning procedure?

Author response:
Q: In algorithm 1 the actions played come from a correlated strategy (line 6)? Can the learning be decoupled by marginalising $\pi$ or the marginalisation has to happen only at the end of the learning procedure?

A: In Algorithm 1 line 6, the actions played come from a correlated strategy. The learning cannot be decoupled by marginalising $\pi$ (as elaborated in the following), and the marginalisation happens at the end of the learning procedure (Algorithm 2). 

Our algorithm falls into the popular centralized Q-learning algorithms for Markov game [3]. In Q-learning algorithm, we keep track of the Q functions (of state and joint actions), and the exploration policy $\pi$ computed by the Q functions (by CCE oracle) in each iteration is correlated in nature. However, given the exploration policies, we are able to extract the marginalized policy at the end of the learning process (certified policy in page 6).

[3] Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. In Advances in Neural Information Processing Systems, 2020.

Reviewer comment (Writing):
It is never explained what CCE has to do with the problem of computing Nash equilibrium. After some tome I arrived at the conclusion that it is the "collapse" idea first introduced in "Cai, Yang, et al. "Zero-sum polymatrix games: A generalization of minmax." Mathematics of Operations Research (2016)" in which the CCE of a $\epsilon$-perturbed zero sum game is an $\epsilon$-CCE of the zero sum games which marginalisation is in turn a $O(\epsilon)$ of the Nash of the zero sum game. I get that this idea was already presented in [36] but the authors should at least presented at a high level this idea.

Author response:
Q: It is never explained what CCE has to do with the problem of computing Nash equilibrium. After some time I arrived at the conclusion that it is the "collapse" idea first introduced in "Cai, Yang, et al. "Zero-sum polymatrix games: A generalization of minmax." Mathematics of Operations Research (2016)" in which the CCE of a $\epsilon$-perturbed zero sum game is an $\epsilon$-CCE of the zero sum games which marginalisation is in turn a $O(\epsilon)$ of the Nash of the zero sum game. I get that this idea was already presented in [2] but the authors should at least presented at a high level this idea.

A: Thanks for pointing out the clarity issue! The equivalence of CCE and NE in two-player zero-sum Markov games is also elaborated in [2] and [4]. We include the relationship between CCE and Nash equilibrium in zero-sum Markov games in our revision.

[2] Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Proceedings of Thirty Third Conference on Learning Theory, 2020.

[4] Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. In Proceedings of the 38th International Conference on Machine Learning, 2021.