From: ICLR.cc/2023/Conference/Paper3728/Authors
title: We are not convinced.
comment: We have fully discussed that theoretically BQL is as efficient as joint Q-learning, which is the most efficient known method to guarantee the convergence and optimality in stochastic environments. **The trivial algorithm you proposed is very much more inefficient than BQL. The trivial algorithm requires that all joint policies (the function of all $s$) will be visited, but BQL only requires that given any $(s,a_i)$, any $P_i(s, a_i)$ can be found in one of previous stages.** They are totally different.

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper considers the setting of cooperative reinforcement learning, and proposed an algorithm with a new best possible operator, which is claimed to be "the first decentralized algorithm in MARL that is guaranteed to converge to the joint optimal policy in the stochastic environments". This paper also proposed practical variant of the algorithm using "the simplified operator", and achieve improvements over several baselines.

While the main idea of the paper is clearly conveyed, several executions of the idea can be potentially improved. From the empirical perspective, the baselines of this paper are lacking. The baseline H-IQL is proposed in 2007. There are many SOTA algorithms proposed after H-IQL (e.g. several papers by Parmer et al.) are not compared. The "simplified version" (the practical version) of best possible operator uses the samples in the reply buffer. The correctness of the algorithm relies on the samples in the reply buffer exhaust all possible combination of policies by other players which seems very inefficient.

From theoretical perspective, the claim of "the first decentralized algorithm in MARL ..." requires several modifications: currently a trivial decentralized algorithm can also achieve the claimed property---at each stage m, each agent randomly sample a deterministic policy, and then play this policy to collect multiple trajectories. At the end, each agent reports the policy from the stage that has the highest average reward. Algorithm 1 (BQL) proposed in this paper also requires this stagewise checkpoint synchronization. It's not hard to show this trivial algorithm enjoys the claimed guarantees. 
justification_for_why_not_higher_score: There several major issues as highlighted in the summary, which are not well addressed in the main paper.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Update
comment: You mentioned that the chance of picking one $\tilde{P}_i$ to be luckily equivalent to the transition with maximal $Q_i$ is close to the chance of choosing the optimal actions for all agents. It is true but it does not mean our method is inefficient. Even in joint learning, which has the lower bound of sample efficiency of decentralized MARL, the optimal actions for all agents have to be chosen enough times. Decentralized MARL is inherently hard, which might be the reason why there are only a few previous methods in the past twenty years. We hope that you can take this into consideration. Looking forward to receiving your agreement.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Thanks for your response!
comment: The main concern is maintaining one replay buffer in neural network implementation, which is an approximation technique considering both effectiveness and fairness. We will answer the two questions in detail.

1. We use a large replay buffer in BQL, which is a common technique in offline RL and can avoid cases where there are very few samples in the buffer. In fact, BQL uses the same replay buffer as IQL. On the other hand, Q-learning will not perform well with a very small buffer in single-agent settings. **For the effectiveness of estimation accuracy of $P_i$**,  we have run experiments in Figure 1(b). The larger size of $|D_i^m|$ means the more accurate estimation of $P_i$.  If $|D_i^m|$ is too small, *i.e.*, 200, the experiences in $|D_i^m|$ are insufficient to accurately estimate the expected value (7). If $|D_i^m|$ is too large, *i.e.*, 10000, the experiences in $|D_i^m|$ are redundant, and the buffer series is difficult to cover all possible transition probabilities given total training timesteps.

2. It can be guaranteed that $|D_i^m|$ sufficiently goes through all possible transition probabilities if total timesteps are sufficiently large, because the agents are continuously updating policies, and we can add manual disturbance of policies as the technique in Section 2.4. We have analyzed the problem of mixing policies in Appendix B. In neural network implementation, the operator does not go through only $P_i$ of deterministic $\pi_{-i}$, but have to go through $P_i$ of stochastic $\pi_{-i}$ (mixing). So the optimality can also be guaranteed but the sample complexity will be high. In Figure 1(a), we show the results of BQL- (with only one buffer). BQL- can also significantly outperform IQL but learns slower than BQL. **Even if the operator cannot go through all possible $P_i$, from the proofs we know that the learned value will continuously be close to the best possible value if the iteration is continuously performed.** For the sample complexity of large-scale agents, we have shown that the ideal sample complexity of joint learning is $O(MS\Pi^N_{i=1} A_i )$, which is the lower bound of decentralized MARL and also increases exponentially with the agent numbers. In this perspective, the high complexity is because **the decentralized MARL itself is hard**. 

The most important thing is that maintaining one replay buffer does not belong to our theoretical analysis, but is a derived practical technique from theoretical analysis to adapt to complex tasks. Although the sample complexity will is high in theory, in implementation we have tried many kinds of replay buffers and found using the same buffer with IQL is the most efficient and the fairest in complex tasks. The gap between theory and implementation is common, and **it is sensible to adopt the efficient and fair implementation** with theoretical optimality, and to evaluate the gap through experiments. We have done a lot of experiments to evaluate the effectiveness and efficiency.

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_NZDY
title: Response to Rebuttal
comment: Sorry for the late reply and thanks for the rebuttal.

"We only maintain one replay buffer, which corresponds to one $\tilde{P}_i$". 

It's less clear how the empirical estimation of $P_i$ based on the replay buffer affects the algorithm. Considering a case where there are very few samples in the buffer and the $\tilde{P}_i$ is far from the true $P_i$, Eq. (7) is estimating the $Q^e$ based on some random transition probabilities, how would this affect the performance of the algorithm? This might be an extreme case example but the intention here is to say that the algorithm performance depends on the estimation accuracy of $P_i$, which is not clearly explained in the paper.


"Relying on the non-stationarity, ideally, the simplified best possible operator can be performed on all possible $\tilde{P}_i$". 

The proof of lemma 4 relies on that the $P_i$ achieving the maximal $Q_i$ value to be selected occasionally from the replay buffer. How to guarantee it happening and what's the sample efficiency for this? The replay buffer is a mix of samples from previous policies, an average on those samples may not even represent any possible policy of other agents $\pi_{-i}$. Say we have a policy $\pi_1$ and policy $\pi_2$, samples collected as a mix of the two policies may not fall within the function class of any of the two policies, especially when constraining the policies to be deterministic only. Also, in terms of efficiency, when the number of agents is large, the chance of picking one $\tilde{P}_i$ to be luckily equivalent to the transition with maximal $Q_i$ is close to the chance of choosing the optimal actions for all agents. Why is the algorithm efficient in this perspective?

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Update
comment: We fine-tuned the hyper-parameter $\lambda$ on 3_vs_1_with_keeper. Smaller $\lambda$ means stronger strength of BQL. The original result of BQL $50.4\pm5.9$ is with $\lambda = 0.5$. When we set  $\lambda = 0.1$, BQL can achieve $65.3\pm7.4$, which significantly outperforms baselines, showing BQL is really effective.

To be honest, we really made an effort in the rebuttal period. If you still think that our paper cannot reach the bar of ICLR, we wish to know the major concerns.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Results of Google Research Football
comment: Dear reviewers,

We test BQL on Google Research Football task academy_3_vs_1_with_keeper. The winning rates are shown in Table 1. Our BQL can achieve stronger and more stable (low standard deviation) performance than other baselines. BQL and baselines are built on the codebase pymarl2.

Table 1. Winning rates of 3_vs_1_with_keeper (%).

| BQL ($\lambda = 0.1$) | BQL ($\lambda = 0.5$)  | MA2QL           | H-IQL           | IQL             |
| -------------  | ------------- | --------------- | --------------- | --------------- |
| $65.3\pm 7.4$ | $50.4\pm 5.9$ | $43.8 \pm 16.2$ | $40.3 \pm 29.4$ | $32.9 \pm 11.6$ |

Are you satisfied with our responses and additional results during the rebuttal period? If so, we hope you could consider raising the score to support us. Thanks for your efforts！



Bests,

Authors

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_oR8C
title: Thanks!
comment: Thanks for the update! I will further discuss some issues in the responses with the other reviewers.

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_o76z
title: Thank you for your response
comment: I thank the authors for their response.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Looking forward to further discussion
comment: Dear Reviewer oR8C,

Thanks for your valuable comments to help the improvement of our paper. We believe that we have addressed all your concerns with corresponding responses and results. Are you satisfied with the responses? Do you have any more questions?

Best regards,

Authors

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Looking forward to your feedback
comment: Dear Reviewer NZDY,

We hope it does not disturb you. We are eager to know whether we have addressed your concerns or not? If you have any further questions, we are happy to answer them.



Thanks again for your time!



Best regards,

Authors

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Additional results
comment: We add BQL+IQL and IQL on the three mujoco tasks, the mean results over the last 5 points of the learning curves are shown in Table 1. BQL+IQL can obtain performance compared with IQL. It is unfair to directly compare BQL+IQL with MA2QL since there are two different schemes. Following the variable-controlling approach, we should compare BQL+MA2QL with MA2QL, and BQL+IQL with IQL, do you think so?

Tabel 1: Results of MoJoCo

|             | BQL+MA2QL       | MA2QL         | BQL+IQL       | IQL          |
| ----------- | --------------- | ------------- | ------------- | ------------ |
| Halfcheetah | $2784 \pm 1049$ | $2059\pm 606$ | $1051\pm 241$ | $512\pm 73$  |
| Hopper      | $997 \pm 30$    | $901 \pm 138$ | $356\pm 158$  | $103\pm 38$  |
| Walker      | $494\pm 158$    | $429\pm 221$  | $456\pm 173$  | $469\pm 202$ |

We run BQL+MA2QL on 5m_vs_6m and corridor, the mean results over the last 20 points of the learning curves are shown in Table 2. Since the results of decentralized methods are very close to those of some CTDE methods, which can be seen as upper bound of decentralized methods, we cannot draw a convincing conclusion from the two tasks.

Tabel 2: Results of SMAC (%)

|          | BQL+MA2QL        | BQL+IQL          | MA2QL             |
| -------- | ---------------- | ---------------- | ----------------- |
| 5m_vs_6m | $46.9\pm6.5$ | $47.6\pm7.1$ | $43.7\pm11.0$ |
| corridor | $84.5\pm5.9$ | $83.1\pm5.7$ | $83.1\pm8.9$  |


Do you have any more suggestions? We are impressed by your professional comments and look forward for more feedback!

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Response
comment: Thanks for your feedback! We will clarify your confusion.

> Is there any 'criterion' when to use BQL or BQL+MA2QL?

Both IQL and MA2QL can be base algorithms of BQL. In our paper, to demonstrate the difference between our operator and the traditional bellman operator, we have to build BQL on IQL for a fair comparison. In practice, it is natural that we should build BQL on the stronger base algorithm. 

> Q1. In Figure 12, does BQL (not BQL+MA2QL) perform worse than MA2QL in MA-Mujoco?

Yes, BQL+IQL performs worse than MA2QL in tasks of Figure 12, because MA2QL shows a much stronger performance than base algorithm IQL, as shown in MA2QL paper. Due to the time limitation, we will add the results of BQL+IQL in revision. When built on IQL, there is an inherent disadvantage in BQL. So it is not the fault of BQL, it is the fault of the base algorithm. We have to stress again that our contribution is the best possible operator. Thus, to verify our contribution, the most important experiments are the comparisons with IQL.

> Q2. How does BQL+MA2QL perform in Figure 13?

In Figure 13, the converged results of 5m_vs_6m and corridor are very close to some CTDE methods. Due to the time limitation, we will run BQL+MA2QL on 5m_vs_6m and corridor in revision.

> Q3. Regarding question 1 "Initialization of Q value," is negative/conservative initialization helpful for learning neural networks?

First, conservative initialization is really inconvenient for a neural network. Second, conservative initialization is unfair. For a fair comparison, we should use the same initialization as IQL. Third, the influence of the initialization of neural networks is complicated, which is studied by many typical methods, e.g., Xavier and He. We believe using typical initialization methods is a better choice.

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_oR8C
title: Questions
comment: Thanks for the reply. I have additional questions regarding the responses.

Q1. In Figure 12, does BQL (not BQL+MA2QL) perform worse than MA2QL in MA-Mujoco? The result of BQL is not included in Figure 12. Please report the result.

Q2. How does BQL+MA2QL perform in Figure 13? 

Q3. Regarding question 1 "Initialization of Q value," is negative/conservative initialization helpful for learning neural networks?

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Follow-up
comment: We add some new results of SMAC: effectiveness of $\lambda$ in 2c_vs_64zg (Figure 9(c)) and performance in 10-agent MMM2 (Figure 16). If you still have some confusion, please let us know. We are expecting more feedback!

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Response to Comments (Part 1)
comment: > MA2QL

First,  we have to claim that BQL can be naturally and theoretically applied on MA2QL, which means **MA2QL is not an opponent, but a base algorithm of BQL**. 

1. The comparison with MA2QL is fair. We use the default hyper-parameters provided by MA2QL. The total update number for each agent in MA2QL is set to be the same as that in IQL. MA2QL switches the stage every K steps and updates one agent in each stage N*K times. In MuJoCo,  K=8000 steps. In SMAC, K = 50 episodes.
2. MA2QL is built on the codebase pymarl2 (with Adam, 16 processings), while BQL is built on the codebase pymarl (with RMSProp, single processing). For fair comparison, we build MA2QL on pymarl. So the results in MA2QL paper are not comparable. We test 5m_vs_6m and corridor in Figure 13, BQL shows similar performance compared with MA2QL, and the converged results are very close to some CTDE methods, *e.g.*, QMIX, indicating that non-stationarity is not a severe problem in these two tasks.
3. In Multi-Agent MuJoCo, we tested both the fully observable setting (Figure 3) and partially observable setting (Figure 10). Note, in MA2QL, agent_obsk=1 in HalfCheetah and agent_obsk=2 in Hopper are still partially observable setting since the observation still cannot cover all state information. BQL can be naturally and theoretically applied on MA2QL, which means MA2QL is not an opponent, but a base algorithm of BQL. We test BQL+MA2QL and MA2QL in Figure 12 to show the effectiveness of BQL. Our implementation of MA2QL is stronger than the version in MA2QL paper, and time is limited, so we run $5 \times 10^5$ steps instead of $3 \times 10^6$ steps.

Finally, we stress again our **theoretical contribution** compared against MA2QL. MA2QL only guarantees the convergence to a Nash equilibrium, but the converged equilibrium may not be the optimal one when there are multiple equilibria, but BQL can guarantee the convergence to optimal joint policy. Moreover, the convergence of MA2QL holds only in on-policy training, but BQL can be performed in non-stationary replay rebuffer.

> H-IQL

In H-IQL, we set $\lambda = 0.1$ in differential games, $\lambda = 0.9$ in Multi-Agent MuJoCo and SMAC, which are carefully fine-tuned. If setting $\lambda$ of  H-IQL to be the same as $\lambda$ of BQL, the performance of H-IQL is extremely poor.

> Sampling from the replay buffer

1. In fact, we have tried the method you proposed that partitions one buffer into several parts in chronological order. However, we found that using the whole buffer without partitioning can achieve better performance in practice, that is the reason why we do not use partitioning. Moreover, using the whole buffer without partitioning is a fair comparison with other baselines.

2. Relying on the non-stationarity in $D_i$ and random noise in the update of neural networks (or manual disturbance of policies), $D_i$ can sufficiently go through all possible transition probabilities if total timesteps are sufficient. Experimental results show that BQL can achieve performance gain even if $D_i$ does not go through all possible transition probabilities. Going through all possible transition probabilities is just a theoretical analysis, but not necessary for experiments.

3. There is a misunderstanding. (6) and (7) have the same sample efficiency,  which is similar to that of joint Q-learning. (7) is more efficient than (6) in terms of computation cost. (6) has to compute the expected values of all possible $P_i$ in each update, but (7) only computes the expected value of any one $P_i$ in each update. More importantly, (7) is convenient for neural network implementation, because we cannot maintain buffer series in advance in complex tasks.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Response to Comments (Part 2)
comment: > multiple optimal policies

Good advice! We run experiments on a one-stage matrix game with multiple optimal joint policies. As shown in Figure 14, BQL can learn coordinated policies,  which verifies the effectiveness of the randomly initialized reward. Moreover, coordination is a challenging problem, which is not solved in many popular MARL methods. For example, as analyzed in [1], value decomposition methods cannot solve the tasks with multiple optimal policies. 

We are running SMAC tasks with large agent numbers. However, coordination is not a severe problem in SMAC. Within only $5 \times 10^5$ steps, BQL and MA2QL can achieve high performance in bane_vs_bane.

| BQL                 | MA2QL               |
| ------------------- | ------------------- |
| $85.2 \% \pm 3.7\%$ | $83.9 \% \pm 5.2\%$ |

We are running 27m_vs_30m, which consumes really a long time (14 days for $10^7$ steps). Like existing popular MARL methods, we mainly focus on convergence and optimality, and propose an easy solution to deal with the cases with severe coordination problems.

> reproducibility

1. Since we are carrying on successive work using the code of BQL, we **promise the public** that we will release the code immediately after the paper is accepted.
2. We use the current version of pymarl with sc2.yaml. However, the version of pymarl in [2] (Samvelyan et al. (2019)) uses the old sc2_beta.yaml. We do not modify IQL code, but the authors of pymarl updated the hyper-parameters.
3. In all methods except MA2QL, each agent samples the same episodes. Synchronized sampling is easy to implement if agents share the same random seed, which is affordable and widely adopted.



[1] Fu, Wei, et al. "Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning." International Conference on Machine Learning. PMLR, 2022.

[2] Samvelyan, Mikayel, et al. "The StarCraft Multi-Agent Challenge." Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. 2019.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Response to Questions
comment: > 1. Initialization of Q value 

Initializing the Q value to be $r_{min}$ is a common technique for theoretical analysis. In practice, we do not need this initialization, just adopting a randomly initialized neural network, like IQL.

> 2. Are the proofs available for stochastic policies? Why can we only consider deterministic policies?

All the lemmas and theorems in our paper are available for stochastic $\pi_{-i}$, since we do not use the restriction of deterministic policies in the proofs. We consider deterministic policies to reduce the complexity of (6). Because deterministic policies are finite, but stochastic policies are numerous. When there is only one optimal joint policy, the optimal joint policy must be deterministic [1]. That is the reason why we can only consider deterministic policies to reduce the complexity.

> 3. How to select a subset of states?

As we have stated, each agent randomly and independently selects a subset of states, according to the uniform distribution.

> 4. Can we directly apply the max operation in (8)?

We cannot directly apply the max operation in (8) in neural network implementation, because the positive random noise of neural network update will be continuously accumulated which may cause value overestimation. As shown in Figure 2. $\lambda = 0$ means the max operation and shows very poor performance due to overestimation.

> 5. The positive random noise of Q

As analyzed in Double DQN [2], noise is a source of overestimation. Specifically, we sample experiences to estimate gradients for backpropagation, and the estimated gradients would have a noisy bias from the true gradients. So the Q values would have noisy errors caused by the estimated gradients. Some values have positive noises, some values have negative noises. In the computation of the target value of IQL (the expectation), the positive noises can be offset by the negative noises. However, in the computation of the target value of BQL (the max), the positive noises will be continuously accumulated, causing severe overestimation. Setting $\lambda < 1$ is to offset the positive noises to avoid overestimation.

> 6. SAC

Actually, ISAC does not theoretically match BQL, but we have to empirically test the effectiveness of BQL on ISAC, otherwise, we will be asked to do it. When applying our operator to ISAC, we only need to modify the update of Q networks. We maintain four value functions: $Q_i^{e1}, Q_i^{e2}, Q_i^{1}, Q_i^{2}$ and target networks $\bar{Q}_i^{e1}, \bar{Q}_i^{e2}$. $Q_i^{ej}$ is updated by minimizing:

$(Q_i^{ej}(s,a_i) - y_i)^2, \quad y_i = r + \gamma(\min_{j=1,2}Q^{j}_i(s',\pi_i)-\alpha \log \pi_i)$

$Q_i^{j}$ is updated by minimizing:

$w(s,a_i)(Q^j_i(s, a_i)-\bar{Q}_i^{ej}(s, a_i))^2$

The policy is updated by maximizing:

$\min_{j=1,2}Q^j_i(s,\pi)-\alpha \log \pi_i$

We use the SpinningUP implementation, the SOTA implementation of SAC, where each ISAC agent has two Q functions. 

> 7. The update of the target network.

The update of the target network uses the default settings of the codebase of IQL implementation. In differential games and Multi-Agent MuJoCo, we softly update the target network with $\tau = 0.995$ (SpinningUP). In SMAC, we update the target network every 200 episodes (PyMARL).

> 8. The region with $r>0.3$ 

As shown in Figure 15, the region with $r>0.3$ is very narrow.

> 9. Is there any explanation regarding the increasing $\lambda$ as the environment becomes more complicated?

If the environment is more complicated, overestimation is more likely to happen. So $\lambda$ should be increased. 



[1] Puterman, Martin L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.

[2] Van Hasselt, Hado, Arthur Guez, and David Silver. "Deep reinforcement learning with double q-learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 30. No. 1. 2016.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Response to Reviewer o76z
comment: We are very happy to see your thorough understanding of our paper! As you pointed out, partial observation and coordination are two important aspects of MARL, we will explain the two questions in detail.

>  Partially observable settings

In section 4.3, we have shown that the optimality is undecidable in partially observable Markov decision processes (POMDPs) [1]. So it is not the fault of BQL that the optimality does not hold in partially observable MDPs. However, the convergence of BQL still holds, because even on the transition probabilities of observation $P_i(o'|o,a_i)$, $Q_i$ under (7) and (8) is still monotonically increasing and bounded above. According to the monotone convergence theorem, BQL can converge. Multi-Agent MoJoCo is a partially observable environment, where each agent can only observe the state of joints and bodies controlled by itself. The limited observation range makes Multi-Agent MoJoCo very challenging, and BQL can obtain performance gain.

> Coordination

Coordination is important and a big topic, but we focus on convergence and optimality in this paper, so we provide an easy but practical solution to address this problem. We also add experiments on a one-stage matrix game with multiple optimal joint policies. As shown in Figure 14, BQL can learn coordinated policies,  which verifies the effectiveness of the randomly initialized reward. We will move the discussion about coordination to the main pages in the final version with an additional page.



[1] Madani, Omid, Steve Hanks, and Anne Condon. "On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems." *AAAI/IAAI*. 1999.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Response to Reviewer hhze
comment: Thanks for your support of our novelty! In the Appendix, we provide an easy trick to address multiple optimal policies, and we will move this part and important results to main pages in final version with an additional page, and zoom in the figures of learning curves. We also add experiments on a one-stage matrix game with multiple optimal joint policies in Figure 14.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Response to Reviewer NZDY
comment: > For the simplified best possible operator, it seems $\tilde{P}_i$ is sampled from all possible transition probabilities, how could this effectively update $Q_i$? In practice, it means to sample from all possible policies of the other agents.

In our implementation, the update of $Q_i$ is effective and efficient. As shown in Algorithm 2, we do not sample from all possible policies of other agents. We only maintain one replay buffer, which corresponds to one $\tilde{P}_i$, and perform the simplified best possible operator on this replay buffer. In this replay buffer, $\tilde{P}_i$ is non-stationary. Relying on the non-stationarity, ideally, the simplified best possible operator can be performed on all possible $\tilde{P}_i$, and thus the operator can converge to the optimum.

That is to say, in practice, the replay buffer and exploration strategy of BQL are the same as those of IQL, the only difference is the loss functions (9) (10). So BQL is efficient and practical, which is verified by experiments.

> Why we can only consider deterministic policies.

We consider deterministic policies to reduce the complexity of (6). Because deterministic policies are finite, but stochastic policies are numerous. When there is only one optimal joint policy, the optimal joint policy must be deterministic [1]. That is the reason why we can only consider deterministic policies to reduce the complexity.

> Similar to the proof of Lemma 2, we can easily prove $\max_{a_{-i}} Q(s, a_i, a_{-i}) \geq Q_i^{k}(s, a_i)$

$Q_i^{k}(s, a_i)$ in (6) takes the max expectation over all possible $P_i$, while $Q_i^{k}(s, a_i)$ in (7) and (8) takes the expectation of any one of possible $P_i$. So $Q_i^{k}(s, a_i)$ in (6) $\geq Q_i^{k}(s, a_i)$ in (7) and (8). In Lemma 2 we prove that $\max_{a_{-i}} Q(s, a_i, a_{-i}) \geq Q_i^{k}(s, a_i)$ in (6). So we easily have $\max_{a_{-i}} Q(s, a_i, a_{-i}) \geq Q_i^{k}(s, a_i)$ in (7) and (8).

> The conditions and assumptions for all lemmas and theorems are not clearly specified.

Our theoretical results are very general, without requiring specific conditions and restrictions. The problem settings and assumptions have been stated in Preliminaries.

> Two baselines

The two baselines [2,3] do not follow our fully decentralized settings. In the first one [2], agents can communicate with their neighbors through a sparsely connected network (in its Preliminaries). We have clarified the difference between our fully decentralized settings and networked settings in related work. The second one [3] is a CTDE method, where the information of other agents can be obtained in the training. Thus comparison with the two baselines is unfair.

BQL is the first method to theoretically guarantee the convergence to optimal policies in fully decentralized MARL. The novelty and theoretical analysis are agreed by other reviewers. We argue our work is novel and theoretically supported. As fully decentralized learning is rarely studied in MARL, it is normal to meet some misunderstandings. If you think we clarify your misunderstandings, we sincerely hope you could take this into consideration and re-evaluate contribution and novelty of our work. We really need your support!



[1] Puterman, Martin L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.

[2] Li, Hepeng, and Haibo He. "Multi-agent trust region policy optimization." arXiv preprint arXiv:2010.07916 (2020).

[3] Kuba, Jakub Grudzien, et al. "Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning." International Conference on Learning Representations. 2021.

From: ICLR.cc/2023/Conference/Paper3728/Authors
title: Thanks for your suggestions! 
comment: We thank all the reviewers for the efforts on reviewing our paper and the insightful suggestions! BQL is the first method to guarantee the convergence to optimal joint policy in stochastic fully decentralized multi-agent tasks, so we are confident that we really make a contribution to this fundamental challenge of MARL. We hope reviewers and area chairs could take this into consideration. All of the questions have been carefully clarified in detail, and the results of multiple optimal policies and more tasks have been updated in revision (Figure 12, Figure 13, and Figure 14). We look forward to further discussions!

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_NZDY
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes the best possible operator for decentralized learning in cooperative multi-agent games. Since the best possible operator is almost impossible to implement in practice, the authors also propose a simplified best possible operator. Experiments are conducted on simple stochastic game, stage game, MPE, MuJoCo and SMAC.

strength_and_weaknesses: Strength:

The experiments are evaluated on multiple types of tasks. The fully decentralized setting for cooperative MARL is appealing.

Weaknesses:

The abstract is a bit misleading at the beginning. It does not clearly specify the scope of the research, only in the last sentence it tells that the experiments are conducted on cooperative multi-agent tasks.

Sec. 2.2,”To reduces the complexity, we can only consider the deterministic policies …” is not well justified.

Sec. 2.3 “Similar to the proof of Lemma 2, we can easily prove…” is less clear.

For the simplified best possible operator, it seems $\tilde{P}_i$ is sampled from all possible transition probabilities, how could this effectively update $Q_i$? In practice it means to sample from all possible policies of the other agents. 

The conditions and assumptions for all lemmas and theorems are not clearly specified. 

Several baseline methods are missed, like:

1. Li, Hepeng, and Haibo He. "Multi-agent trust region policy optimization." arXiv preprint arXiv:2010.07916 (2020).

2. Kuba, Jakub Grudzien, et al. "Trust region policy optimisation in multi-agent reinforcement learning." arXiv preprint arXiv:2109.11251 (2021).

The numerical results should also be provided in tables apart from just the learning curves.

There are lots of typos in the paper to be fixed.

clarity,_quality,_novelty_and_reproducibility: The theoretical support of the proposed method is not clearly provided. 

The quality of writing is bad and the results are not complete.

The idea of marginalizing over other agents in the joint Q-value is not novel enough.

summary_of_the_review: The paper generally needs a major revision for improving its quality in terms of writing, technical contents, method description and result reports. More baseline methods should be compared. The theoretical justification needs to be provided with more details.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_hhze
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes what they call "the best possible operator" for decentralized Q-learning when there are multiple agents. The operator updates the policies of each agent individually at each step and can be shown to converge to the optimal joint policy. However, the operator is too complex and impractical. What the paper really proposes is the "simplified best possible operator" which computes only one of the possible transition probabilities but still can be proved to converge to the best optimal policy. The algorithm is then called "best possible Q-learning (BQL).  The algorithm can be implemented with a Q-table for each agent por with a neural network for each agent. The paper evaluates BQL on a variety of tasks (stochastic games, etc) and compares the performance with the performance of different algorithms.
strength_and_weaknesses: Strengths: There is novelty in the algorithms proposed and the guarantee to converge to the joint optimal policy is important. The optimality of the joint policy found is proved formally. 
Weaknesses: Finding the optimal policy is guaranteed only when there is only one optimal policy.  This is reported in the Appendix, not in the paper. Many results are shown in the Appendix. There are many acronyms, but not all of them are defined.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the work is original to the best of my knowledge. The graphs of the experimental results are too small and hard to read.  
summary_of_the_review: A well-written paper that presents an algorithm, BQL, for multi-agent reinforcement learning that guarantees the learning process will converge to the optimal joint policy when the optimal policy is unique.  BQL is computationally more efficient than algorithms that explore all possible transition probabilities.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: None
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_o76z
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes _best possible Q-learning_, a multiagent reinforcement learning approach that - being decentralized - ensures convergence to optimality. The paper assumes full state observability, i.e., all agents have access to the full state of the environment. To ensure convergence to the optimal joint policy, the paper introduces the _best possible operator_, which at its core is similar to a Bellman operator for each agent $n$ in a multiagent MDP that considers that the other agents follow a greedy policy, i.e., agent $n$'s $Q$-function, $Q_n$, is updated as

$$Q_n(s,a_n)\leftarrow\max_{a_{-n}}\mathbb{E}[r+\gamma\max_{a'}Q_n(s',a')\mid s'\sim P(\cdot\mid s,a_n,a_{-n})],$$

with $a'\in A_n$. The paper actually uses a more evolved formulation for $\mathcal{T}$ using transition probabilities, but unless if I missed something, it seems to me that it is equivalent to the one above. The paper then proposes a _simplified best possible operator_, where the maximization over $a_{-n}$ is replaced by a (fixed) arbitrary choice of $a_{-n}$, leading to the computation of an intermediate Q-value, dubbed $Q^e_n$, such that

$$Q^e_n(s,a_n)=\mathbb{E}[r+\gamma\max_{a'}Q_n(s',a')\mid s'\sim P(\cdot\mid s,a_n,a_{-n})],$$

leading to the update

$$Q_n(s,a_n)\leftarrow\max(Q_n(s,a_n),Q^e_n(s,a_n)).$$

The above operator is then turned into a Q-learning algorithm, by allowing the different agents to interact in episodes where their policy is essentially held fixed. The experiences collected during different episodes provide an estimate for $Q^e_n$ above, and $Q_n$ is then updated accordingly.
strength_and_weaknesses: **Strengths**

The paper addresses an interesting problem and provides an interesting approach, which is reminiscent of the distributed IQL of Lauer and Riedmiller mentioned in the paper. 

**Weaknesses**

The fact that the paper assumes full state observability is, in my view, an important setback, given current research in MARL mostly considers that agents have only access to local observations. Granted, the paper does evaluate its approach in a partially observable setting, but (i) the theoretical grounding of the algorithm does not hold for partially observable settings (at least not directly); (ii) from the portrayed experiments it is difficult (at least for me) to assess how robust the algorithm is to general partially observable settings. 

Another important aspect that is not discussed significantly in the paper is related with _coordination_, since in scenarios where multiple policies exist it does not seem to me that the agents necessarily converge to a coordinated policy. The paper refers to Appendix D for a solution to the case where multiple optimal policies may exist (which I'm convinced is the most common scenario), but I believe that this is an important issue that should have its space in the main body of the paper. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is well organized and clearly written.

**Quality**

I did not check the derivations in detail, but the results portrayed strike me as reasonable.

**Originality**

I believe the proposed approach -- although reminiscent of previous approaches in the literature -- is novel.

**Reproducibility**

I am not completely sure that the provides sufficient information to reproduce the experiments (namely, details of the architectures used), but I believe that it is easily solvable by providing access to a code repository in the final version of the paper.
summary_of_the_review: The problem addressed in the paper is relevant (although in a somewhat constricted setting) and the contribution novel and interesting. There are some aspects of the contribution that could be discussed in greater detail. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3728/Reviewer_oR8C
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper designs a new decentralized learning algorithm in cooperative multi-agent reinforcement learning where no communication or parameter sharing among agents is allowed. From each agent’s point of view, the transition probability is non-stationary due to the other agents’ changing policies. The proposed best possible operator considers the maximum target over the available transition probabilities and is claimed to produce the optimal policy assuming the optimal joint policy is unique. Next, a simplified best-possible operator is suggested by randomly selecting one possible transition probability followed by a monotonic update and is claimed to produce the optimal joint policy under the same assumption. The authors practically applied the operator to action-value function learning with neural networks, and numerical results showed promising results in some cooperative multi-agent environments.
strength_and_weaknesses: Strength: This paper is overall well-written and easy to read. The authors tried to show theoretical support for the proposed algorithm under a specific assumption.

Weaknesses: The authors lack experiments for fair comparisons with the considered baselines. In addition, there is a gap between the proposed simplified operator and its implementation regarding sampling data. Reproducibility could not be well verified. Investigation into the multiple optimal policies was not considered. Please see the following comments and questions.


<Comments>

C1. Regarding a fair comparison with MA2QL

- The MA2QL paper states that “for a fair comparison, the total number of Q-function updates for each agent in MA2QL is set to be the same with that in IQL.” Is this rule explicitly applied to the conducted experiments? Please clarify the (fine-tuned) training steps for updating an agent at each turn in every environment.
- In Fig.4, there are three easy SMAC environments and one hard environment (2c_vs_64zg). In MA2QL paper, three other environments are considered(3s_vs_4z (easy), 5m_vs_6m (hard) and
corridor (super hard)). Among the three environments, MA2QL shows superiority over IQL in the latter two environments. For a fair comparison with MA2QL, SMAC experiments in those three environments should be performed.
- In Multi-agent Mujoco, BQL only considered two-agent cases in the partially observable setting, which seems an empirical limitation of this paper. On the other hand, MA2QL conducted experiments on 2-agent HalfCheetah, 3-agent Hopper, and partially observable 3-agent Walker2d in multi-agent MuJoCo. For a fair comparison with MA2QL,  Multi-agent Mujoco experiments in those three environments are required.

C2. Regarding a fair comparison with H-IQL

- As far as I understand, the H-IQL method represents the equation in Appendix A. If yes, then the authors should report the (fine-tuned) value of $\lambda (<1)$.

C3. Regarding sampling from the replay buffer

- In my opinion, the (hidden) key part of BQL is how to sample data as well as the formulation itself in Eqs. 9 and 10. There is a gap between Eq. 7 and Eq. 9 (with one replay buffer) regarding sampling data. The authors explicitly need any method to reduce the gap. For example, I think the minibatch should contain samples from “similar” transitions. Partitioning one buffer into several parts in chronological order, and choosing one part of them for sampling can apply to Eq. 7. (Note that this partitioning is different from using buffer series in Appendix B since the former uses one buffer, but just changes the sampling method inside the same buffer.)
- Can we guarantee that $\mathcal{D}_i$ sufficiently goes through all possible transition probabilities? (page 6)
- On page 5 and Appendix B, the authors claim the sample efficiency of Eq. 7 over Eq. 6. While Eq. 7 is practical since we ideally require samples from every $\pi_{-i}$ to compute Eq. 6. However, other than that, the explanation in the second paragraph on page 13 is not clearly understood, especially the complexity of Eq.6. In my opinion, this additional explanation requires clarification.


C4. Regarding multiple optimal policies & scalability

- We can extend the matrix game on page 8 to observe what happens to BQL when there are multiple optimal policies. We can also check whether the proposed approach in Appendix D is practically applicable in this case. 
- Another promising experiment can be a scalability test on the SMAC environment (e.g., 10m vs 11m, 27m vs 30m, bane_bane) since there may be a relatively higher possibility of multiple optimal policies when the number of (homogeneous) agents increases. This SMAC test can also strengthen the scalability of BQL. (I checked Appendix C.2.)

C5. Regarding reproducibility 

- I could not check the reproducibility of the experiments as the authors did not provide their implementation codes.
- Why are the performance of IQL in 2c_vs_64zg and 1c3s5z much higher than those in *Samvelyan et al. (2019)? In the paper, the win rates are 7% and 21%, respectively, while in this paper, about 50% and 70%, respectively. Did the authors tune the IQL code? * Samvelyan et al., The StarCraft Multi-Agent Challenge, 2019.
- For clarity, do the BQL, H-IQL, and IQL use synchronized sampling (i.e., each agent samples the same episodes) or not? 


<Questions>

1. On page 3, does “$Q_i^0$ is initialized to be the minimal return” mean  $Q_i^0=-r_{min}$? Does this initialization is applied in Algorithms 1 and 2? (I could not find any explicit mention regarding this.)
2. Are the proofs of Lemma 2, 3, and 4 available for stochastic $\pi_{-i}$? For example, the first inequality in Lemma 2 may be valid for $P_i^*$ under stochastic $\pi_{-i}$. If yes, is that a reason for considering only deterministic policies? 
3. On page 5, can the authors clarify how to select “a subset of state $S_i^m$”?
4. Instead of Eq.10, can we directly apply the max operation as seen in Eq. 8?
5. Could you explain “the positive random noise of $Q_i$”? (page 6) The necessity of $\lambda$ in Eq. 10 is empirically appealing, but less conceptually.
6. Regarding independent SAC in Appendix C.3
- How can BQL be applied to SAC as (i) SAC contains entropy term and (ii) the considered policy is stochastic, unlike deterministic in DDPG? This part should be clarified with the explicit formulation.
- Does independent SAC use two Q functions for each agent? Two $\bar{Q}_i^e$ functions for each agent? 
- What is the base code for implementing independent SAC?

7. What is the target update period or coefficient in Line 8 of Algorithm 2?
8. On page 8, it seems that “the narrow region $r>3$ is surrounded by the region with $r=0$” is written in reverse order. 
9. Is there any explanation regarding the fine-tuned value of $\lambda$ increasing as the environment becomes more complicated?

Minor: $max_{P_i(s'|s,a)} \rightarrow max_{P_i(\cdot|s,a)}$ in Eq.6.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to read. Arguing the theoretical background for the proposed method can support the novelty of the paper. However, more details including additional experiments should be significantly expanded, and detailed explanations should be provided for clarity and to reduce the logical gap. Finally, since the authors did not provide their source code, reproducibility verification was impossible.



summary_of_the_review: The target of this paper is fully decentralized multi-agent reinforcement learning (MARL), an emerging area in the MARL community. However, the numerical study should be expanded and the proposed algorithm should be fairly compared with the baselines, guaranteeing reproducibility verification. In addition, the authors need to clarify unclear technical explanations to support the main claim. 


correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough