Paper: Multi-Agent Interpolated Policy Gradients (ICLR-2024)

Reviewer comment (Novelty):
Despite the fact that the idea is novel, it is hard to fully evaluate the benefits of the contribution given some relevant work has seemingly been missed by the authors; specifically, these works seem highly relevant: [1] Kuba, Jakub Grudzien, et al. "Settling the variance of multi-agent policy gradients." Advances in Neural Information Processing Systems 34 (2021): 13458-13470. [2] Rashid, Tabish, et al. "Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning." Advances in neural information processing systems 33 (2020): 10199-10210. [3] Mguni, David Henry, et al. "MANSA: learning fast and slow in multi-agent systems." International Conference on Machine Learning. PMLR, 2023.

Author response:
1. **Related works**.

In the revised version of our paper, we have incorporated a discussion on references [a][b][c]. 

Reference [a] is particularly relevant to our work as it addresses variance, and we emphasize that there is no conflict between [a] and our method. It is possible to integrate an optimal baseline from [a] into our method by replacing the advantage estimator in Equation 10 with Equation 12 from [a]. However, this modification requires the estimation of joint Q-functions, and GAE is not compatible, potentially impacting the performance of PPO-based gradients.

Reference [b], a milestone work on value factorization, shares similar ideas when analyzing the Bellman operator of a factorized function class, and we have included it in the related works.

Reference [c] proposes a promising approach using switch control to dynamically switch from independent learning to centralized learning. We believe similar methods could be applied to dynamically adjust the weight parameter $\nu$ in our approach. However, achieving this requires further efforts in both theoretical and experimental aspects, as discussed in the related and future works section of our revised paper.

[a] Kuba, Jakub Grudzien, et al. "Settling the variance of multi-agent policy gradients." Advances in Neural Information Processing Systems 34 (2021): 13458-13470.

[b] Rashid, Tabish, et al. "Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning." Advances in neural information processing systems 33 (2020): 10199-10210.

[c] Mguni, David Henry, et al. "MANSA: learning fast and slow in multi-agent systems." International Conference on Machine Learning. PMLR, 2023.

Reviewer comment (Novelty):
Though [2] tackles value-based methods it is still worth discussing in the context of this work. Similarly, the method in [3] learns the best set of states to switch from using a decentralized critic to a centralized critic. While the specific algorithms in [3] are value-based, it is a plug & play method that seemingly captures gradient methods. Without including a discussion on [3], it is hard to know how much this approach could be useful, since the authors' approach has a fixed weighting parameter for all states whereas [3] can be viewed as having a weighting variable whose optimal value {0,1} is learned for each state.

Author response:
1. **Related works**.

In the revised version of our paper, we have incorporated a discussion on references [a][b][c]. 

Reference [a] is particularly relevant to our work as it addresses variance, and we emphasize that there is no conflict between [a] and our method. It is possible to integrate an optimal baseline from [a] into our method by replacing the advantage estimator in Equation 10 with Equation 12 from [a]. However, this modification requires the estimation of joint Q-functions, and GAE is not compatible, potentially impacting the performance of PPO-based gradients.

Reference [b], a milestone work on value factorization, shares similar ideas when analyzing the Bellman operator of a factorized function class, and we have included it in the related works.

Reference [c] proposes a promising approach using switch control to dynamically switch from independent learning to centralized learning. We believe similar methods could be applied to dynamically adjust the weight parameter $\nu$ in our approach. However, achieving this requires further efforts in both theoretical and experimental aspects, as discussed in the related and future works section of our revised paper.

[a] Kuba, Jakub Grudzien, et al. "Settling the variance of multi-agent policy gradients." Advances in Neural Information Processing Systems 34 (2021): 13458-13470.

[b] Rashid, Tabish, et al. "Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning." Advances in neural information processing systems 33 (2020): 10199-10210.

[c] Mguni, David Henry, et al. "MANSA: learning fast and slow in multi-agent systems." International Conference on Machine Learning. PMLR, 2023.

Reviewer comment (Theory):
The theoretical analysis, though insightful, did not allow me to fully grasp an improvement in the performance of the proposed method with regard to variance and bias. I was expecting to see analytic results indicating that, for a given variance, the method achieves a reduced level of bias, and similarly, for a given level of bias, the method achieves a lower variance compared to the standard objective. While this has been shown in the ablation study, I did not see the corresponding analytic statements.

Author response:
2. **Analytical Comparison of Bias and Variance**.

Regarding the question on achieving reduced bias and variance, our method intentionally introduces bias to reduce variance compared to the standard unbiased objective. 

In Equation 10, the first term, requiring a sample from the joint action space, contributes to high variance, while the second term does not. Consequently, in contrast to the standard objective ($\nu=0$), our method reduces the variance of the high variance term by a factor of $1-\nu$, simultaneously introducing a low variance term with a coefficient of $\nu$. For a more comprehensive understanding, refer to Equation 13, where the variance of the high variance term is explicitly reduced by $(1-\nu)^2$.

It's important to note that, as the standard objective is unbiased, our method does not achieve reduced bias compared to it. The term "low bias" in our context reflects that the additional bias introduced by the factorized function is comparatively modest. This acknowledgment is particularly pertinent considering that a factorized function has the potential to introduce significant bias.

Reviewer comment (Novelty):
What are the benefits/weaknesses as compared to [3] Mguni, David Henry, et al. "MANSA: learning fast and slow in multi-agent systems." International Conference on Machine Learning. PMLR, 2023?

Author response:
1.  **Related works.**

Reference [c] addresses a different problem than the one tackled in our paper. The primary contribution of MANSA lies in establishing a plug-and-play learning framework aimed at reducing the cost of centralized learning (CL) by dynamically switching between CL and independent learning (IL) while still maintaining key convergence properties.
It's important to note that our focus is solely on CL. While [c] mentions issues related to the complexity of CL or structural limitations on value factorization, it opts for a switch to IL rather than addressing these challenges directly. Specifically, certain challenges arise when MANSA utilizes CL, particularly in the case of SMAC, where the percentage of CL calls in MANSA is notably high (around 70%).

In contrast, our paper directly tackles the intricacies under CL, associated with joint value functions and the biases resulting from value factorization, providing a comprehensive analysis of the bias-variance trade-off problem. Consequently, these two methods serve distinct purposes. While dynamically adjusting $\nu$ like MANSA presents a promising avenue for future work, the existence of MANSA does not diminish the significance of our contribution.

Reviewer comment (Theory):
For a given variance, can it be shown analytically that the method achieves a reduced level of bias and similarly, for a given level of bias, does the method achieve a lower variance as compared to the standard objective?

Author response:
2. **Analytical Comparison of Bias and Variance**.

Regarding the question on achieving reduced bias and variance, our method intentionally introduces bias to reduce variance compared to the standard unbiased objective. 

In Equation 10, the first term, requiring a sample from the joint action space, contributes to high variance, while the second term does not. Consequently, in contrast to the standard objective ($\nu=0$), our method reduces the variance of the high variance term by a factor of $1-\nu$, simultaneously introducing a low variance term with a coefficient of $\nu$. For a more comprehensive understanding, refer to Equation 13, where the variance of the high variance term is explicitly reduced by $(1-\nu)^2$.

It's important to note that, as the standard objective is unbiased, our method does not achieve reduced bias compared to it. The term "low bias" in our context reflects that the additional bias introduced by the factorized function is comparatively modest. This acknowledgment is particularly pertinent considering that a factorized function has the potential to introduce significant bias.

Reviewer comment (Novelty):
Although this paper is for the multi-agent case and also employs a different approach in the implementation (on-policy instead of off-policy), the idea and the bounds are too similar to that of the single-agent paper referenced in section 5. In this review's opinion, this makes the result incremental and not novel and significant enough for consideration at this venue.

Author response:
We would like to clarify certain aspects where there might be a misunderstanding regarding the novelty and significance of our work.

Firstly, our approach significantly deviates from single-agent papers referenced in Section 5. The integration of a joint Q-function and factorized function in our work is motivated by a distinct trade-off between expressive ability and learning difficulty. As detailed in Appendix A.1, our exploration of a convex combination highlighted its unsuitability for value-based methods, leading us to focus on policy-based methods. 
Although the resulted policy gradients share some similarity with IPG, our motivation is fundamentally different from any single-agent paper referenced.

Secondly, our theoretical results, particularly Proposition 4, provide distinctive insights into bounding the objective function. The analysis of bias induced by a factorized Q function, discussed in Section 4.2, represents a novel contribution absent in single-agent settings. 
Additionally, in Section 4.3, we introduced the concept of compatible function approximation in multi-agent reinforcement learning, exploring its relationship with value factorization.

Thirdly, the “different approach in the implementation (on-policy instead of off-policy)” is not an arbitrary choice. It aligns seamlessly with our theoretical findings, as used in Equation 14 and later on, and serves to prevent poor performance, as detailed in Appendix F. This approach is a deliberate and well-justified decision based on our theoretical framework.

In summary, while there may be similarities in addressing the bias-variance trade-off, our work represents a distinct and valuable contribution to the field. 
The strong relationship between the idea and bounds to value factorizations in the multi-agent settings sets our paper apart.
Our implementation choices are grounded in the theory and have been carefully considered to yield consistent and meaningful experimental results. We believe these aspects collectively establish the novelty and significance of our proposed method.

Reviewer comment (Writing):
A number of inconsistencies with the notations that make it hard for the average reader to precisely follow the claims of the paper. For instance, the notations $\nabla J$ and $\nabla \hat{J}$ are used interchangeably whereas they are not the same (look at and compare equations (1), (10), and (12)). Also, $\hat{Q}$ is used in equation (12) despite it not being properly defined until equation (14). Plus, sometimes the notations $\hat{Q}$ and $Q^\mu$ are used interchangeably even though they are clearly different when $Q^\mu$ is not in the function class $\mathcal{Q}$.

Author response:
We appreciate the reviewer's observation regarding inconsistent notations, and we would like to provide further clarification. In our paper, $J$ denotes the standard unbiased objective for policy gradient, while $\hat{J}$ represents the objective of our proposed method. Additionally, $\hat{Q}$ is introduced in the third line of the second paragraph of section 3.2, serving as an estimation of $Q^\mu$. To enhance consistency, we have updated the notations in the paper. Specifically, in Equation 12, we replaced $J$ with $\hat{J}$, and in Equation 10, $Q^\mu$ has been replaced by $\hat{Q}$.

Reviewer comment (Theory):
There seems to be a mistake in the proof of Proposition 4. The upper bounds $\frac{1}{2} L \sigma^2$ and $L \sqrt{mn}$ are supposed to be on the absolute value of the difference of $Q^\mu (s,a)$ and its first order estimate (initialed at $\mu(s)$); however, they are apparently used as upper bounds on $(Q^\mu (s,a) - \overline{Q}(s,a))^2$. So it seems the result should have been something like $c_1 L^2 \sigma^4 + L^2 mn \sigma^2$ instead of $c_1 L \sigma^2 + L \sqrt{mn} \sigma^2$ which would change the final bound as well. On another note, the absolute value should be outside the expectation for the second term in the last inequality of equation (27), and that is what makes the last argument of the proof (Proposition 4) possible.

Author response:
Certainly, we acknowledge the oversight in Proposition 4. In response, we have revised both the proposition and its proof in the paper. Notably, we have refined the definition of $\sigma$ for conciseness.

Reviewer comment (Experiments):
The empirical results for the proposed method are not convincing. Although in the GRF domain the proposed method gives better performance as compared to baselines, the same is not true for the SMAC domain. If results on another benchmark could be provided, it would make the empirical results stronger.

Author response:
While our method outperforms other baselines, it is essential to address the QMIX's comparable performance in several SMAC maps. Notably, QMIX is an off-policy method, incurring higher computational intensity and operating at about one-third the speed of our method. We have incorporated an additional plot illustrating compute time on the x-axis for a comprehensive comparison, as shown in Figure 8 of the revision of our paper.

Reviewer comment (Experiments):
In Algorithm 1, you mention that you use recurrent neural networks in policy networks $\pi_\theta$, state-value networks $V^\varphi$, and action-value networks $Q^\psi$. Given that in your setting each agent observes the whole state, I don't see why you need it. Also, how would the performance be affected by its absence?

Author response:
As indicated in the fourth line of the preliminary section, our approach is designed for Dec-POMDP, and the choice of full observation is made to simplify theoretical analysis. Given that SMAC is partially observable, the integration of RNNs proves beneficial for enhancing performance.

Reviewer comment (Experiments):
Effect of bias on the performance of the algorithm: Does a low value of $\nu$ give a better performance in most of the scenarios? I know that the effect of $\nu$ is investigated in ablation studies, but since the number of experiments there is small, I was wondering if the behavior seen in ablation studies holds in general or not.

Author response:
In our experiments, we observed that a $\mu$ value of 0.3 in SMAC and 0.5 in GRF yielded optimal performance, which can hardly be considered "low". However, in tasks with unnormalized reward, the choice of $\mu$ depends on the reward scale. Notably, PPO employs normalized advantage, resulting in $Q^\mu$ values in our methods that may be one order of magnitude larger than $A^\pi$, necessitating a smaller $\mu$. For instance, in MPE, we set $\mu=0.05$.

Reviewer comment (Experiments):
For the GRF domain, QMIX was not included in the baselines. Is there a reason for this?

Author response:
Addressing the absence of QMIX in GRF, as explained in the third paragraph of section 6.1, QMIX was not included due to its reported inferior performance in the MAPPO's paper.