Paper: Value Factorization for Asynchronous Multi-Agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
Some of the experimental results, although going in the right direction, are not sound enough to actually credit the proposed theoretical claims.

Author response:
3. Regarding the results, they are key to support the claims of our paper. 
    - The fact that the performance "ranking" of the methods follows the primitive-action case, shows that our formalization based on MacDec-POMDPs is sound and works well in practice. 
    - They demonstrate the importance of the temporal-consistency problem that has not been considered by macro-action literature before. In particular, introducing the joint macro-state is a simple yet powerful way to address such an issue. 
    - These results also serve to compare previous macro-action-based works, showing that (i) it is possible to design principled value factorization methods for asynchronous Multi-Agent Reinforcement Learning (MARL), (ii) these CTDE value-based algorithms perform better than the current asynchronous MARL.
    - Considering the superior performance over existing methods, our results set a new baseline for future CTDE macro-action-based works.

Reviewer comment (Writing):
The background section is a bit rushed. For example, you mention the Q-function in Section 2.1, but you are never actually defining it formally or even explaining what that is intuitively. Although probably every reader of this paper will already know this, the point of the Background section is exactly to provide a brief recap of such basic notions, and thus I think it is important to be as accurate and comprehensive as possible here.

Author response:
- We agree the background section could appear rushed as we omitted a few details to fit the page limit. We revised it, including a definition for Q, clarified the meaning of $\tau$, and further detailed the limit of QPLEX’s decomposition (discussed in the next point). We kindly ask the reviewer to point out any additional missing details in Section 2 that we will carefully incorporate.

Reviewer comment (Theory):
Why not requiring the values not to be non-positive in your proposed Adv-IGM is a more general condition? The non-positive nature of the advantage function is not a superimposed limitation, but rather an intrinsic property of this function, and as such I do not clearly see how this can be defined as a limitation. Also, it is not entirely clear to me how you can avoid the decomposition in Equation (4) with your proposed formulation. Please argument a bit more in this respect to make your claims clearer.

Author response:
- Regarding Adv-IGM, we agree the non-positive nature of the advantage function is an intrinsic property of it. However, considering the discussion in the dueling networks’ work [A], allowing the advantage function to assume arbitrary values typically leads to significantly higher performance. As such, Adv-IGM represents a more general condition as it will allow future factorization schemes to learn arbitrary advantage functions, inheriting the benefits of the single-agent dueling networks case. For example, the decomposition of Eq. 4 could be avoided by employing a principled version of dueling networks, where agents learn separate streams for state and advantage value functions (without any value restriction). This is an exciting direction to investigate in the future but goes beyond the scope of our paper.

Reviewer comment (Experiments):
Why not comparing also to Cen-MADDRQN? It would have been interesting to compare the obtained results not only against a decentralized solution (to assess the improvements of the proposed technique w.r.t. that), but also against a centralized solution, to compare how well is your decentralized technique doing against a (more expensive) centralized one, as well as assessing how such a decentralized technique can scale better when problems become more complex.

Author response:
- Regarding Cen-MADDRQN, we performed additional experiments (each over 20 independent runs) in the most complex domains: BP-30, WTD-T1, and CT-10 (Table 1 and Appendix E). In CT-10, Cen-MADDRQN showed poor performance due to the high number of agents employed in the task. Similarly, Cen-MADDRQN also has lower performance in WTD-T1 with respect to our best-performing solution, AVF-QPLEX. Conversely, BP-30 is characterized by a few agents and a joint observation of limited size, and the fully centralized baseline obtains higher performance than AVF methods.

Reviewer comment (Novelty):
The comparison results of different mixers seems a bit expected: these are indeed what one would expect from the standard literature in the primitive-action setting. Indeed, QMIX is a generalization of VDN (additivity is a sub-case of monotonicity), and QPLEX in a generalization of both (as monotonicity is a sufficient but not necessary condition for IGM). What is the added value of such a comparison here? What is the additional bit of knowledge that the reader can gain from your proposed results?

Author response:
- Regarding the results of different mixers, extending factorization methods to asynchronous MARL requires the asynchronous buffer and the use of temporal consistent joint information. The underlying mixing architectures of the primitive algorithms remain the same, and it makes sense the general results are the same in the primitive setting. The added value in our experiments is the ablation study conducted to support our claims on the importance of temporal consistency. Without using a joint macro-state, in fact, AVF methods fail at solving even the easiest macro-action tasks (BP-10 and WTD-S).

Reviewer comment (Experiments):
The comparison between using the true state vs. using the macro-state in AVF is interesting, but only reporting results on a single problem is not convincing enough to claim that the true state is not a good conditioning signal. Could you please add similar results on another benchmark to prove that your claim generalizes beyond the specific WTD-S setting?

Author response:
- Concerning the ablation study, we performed an additional set of experiments in BP-10. In particular, the new Figure 5 shows how AVF methods based on the true state fail at solving the easiest configuration of BP and WTS. The high number of independent runs we consider in all our experiments is also a strong indicator of the significance of these results. We further enhanced our ablation study by considering AVF-{QMIX, QPLEX} employing the only macro-state of the agent which terminated its macro-action (i.e., the second mechanism discussed in Section 3.2.). As expected, the noise introduced by such a method leads to significantly lower performance than the joint macro-state employed in Fig. 5, but represents a better-performing solution than using the true state.

Reviewer comment (Evaluation):
Results of the primitive-action algorithms and their macro-action counterparts on BP-10 do not seem to match: as you claimed in Section 3, your proposed methods should reduce to the primitive case when macro-actions have a single-step duration. So what is the difference here? Why are their performances not matching with those shown in Figure 4? If you are comparing these primitive-action algorithms on a setting with temporally-extended actions, that would be an unfair comparison. If not, however, I would expect their performance to be on-par with your AVF versions. Could you explain this a bit more in detail?

Author response:
- There has been a misunderstanding on the results of the primitive-action algorithms. We tested the primitive version of VDN, QMIX, and QPLEX in the primitive version of BP-10 presented in (Xiao et al., 2020a). These primitive methods only employ 1-step moving and pushing actions, as they cannot employ the temporally extended actions of their asynchronous counterpart. We added a description of the primitive BP in Appendix D.1. As clearly stated in Section 4, we remark such a comparison is not fair since primitive methods cannot leverage the expert knowledge of low-level policies used by macro-action-based methods. Intuitively, this is why their performance is not on par with the asynchronous versions.

Reviewer comment (Novelty):
The paper introduces the concept of IGM for macro-actions. However, it's worth noting that this introduction may seem less impactful, as existing value factorization algorithms can inherently fulfill this requirement as long as state and action representations are adjusted in accordance with prior research, which is precisely what the paper demonstrates.

Author response:
Regarding the impact of IGM on macro-actions, we believe the claim that *“existing value factorization algorithms can inherently fulfill this requirement as long as state and action representations are adjusted in accordance with prior research”* is severely misleading. In the following, we clarify our point of view, which also addresses the two reviewer's questions.
**Without Mac-IGM and MacAdv-IGM, existing value factorization algorithms with adjusted state and action representations would not satisfy the action selection consistency**, since *Mac-IGM and MacAdv-IGM must consider what happens at the centralized level and decentralized level, separately*. For example, while the latter follows from the definition of a macro-action (i.e., agent $i$) selects a new macro-action based on the termination condition $\beta_i$ of the low-level policy of the previously executed macro-action), the former requires the conditional operator discussed in Eq. 5. *Moreover, MacAdv-IGM relaxes the assumptions of Adv-IGM* on the non-positive advantage values, while still guaranteeing consistency in the action selection as shown in the proof provided in Appendix A. Finally, *the formalization of Mac-IGM and MacAdv-IGM also allowed us to draw conclusions about the relationships over the primitive case*. In particular, we demonstrated how they represent more general classes of functions with respect to their primitive counterparts. It is worth *remarking on the temporal consistency problem* we discussed in Section 3.2 as naive implementations of asynchronous value factorization methods considering the actual state of the environment introduce significant noise in the training process, leading to policies that fail at solving even the most simple tasks (Fig. 5).

Reviewer comment (Experiments):
The experimental scenarios employed in this study are simplistic.

Author response:
Regarding the experimental scenarios, the reviewer’s claim *“The experimental scenarios employed in this study are simplistic.”* also deems scarce attention to our experiments and limited knowledge of the asynchronous MARL literature. *We used most of the environments proposed in the latter, including a more complex version of the capture-target domain*. In addition, our response (as well as previous macro-action works such as Xiao et al., 2020a) shows that previous **synchronous, primitive approaches fail to solve even the most simple setup (BP-10). This is clear evidence of the complexity of these tasks that, we remark, represent the standard benchmark in the asynchronous MARL literature**. The complexity arises from a high degree of partial observability, combined with strong coordination requirements; characteristics that represent a serious challenge for existing MARL methods.

Reviewer comment (Theory):
The paper emphasizes that defining Mac-IGM and MacAdv-IGM is not trivial. But it seems that the primary distinction lies in considering only terminated macro-actions. Are there other complexities that merit attention?

Author response:
Regarding the impact of IGM on macro-actions, we believe the claim that *“existing value factorization algorithms can inherently fulfill this requirement as long as state and action representations are adjusted in accordance with prior research”* is severely misleading. In the following, we clarify our point of view, which also addresses the two reviewer's questions.
**Without Mac-IGM and MacAdv-IGM, existing value factorization algorithms with adjusted state and action representations would not satisfy the action selection consistency**, since *Mac-IGM and MacAdv-IGM must consider what happens at the centralized level and decentralized level, separately*. For example, while the latter follows from the definition of a macro-action (i.e., agent $i$) selects a new macro-action based on the termination condition $\beta_i$ of the low-level policy of the previously executed macro-action), the former requires the conditional operator discussed in Eq. 5. *Moreover, MacAdv-IGM relaxes the assumptions of Adv-IGM* on the non-positive advantage values, while still guaranteeing consistency in the action selection as shown in the proof provided in Appendix A. Finally, *the formalization of Mac-IGM and MacAdv-IGM also allowed us to draw conclusions about the relationships over the primitive case*. In particular, we demonstrated how they represent more general classes of functions with respect to their primitive counterparts. It is worth *remarking on the temporal consistency problem* we discussed in Section 3.2 as naive implementations of asynchronous value factorization methods considering the actual state of the environment introduce significant noise in the training process, leading to policies that fail at solving even the most simple tasks (Fig. 5).

Reviewer comment (Novelty):
It is apparent that extending existing value factorization methods to macro-action settings would inherently adhere to Mac-IGM. What is the significance of introducing Mac-IGM, and does it offer advantages in the development of algorithms, especially in cases where a direct extension might violate Mac-IGM? Further clarification on this aspect would enhance the paper's contribution.

Author response:
We appreciate your additional feedback. We hope that our efforts to clarify our point of view exhaustively, will not be misinterpreted as attempting to address potential loopholes in the raised doubts.

The short answer is that Mac-IGM (similar to the primitive IGM) does not provide any guidance in enhancing implementations as these principles only formalize an action selection consistency requirement. For example, IGM does not give insights on using a monotonic mixer as in QMIX to have consistent action selection (the same holds for Mac-IGM).
Conversely, Adv-IGM proposed in the QPLEX paper (Wang et al., 2016) suggests a fixed decomposition scheme for the design of Adv-IGM-based algorithms. In contrast, MacAdv-IGM relaxes such decomposition by allowing arbitrary learning of advantage and state (history) value functions. We discuss the benefits of MacAdv-IGM in our response to Reviewer eKQs. In addition, we drew the relationship between the class of functions representable by the primitive and macro-action-based formalization, which is also part of our first contribution regarding \{Mac/Mac-Adv\}-IGM.

The state and action modifications and their effects on the centralized buffer are the main changes that allow AVF-based methods to work (as discussed in the pseudocode in Appendix B). Despite the claimed lack of methodological novelty, value factorization in the asynchronous framework would not work without these (as confirmed by the extended ablation study). Our experiments, in fact, show that "direct extensions" (i.e., using the environments' state) satisfy Mac-IGM, but fail at learning good policies due to the noise introduced by a temporal inconsistent state (as shown using the Implicit Function Theorem in Section 3.2). Overall, the macro-action literature has not considered factorization so far, nor the asynchronous nature of extra information. We believe finding simple yet effective solutions for addressing these issues is a benefit of our work and not a limitation.

In this direction, we want also to point out two important things:

- Methodological modifications on state/actions space have led to widely considered state-of-the-art multi-agent reinforcement learning (MARL) algorithms (for instance, MAPPO [D] is simply a decentralized PPO algorithm with a centralized critic). In this vein, while our practical modifications could be perceived as relatively simple, the empirical impact of our work is supported by a wide variety of experiments run over a high number of independent runs. This confirms their statistical significance [E], the correctness and impact of the proposed AVF-methods.

- We also emphasize that our work aims to lay the groundwork for future research on asynchronous value factorization, and is not proposing a novel factorization methodology. In fact, regarding Eq. 5, we explicitly claim in Section 3.2 that "AVF-based algorithms are trained end-to-end to minimize Eq. 14, which resembles the fully centralized case of Eq. 5". The paper's contribution, in this case, lies in the fact that Eq. 14 uses history-state-based macro-action value functions due to the use of a mixing network, which gives significant performance advantages over Eq. 5. This aspect has been clarified in the previous paper revision, where we included results for the fully centralized method that use Eq. 5.

Finally, regarding the lack of new benchmarks, we did show a 10-agent version of CT as existing macro-action-based methods already solved the original CT-2. Nonetheless, setting new benchmark environments was not the scope of our paper. We believe the field of asynchronous MARL should first focus on developing principled methods to solve existing well-known tasks as there is still work to do before achieving optimal performance in existing domains.

Overall, we think all the updates to our manuscript have significantly clarified and improved our contribution, as also recognized by Reviewer JQAT. Even though our visions on (asynchronous) MARL literature can be different, we really appreciate your feedback as we are sure it will help us to improve our future works. We remain available to answer any other doubt that might arise before the discussion period ends.

[D] C. Yu et al., "The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games," in NeurIPS 2022.

[E] P. Henderson et al., "Deep Reinforcement Learning that Matters," in AAAI, 2018.

Reviewer comment (Novelty):
It appears that Mac-IGM and MacAdv-IGM are natural extensions of IGM, and the following propositions hold in a straightforward manner.

Author response:
Regarding Mac-IGM and MacAdv-IGM, they *are novel formalizations that ensure the consistency between local and global macro-action selection*. This **is not straightforward since it takes into account the temporally extended nature of macro-actions that require considering the local and global cases separately**. During centralized action selection, the joint value has to condition on ongoing macro-actions $m^-$ to avoid biases in the joint estimation, while decentralized action selection requires consistent macro-action selection for only $|M^i|$ agents (i.e., agents whose macro-action is over). Moreover, Mac-AdvIGM differs from the original (primitive) formalization since it does not enforce the decomposition from Q-values as in (J. Wang et al., 2021). We believe this aspect has been significantly overlooked by the MARL literature, and has the potential to allow learning the value of a state (or history) separately. Such a decomposition has been shown to significantly improve performance in single-agent setups based on dueling networks [A]. However, this goes beyond the scope of our original submission, and we intend to explore this in future work. In addition, **we further extend the Mac-IGM and MacAdv-IGM contribution by drawing the relationships between the classes of functions that can be represented by the primitive and asynchronous IGM principles**. Such a result is formalized in Proposition 3.5, where we *show that the macro-action principles are more general versions of the primitive ones as they can represent value functions on both primitive and temporally extended action spaces*. It follows that AVF implementations of existing primitive baselines represent a broader class of functions over the primitive counterpart.

Reviewer comment (Experiments):
[About Weakness 2] Have you experimented with other methods that utilize temporally consistent states? If so, could you provide the results and explain why these approaches have an advantage over others?

Author response:
Regarding temporal consistency, the proposed method is a simple yet effective mechanism that has been successful in achieving good performance and stable learning. We discussed three different representations to deal with the temporal consistency problem and provided an ablation study showing the crucial role of the proposed mechanism (which has been significantly extended in the paper revision. Please, refer to the new Fig. 5 for further details.). Given the novelty represented by asynchronous MARL in the literature, **we are not aware of any other method that raises the problem of utilizing temporally consistent states in the mixer, nor other solutions for tackling such a novel problem**. Hence, *we kindly ask the reviewer to provide any reference that will allow us to perform additional experiments with other methods that utilize temporally consistent states*.

Reviewer comment (Experiments):
Could you please provide more results of Mac-IAICC on other environments that have already been considered? It would be greatly appreciated if you could include learning curves.

Author response:
Regarding **Mac-IAICC, we performed an additional set of experiments (20 independent runs) on CT-10**, confirming the challenges posed by these environments for the policy-gradient baseline. To further strengthen our claims on the superior performance of AVF methods, **we also added a comparison with the fully centralized Cen-MADDRQN** (Xiao et al., 2020a) in the most complex setups (i.e., BP-30, WTD-T1, CT-10). Please, refer to the revised manuscript (Appendix E and the revised Table 1) and our answer to Reviewer E8n9 for further details about these results. Finally, as requested by the reviewer, we included the learning curves for both Mac-IAICC and Cen-MADDRQN in the supplementary material.

Reviewer comment (Experiments):
For fairness, authors compare AVF algorithms with primitive value factorization methods in the primitive action version of BP-10, but they didn’t provide any explanations about what changes have been made in the primitive action version of BP-10 from the original version.

Author response:
Regarding the primitive baselines, **we used the primitive version of BP-10 presented in the original Dec-MADDRQN paper** (Xiao et al., 2020a). In summary, agents can only perform 1-step movement and push actions. We included an exhaustive discussion of the primitive version in Appendix D.1.

Reviewer comment (Experiments):
Regarding the use of extra information in the mixer, the authors list three different mechanisms, but the second mechanism that using the macro-state of the agent whose macro-action has terminated is missing in the experiment.

Author response:
Concerning the extra information in the mixer, we did not include the second mechanism (i.e., using the macro-state of the terminated agent) as it showed significantly lower performance than using the joint macro-state in our preliminary experiments (due to the introduced noise for the other agent values). Nonetheless, we **significantly extended our ablation study** (Figure 5), discussing the lower performance of AVF-{QPLEX, QMIX} based **on the second mechanism and the real state**, both in WTD-S and (newly) in BP-10. Please, refer to the new Fig. 5 for further details.

Reviewer comment (Theory):
As shown in Figure 2, if the macro-action of agent at the current timestep is not terminated, its macro-observation will not change. But I think the reality is that a change in macro-action of any other agent will lead to a change in the macro-state, and ultimately lead to a change in the macro-observation of agent i. Can you explain your ideas about this?

Author response:
Regarding the macro-observations, it is true that a change in the macro-action of any agent will influence the macro-observation of agent i. However, following the MacDec-POMDP formalization (Section 2, Amato et al., 2019), *agent $i$ can only observe such influence upon the termination of its own macro-action $m^i$ when it receives its updated macro-observation*. Conversely, if agents receive a new macro-observation and select a new macro action every time any agent terminates a macro-action, they will likely converge to a primitive action-based solution (this aspect has also been discussed by previous macro-action works).

Reviewer comment (Novelty):
In the field of macro-action based MARL, are there other CTDE value-based algorithms?

Author response:
To the best of our knowledge, *there are no principled CTDE value-based algorithms in the field of macro-action-based MARL*, which further motivates the contributions of our work.

Reviewer comment (Reproducibility):
The authors mention low-level policy in the Preliminaries, but they do not mention how to select actions for low-level policy in AVF algorithms. In my opinion, if low-level policies do not work during the duration of a macro-action, then the macro-action can be simulated by executing a specific micro-action continuously over a period of time. Is it right?

Author response:
*It is correct that a macro-action can be simulated by executing a 1-step for a longer period*. For example, the “push” macro-action in BP domains is simply a repeated 1-step move forward action.

Reviewer comment (Reproducibility):
Different agents’ macro-actions start and end at different time steps, how to choose the duration time $\tau$ of the joint macro-action?

Author response:
The *duration time $\tau$ of the joint macro-action* is used for the joint reward accumulation and *is set to the time interval between any two macro-action terminations*. We clarified this in Section 2.2.

Reviewer comment (Novelty):
As far as I know, in the value factorization algorithms, in addition to the IGM and Adv-IGM principles, [1] proposes DIGM principle and [2] proposes RIGM principle. Can you discuss whether and how these two principles can be applied to the asynchronous framework?

Author response:
While it would be interesting to *apply DIGM and RIGM* to the asynchronous framework, we believe it is out of the scope of our work. In particular, *it would also require formalizing distributional macro-action-based MARL algorithms*, which have not been currently investigated by the few published works considering asynchronous MARL.

Reviewer comment (Evaluation):
What is the measure compared in Table 1? Average win rate or average return?

Author response:
Regarding minor questions, we appreciate the attention reserved by the reviewer for our manuscript. We fixed Eq. 5 and the Dec-POMDP tuple. Moreover, Table 1 reports the average return as all the other experiments (but it can be also interpreted as an average win rate given the nature of the reward function in CT-10). We clarified this in the table’s caption.