Paper: MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment (ICLR-2024)

Reviewer comment (Experiments):
The states of the environments adopted by this paper are low-dimension vectors. It is hard to capture the causal relationship between image state and reward.

Author response:
Thanks for the question. This work focuses mainly on state-based tasks. 1) For the setting where states are not defined, such as image-like input, our work can be applied by employed in the latent state space, which requires learning a latent vector representation. 2) For a large number of dimensions of state, this would increase the complexity of the causal modelling and thus require more efficient causal discovery methods. 3) As an advantage, by learning the causal structure, we can constrain the optimization of the models over a small subspace of state and action, resulting in a lower requirement of parameters of the neural network. 4) considering that we wanted a fairer comparison with other baselins under this setting, we followed the same data format and level of dimension of states with [1][2][3][4]. Overall, the main focus of this work is to address the team rewards credit assignment problem under the offline setting. Therefore, we consider the standard and general state setting.

- [1] Samvelyan, Mikayel, et al. "The starcraft multi-agent challenge." arXiv (2019).
- [2] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv (2020).
- [3] Pan, Ling, et al. "Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification." ICML. PMLR, 2022.
- [4] Yang, Yiqin, et al. "Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 10299-10312.

Reviewer comment (Writing):
The overall writing quality needs to be improved. There are some grammatical errors.

Author response:
Thanks for pointing it out. We will revise our paper carefully in the revision.

Reviewer comment (Experiments):
Although the evaluation of MACCA has been conducted in multiple benchmarks to demonstrate its generalizability, it is questionable whether the method has top performance in all of the environments that belong to a specific benchmark. For example, the SMAC benchmark has more than 20 original battle scenarios, including a few Super Hard challenges. However, only three of the maps, including a single Super Hard challenge, are shown in the results.

Author response:
Thank you for your concern. Our paper mentions that the primary reason for choosing the SMAC benchmark is to demonstrate performance in larger-scale scenarios (Section 5.2). Accordingly, we focused on maps that are challenging and involve more than five agents. However, considering your concern and acknowledging that the extensive experimentation may further substantiate MACCA's SOTA performance, we will include additional Super Hard map in our next revision.

Reviewer comment (Novelty):
Regarding the MACCA architecture, clearly, the policy learning part is crucial for the model. However, this part is mostly taken from previously established methods such as I-CQL, OMAR and MA-ICQ, especially the term \( J_{\pi} \). Also, after looking at the experiments in different environments, it is not clear whether MACCA-CQL, MACCA-OMAR, or MACCA-ICQ has the SOTA performance overall.

Author response:
Thank you for highlighting this aspect of the MACCA architecture. Firstly, it's important to note that MACCA is designed to be a plug-and-play method, capable of being seamlessly integrated with other offline MARL frameworks. This adaptability is, in fact, one of MACCA's strengths, as it allows for versatile applications across a range of existing methods.

| Env/Algo| MACCA-CQL | MACCA-OMAR| MACCA-ICQ|
| -------- | -------- | -------- |-------- |
| MPE     | +847.9%   | +62.0%      | +237.8%     |
| MA-Mujoco| +24.0%   | +44.6%      | +92.3%      |
| SMAC    | +788.5%   | +154.17%    | +271.77%      |


Secondly, regarding the performance of MACCA in different environments, we provide detailed data in the paper to prove its efficacy. Here, we summarize the results in the table above, including the average percentage improvement of all MACCA-based methods in these three environments. According to the results presented in our paper, MACCA-based algorithms consistently meet or exceed SOTA performance in all test environments and maps. This is demonstrated by the bold entries in Tables 1 and 2, which highlight the best performance in the experiments.

Reviewer comment (Reproducibility):
The code for the proposed method is not included in the submission. The disclosed information about the hyper-parameters and environmental settings is limited.

Author response:
We appreciate your feedback. The description of hyperparameters is in Appendix D, including the model structure and specific hyperparameters. The environment settings are in Appendix C, including the configuration and data set generation methods of three different environments. Also, to enhance clarity and reproducibility, we have uploaded the code for our proposed method to an anonymous repository, accessible at: https://anonymous.4open.science/r/MACCA_ICLR/.

Reviewer comment (Writing):
There is a long and confusing sentence in section 4.2: "Its primary objective is to ... reduce the number of features that a given depends on, ... mitigates the risk of overfitting." What does "reduce the number of features that a given depends on" mean?

Author response:
We will clarify this in the future version: the $L_{\mathrm{reg}}$ can reduce the difficulty of estimation of the reward function since it can clear the redundant features during training by reducing the sparsity of the causal structure $\mathcal{S}_{s r}$ and $\mathcal{S}_{a r}$ (defined in Section 5.3), in order to mitigate the risk of overfitting.

Reviewer comment (Experiments):
Can you mention how many independent tasks do the Multi-agent Particle Environment and the Multi-agent MuJoCo have, respectively?

Author response:
Yes, as described in Appendix C, there are three scenarios (PP, CN and World) in the Multi-agent Particle Environment and one scenario (Half-Cheetah) in Multi-agent MuJoCo.

Reviewer comment (Experiments):
In section 5.3 you mentioned that "It is important to note that our method is not highly sensitive to the hyperparameters despite using them to control the learned causal structure." Can you justify this argument?

Author response:
Thank you for the question. To investigate the impact of the hyperparameters, we refer to the results presented in Section 5.3 Table 4. This ablation study discusses how different values of \(\lambda_1\) affect the sparsity of the learned causal structure. The results indicate that while varying $\lambda_1$ leads to changes in the sparsity of the causal structure, the impact on the convergence of policy learning is relatively minimal.

| $\lambda_2$ / t | 1e4   | 5e4  | 1e5 | 2e5 |
| -------- | -------- | -------- | -------- | -------- |
| 0     |    17.4 ± 15.2(0.98) |  93.1 ± 6.4  (1.0)  | 105 ± 3.5  (1.0)  | 107.7 ± 10.2 (1.0)   | 
| 0.007    |  19.9 ± 12.4 (0.88) | 90.2 ± 7.1 (1.0)   |  108.8 ± 4.0 (1.0) | **111.7 ± 4.3** (1.0)    | 
| 0.5    | 13.3 ± 11.1 (0.68)    | 100.5 ± 14.0  (0.84)   | 102.9 ± 16.4 (0.87)  | 108.4 ±  6.4 (0.98) | 
| 5.0    | 2.3 ± 9.8  (0.0)    | -1.3 ± 25.4   (0.34)  | 70.4 ± 18.0  (0.62)   | 100.1 ±  7.4  (0.75)  | 

Additionally, we add an additional ablation study on $\lambda_2$ as shown in the above table, which presents the mean and the standard variance of the average normalized score with diverse $\lambda_2$ in the MPE-CN task. The value in brackets is the sparsity rate $\mathcal{S}_ {ar}$ of $\hat{\boldsymbol{c}}_ t^{i, a \rightarrow r}$, whose definition can be found in Section 5.3. For all values of $\lambda_2$, the sparsity rate $\mathcal{S}_ {a r}$ consistently begins from zero. Over time, there is a discernible increase in $\mathcal{S}_{a r}$, and the convergence rate slows down with the increase of $\lambda_2$. And, the convergence rate to an elevated sparsity rate decelerates with the increase of $\lambda_2$. This pattern intimates that higher $\lambda_2$ values engender a more measured modulation in the causal impact exerted by actions on individual rewards. More importantly, despite the variation in $\lambda_2$ values, the average normalized scores converged towards a similar range.

Reviewer comment (Experiments):
You showed the visualizations of two causal structures in MPE. Have you done similar work in MA-Mujoco and SMAC?

Author response:
Thank you for the question. The primary reason we focused on visualizations in MPE is due to its comparatively smaller state and action spaces relative to MA-Mujoco and SMAC. This smaller scale facilitates a more straightforward presentation and more accessible human analysis. Taking into account your suggestions regarding extending these efforts to MA-Mujoco and SMAC, we recognize its value and will update it in future revision.

Reviewer comment (Writing):
In the sub-section "Visualization of Causal Structure." in 5.3, what do "S2R" and "A2R" stand for?

Author response:
Thanks for pointing out these typos, "S2R" refers to Figure 2.a, which is "Causal Structure: States to Individual Rewards", and "A2R" refers to Figure 2.b, which is "Causal Structure: Individual Reward Actions".

Reviewer comment (Theory):
The presumed data-generating process is restrictive.

Author response:
Thank you for the question. We understand your concern regarding the expressive capability of our linear model in MACCA. Firstly, it is essential to emphasize that MACCA is adaptable for some specific nonlinear team reward functions. For instance, in cases where the team reward $R$ is defined as either the maximum or minimum of individual rewards $r_t^i$ in the team, the identifiability of individual rewards under this setting is retained, and our theoretical proofs remain valid. Secondly, our linear configuration does not directly impact the training of the network. Although our model assumes the team reward $R$ as the sum of individual rewards, the primary output of the $\psi_r$ is the estimated individual rewards $\hat{r}_t^i$, not directly generating the estimated team reward $\hat{R}_t$. Finally, team reward can be modelled as linear in many scenarios, particularly in cooperative games. A pertinent example is urban traffic management, where multiple agents control traffic signals at various intersections. Here, the traffic flow at each intersection and the overall throughput of the system exhibit a linear relationship. By aggregating the contributions from each intersection, the effectiveness of the entire traffic management system can be effectively quantified. Therefore, the linear model is sufficient so far, and we are considering discussing more complex settings and providing corresponding solutions in future work.

Reviewer comment (Theory):
Have the authors ever considered extending the algorithm to scenarios where the team reward cannot be decomposed as the sum of individual rewards? What can be the possible solution?

Author response:
Thank you for the question. Yes, we will consider extending to such non-sum scenarios in future work. The possible direction is to use some nonlinear integration function to generate the team reward instead of the sum function.

Reviewer comment (Reproducibility):
The design specifics of $c^{i, s \rightarrow r}$ and $c^{i, a \rightarrow r}$ are not elaborated upon, and the particular networks used are not clearly mentioned.

Author response:
**Answer w2:** Apologies for any confusion. In fact, we have introduced the design specifics of $c^{i, s \rightarrow r}$ and $c^{i, a \rightarrow r}$ in Section 4.2 and describe the networks used in Appendix D.2. The masks $\boldsymbol{c}^{i, \boldsymbol{s} \rightarrow r} \in\{0,1\}^{D_s}$ and $\boldsymbol{c}^{i, \boldsymbol{a} \rightarrow r} \in\{0,1\}^{D_a}$ in equation 2 denote as the binary masks in which control if a specific dimension of the state $s$ and action $a$ impact the individual reward $r_t^i$, separately. To estimate them, we use $\psi_g^{s \rightarrow r}$ and $\psi_g^{a \rightarrow r}$ in equation 3, the network we used share the same architecture but the parameters. It includes three fully-connected layers with a hidden size of 256, followed by an output layer with a scalar output. Each hidden layer is activated using the rectified linear unit (ReLU) activation function.

Reviewer comment (Reproducibility):
There isn't much explanation regarding the setting of the hyperparameter $h$, nor is there any mention of whether the value of $h$ remains consistent across different offline environments.

Author response:
**Answer W3&Q2:** Thank you for your query. The design specifics of hyperparameter $h$ is elaborated in Section 4.3 Individual Rewards Assignment, and the value of $h$ is also elaborately in Appendix D.3 Table A2.

The hyperparameter $h$ is a threshold to determine the existence of the causal edges after we get the scalar output of $\psi_g^{s \rightarrow r}$ or $\psi_g^{a \rightarrow r}$ in equation 3. During the inference, i.e., predicting the individual rewards for policy optimization, the binary masks to capture the causal structure are obtained by $\boldsymbol{c} =I(ReLU(o)>h)$ where $o$ is the output of the last layer of $\psi_g^{s \rightarrow r}$ or $\psi_g^{a \rightarrow r}$, and $I$ is indicator function. The value of the $h$ is invariant across different experiments, which is $0.1$, as shown in Table A2.

Reviewer comment (Theory):
Is the MACCA algorithm applicable to online reinforcement learning? Because it seems that in online reinforcement learning, causal relationships can also be applied, and the causal relationships among states, actions, and individual rewards can be constructed through Bayesian networks. Why is it emphasized that the MACCA algorithm is primarily for offline environments?

Author response:
**Response to W1&Q1:**  We appreciate your interest in the potential application of our MACCA framework to online MARL settings. Our choice to focus on the offline credit assignment setting was driven by several primary considerations. 

Firstly, the restricted exploration in offline settings poses unique challenges to this credit assignment problem. When relying only on team rewards, bias in offline data or significant performance differences between agents can seriously affect the accuracy of the evaluation. In contrast to offline settings, online credit assignment algorithms like COMA or SQDDPG can iteratively refine their evaluations through continuous interaction with the environment. Therefore, the challenge that MACCA focuses on in offline environments is accurate agents' contribution based on offline data, which is crucial to understanding the role of each agent and optimising overall team performance in scenarios with restricted exploration.

Secondly, an essential aspect of MACCA is the training of causal models, which significantly benefits from the stable and consistent data distribution typically found in offline reinforcement learning environments. In offline settings, data often comes from policies with comparable performance levels, leading to a more uniform distribution. This stability is advantageous for causal learning, as it reduces the variability and potential distribution shifts that can occur in online settings, particularly in online and on-policy scenarios where an experience replay buffer is not utilized. Off-policy methods in online settings may offer some mitigation against this issue, but they are not without their challenges, such as potential shifts in the experience replay buffer's distribution. Given these considerations, our focus on offline settings was a deliberate choice, aligning well with the objectives and strengths of the MACCA framework.

Furthermore, MACCA has demonstrated superior performance across three distinct environments. We believe that these results affirm the efficacy of MACCA in the offline setting and hope this addresses any concerns regarding our choice of focus and the potential of MACCA in different application scenarios.

Reviewer comment (Experiments):
The paper's explanation regarding the setting and values of the hyperparameter $h$ seems to be unclear. Is the value of $h$ set the same across all offline datasets?

Author response:
**Answer W3&Q2:** Thank you for your query. The design specifics of hyperparameter $h$ is elaborated in Section 4.3 Individual Rewards Assignment, and the value of $h$ is also elaborately in Appendix D.3 Table A2.

The hyperparameter $h$ is a threshold to determine the existence of the causal edges after we get the scalar output of $\psi_g^{s \rightarrow r}$ or $\psi_g^{a \rightarrow r}$ in equation 3. During the inference, i.e., predicting the individual rewards for policy optimization, the binary masks to capture the causal structure are obtained by $\boldsymbol{c} =I(ReLU(o)>h)$ where $o$ is the output of the last layer of $\psi_g^{s \rightarrow r}$ or $\psi_g^{a \rightarrow r}$, and $I$ is indicator function. The value of the $h$ is invariant across different experiments, which is $0.1$, as shown in Table A2.

Reviewer comment (Experiments):
In the ablation study section, the impact of $\lambda_1$ on the causal structure, specifically its influence on the state's effect on individual rewards, was explored. However, has the impact of $\lambda_2$ on the causal structure been considered? I didn't see this part being researched in the paper.

Author response:
**Answer 3:** Thank you for your question. We have conducted additional experiments on $\lambda_2$ and show the results in the table below.

| $\lambda_2$ / t | 1e4   | 5e4  | 1e5 | 2e5 |
| -------- | -------- | -------- | -------- | -------- |
| 0     |    17.4 ± 15.2(0.98) |  93.1 ± 6.4  (1.0)  | 105 ± 3.5  (1.0)  | 107.7 ± 10.2 (1.0)   | 
| 0.007    |  19.9 ± 12.4 (0.88) | 90.2 ± 7.1 (1.0)   |  108.8 ± 4.0 (1.0) | **111.7 ± 4.3** (1.0)    | 
| 0.5    | 13.3 ± 11.1 (0.68)    | 100.5 ± 14.0  (0.84)   | 102.9 ± 16.4 (0.87)  | 108.4 ±  6.4 (0.98) | 
| 5.0    | 2.3 ± 9.8  (0.0)    | -1.3 ± 25.4   (0.34)  | 70.4 ± 18.0  (0.62)   | 100.1 ±  7.4  (0.75)  | 

This table shows the mean and the standard variance of the average normalized score with diverse $\lambda_2$ in the MPE-CN task. The value in brackets is the sparsity rate $\mathcal{S}_ {ar}$ of $\hat{\boldsymbol{c}}_ t^{i, a \rightarrow r}$, whose definition can be found in Section 5.3. For all values of $\lambda_2$, the sparsity rate $\mathcal{S}_ {a r}$ consistently begins from zero. Over time, there is a discernible increase in $\mathcal{S}_{a r}$, and the convergence speed slows down with the increase of $\lambda_2$. This pattern intimates that higher $\lambda_2$ values engender a more measured modulation in the causal impact exerted by actions on individual rewards. Furthermore, despite the variation in $\lambda_2$ values, the average normalized scores across different $\lambda_2$ settings eventually converge towards a similar level.

Reviewer comment (Novelty):
In the related work section on multi-agent credit assignment, it seems that recent approaches in value decomposition, such as wqmix, qplex, resq, etc., have not been considered. These methods tackle the credit assignment challenge among agents using value decomposition techniques and, from what I understand, they have demonstrated commendable performance. Have you considered applying the methods from wqmix, qplex, resq, etc., to offline multi-agent environments?

Author response:
**Answer Q4:** Thank you for the question. As reflected in Table 3, we have conducted a comprehensive comparison with SOTA online off-policy credit assignment baselines (including QMIX, SHAQ, and SQDDPG) and their combination with offline learning algorithms to maintain a fair and unbiased comparison. We noticed that the plethora of QMIX variants that employ diverse mixing networks for value decomposition, their application in offline settings is indeed worth considering for a more exhaustive comparison. Nevertheless, our comparative analysis primarily concentrates on explicit credit assignment methods. Upon a thorough evaluation of both the performance and robustness of these implicit credit assignment methods in online scenarios, we opted to utilize QMIX as a representative model for all value decomposition approaches.