Paper: APC: Predict Global Representation From Local Observation In Multi-Agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
While updating the critic network, the gradient generated by prediction loss in the algorithm will update the state representation. Is this training stable?

Author response:
We block the gradient update of the global representations provided by the critic when calculating the prediction error to ensure that the critic’s update only considers the error of estimating the state-action value.

Reviewer comment (Theory):
How to ensure convergence? What is the loss error compared to learning during training?

Author response:
For the actor network, we only added a prediction module to learn the global representations and provide additional information for the subsequent decision-making process. So it will not affect the convergence of the network.

Reviewer comment (Theory):
The primary concern in this paper revolves around the significance of predicting the representation in the critic. The representations in the critic undergo drastic changes, exhibit high levels of instability, and lack interpretability. The author provides intuitive reasons for its effectiveness but lacks theoretical support and experimental analysis. Furthermore, due to the complexity of the algorithm architecture, including the use of GRU, attention mechanisms, and complex AC network structure, analyzing why this method is effective becomes more challenging.

Author response:
In fact, our network structure is not as complicated as you described. The structure of the critic network is also used by MAT (MAT-Dec as well). In the actor network, we only added a GRU module for learning global representation compared to MAT-Dec.

Reviewer comment (Theory):
Why do we need to the loss p? Directly sharing the critic's input with the actor might be a more straightforward approach.

Author response:
Sharing the global information provided by the critic directly with the actor would break the CTDE framework. This will make the algorithm centralized and require global information as input during the execution phase, which is not allowed in many application scenarios. (This point has been mentioned in the Introduction section.)

Reviewer comment (Experiments):
Intuitively, it seems that the proposed predictive loss in this paper should not rely on such complex architectural designs. Only through a comparison under identical architectures can it be demonstrated whether the proposed loss in this paper is truly effective. Would there be a performance gain when comparing under the same structure?

Author response:
As A2 has mentioned, our network is essentially based on MAT-Dec with the addition of a representation prediction module based on GRU. And the experimental results do show significant improvement compared to MAT-Dec which proves the effectiveness of APC. However, we still appreciate the reviewer’s suggestion and will further try to add the representation prediction module to MAPPO and observe the experimental results.

Reviewer comment (Experiments):
Could the authors provide a performance comparison on different tasks using the same network architecture as the baseline?

Author response:
As A2 has mentioned, our network is essentially based on MAT-Dec with the addition of a representation prediction module based on GRU. And the experimental results do show significant improvement compared to MAT-Dec which proves the effectiveness of APC. However, we still appreciate the reviewer’s suggestion and will further try to add the representation prediction module to MAPPO and observe the experimental results.

Reviewer comment (Experiments):
Besides adding the predicting loss, the authors also utilize the residual connections in the network design, the authors do not show the ablation study of this trick and whether it could boost the performance of the compared baselines.

Author response:
***A3***: Due to time constraints, we only conducted ablation experiments about the number of layers on the 6h vs 8z map in SMAC, and the results are as follows:

| APC with mlp1 | APC with mlp3 | APC with mlp5 |
| -------- | -------- | -------- |
| 66.7(3.3) | 81.8(7.1) | 72.3(3.6) |

We changed the number of layers in the MLP of the actor (with 1, 3, and 5 layers). It can be seen that as the number of layers in the network increased from 1 to 3, the winning rate increased significantly. However, when the number of layers increased to 5, the network’s performance slightly decreased, which is consistent with the point mentioned in this paper that information transmission and gradient updates become more difficult as the number of layers in the network increases. We believe that the ablation experiments in the article are sufficient to attribute the performance of APC to global representation prediction rather than changes in network layers or tricks like residual connections.

Reviewer comment (Reproducibility):
Could the authors provide the parameter numbers of APC and the compared baselines?

Author response:
***A1***: Here are the parameter numbers (on the 6h vs 8z map) of APC and the compared baselines:

| APC | MAT-Dec | MAPPO | QMIX |
| -------- | -------- | -------- | -------- |
| 116513 | 60361 | 72553 | 73871 |

Most of the additional parameters of APC come from the prediction module in the actor, which is mainly based on GRU.

Reviewer comment (Experiments):
Could the authors provide the ablation studies of the number of network layers and residual connections?

Author response:
***A3***: Due to time constraints, we only conducted ablation experiments about the number of layers on the 6h vs 8z map in SMAC, and the results are as follows:

| APC with mlp1 | APC with mlp3 | APC with mlp5 |
| -------- | -------- | -------- |
| 66.7(3.3) | 81.8(7.1) | 72.3(3.6) |

We changed the number of layers in the MLP of the actor (with 1, 3, and 5 layers). It can be seen that as the number of layers in the network increased from 1 to 3, the winning rate increased significantly. However, when the number of layers increased to 5, the network’s performance slightly decreased, which is consistent with the point mentioned in this paper that information transmission and gradient updates become more difficult as the number of layers in the network increases. We believe that the ablation experiments in the article are sufficient to attribute the performance of APC to global representation prediction rather than changes in network layers or tricks like residual connections.

Reviewer comment (Theory):
Could the authors provide more explanations about the choice of Euclidean Distance as the distance metric?

Author response:
***A4***: We tested both the Euclidean distance and the MSE loss function, and found no significant difference between these two. So we used the Euclidean distance as the loss function.

Reviewer comment (Novelty):
The methodological contribution other than predicting the global representations (i.e., Loss_{prediction}) was unclear to me. Although the idea of APC is simple and effective, comparison with CTDE baselines seemed to be obvious because APC can access the global information in the training phase.

Author response:
Yes, our main contribution is making the actor learn the global representations of the centralized critic so that the critic can provide more information to help with training. Although only adding a prediction loss seems like a small innovation, we believe that the idea of utilizing more information from the critic during training is concise and novel enough.

Reviewer comment (Writing):
Careful introduction may be necessary for the description of the proposed method.

Author response:
Thanks for your suggestion, we'll add more detailed explanations and references in the latest version. In Eq.(1) $T$ denotes the batch size and $V_\phi(s)$ denotes the state value. The structure of the critic is mainly based on MAT (it has been mentioned in the paper).

Reviewer comment (Writing):
3.2: Careful introduction of the proposed method may be required. There have been many actor-critic algorithms and what and why the authors select the base algorithm may be necessary for understanding the formulation. In particular, before Eq. (1) is it better to add some explanations (with references)? And for minor points, T and V are undefined. In my understanding, the critic is not the paper’s original point.

Author response:
Thanks for your suggestion, we'll add more detailed explanations and references in the latest version. In Eq.(1) $T$ denotes the batch size and $V_\phi(s)$ denotes the state value. The structure of the critic is mainly based on MAT (it has been mentioned in the paper).

Reviewer comment (Novelty):
In my understanding, Loss_{prediction} is only the original point in this paper. Is it correct? Or is there any improved point? The methodological contribution other than predicting the global representations (i.e., Loss_{prediction}) was unclear to me.

Author response:
Yes, our main contribution is making the actor learn the global representations of the centralized critic so that the critic can provide more information to help with training. Although only adding a prediction loss seems like a small innovation, we believe that the idea of utilizing more information from the critic during training is concise and novel enough.

Reviewer comment (Experiments):
How about the ablation study about the GRF environment? I expected the results to be unstable, but some insights can be obtained.

Author response:
Due to time constraints, we only conducted ablation experiments about the counterattack easy task in GRF, and the results are as follows:

| APC | APC without GRU | APC without loss_pred |
| -------- | -------- | -------- |
| 0.938(0.044) | 0.854(0.097) | 0.896(0.064) |

The results are still consistent with those in the paper.