Paper: Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The dimension of $x$ is very small (i.e. 4 according to table 4 in the appendix). It's doubtful that it can reconstruct the global state.

Author response:
A major objective of our embedding function is to construct semantically similar memories clustered close. This is quite different from the objective of a generative model whose main purpose is to reconstruct inputs. As presented in the manuscript, considering reconstruction loss shows its merits in semantic embedding with $\textrm{dim}(x)=4$.
    
In addition, we use $\textrm{dim}(x)=4$ is for fair comparison with EMC, which uses $\textrm{dim}(x)=4$ in their episodic memory construction.

Reviewer comment (Theory):
The approach adopts a state embedding instead of random projection. Does this make the approach more hard to converge?

Author response:
As the reviewer mentioned, semantic embedding requires additional training. However, adopting semantic embedding instead of random projection can improve the convergence of "policy" that we want to optimize because EMU guarantees the statistically coherent incentive toward desirable states. 
    
Also, we can check the results from Figures 15-16 and EMU (XXX-No-SE) in ablation studies presented in Figure 9 and Figure 22. For small $\delta$, adopting semantic embedding or random projection shows not much difference in terms of convergence, indirectly viewed from the performance variance. However, their convergence speed is distinctively different. In other words, semantic embedding shows a much faster convergence to optimal policy. For large $\delta$, the semantic embedding (dCAE) shows a better convergence performance than a random projection.

Reviewer comment (Experiments):
EMU needs to set a return threshold to determine the desirability of a trajectory. This may require some domain knowledge to properly determine it, even when using R_{max}. This knowledge may partially explain its outperformance.

Author response:
First, thank you for the valuable comments. The reason we argue our major contribution lies on cooperative MARL is that in such tasks, the common goal is often explicitly specified such as scoring a goal in GRF or defeating all enemies in SMAC. By utilizing that signal, we can determine whether a trajectory is desirable or not and use that signal to prevent local convergence. However, if there is no such explicit-common goal, one may resort to a domain knowledge or a preference on the level of return to be deemed successful.

Reviewer comment (Evaluation):
When the key encoder is updated, the proposed method needs to update all keys in the memory, which seems quite computationally intensive.

Author response:
We present the computational burden to update all key values in the episodic memory and to train $f_{\phi}$ and $f_{\phi}$ in **Appendix C.3**. Sorry for that if it was hard to find. In corridor task which has high-dimensional state space, the training and update only took less than 2 seconds at most, which is certainly negligible compared to MARL training.

Reviewer comment (Experiments):
It may be interesting to compare the proposed method with MAPPO.

Author response:
Since MAPPO relies on on-policy and other baseline algorithms including EMU are off-policy, it is difficult to set hyperparameters such as batch size, training iterations for a fair comparison. Thus, we work on the training framework and the corresponding hyperparameters, and if the time is available, we will present the result by the end of the rebuttal period.

**(updated)** We present the performance comparison of EMU with MAPPO in **Appendix D.12**. Please check the revised manuscript.

Reviewer comment (Experiments):
Can the author explain how to set the return threshold for desirability in experiments? Is this threshold dynamic or fixed?

Author response:
In our experiment, it is fixed. In cooperative MARL task, "desirability" is often determined explicitly as there is a common goal such as scoring a goal or defeating all enemies. Thus, in the task of SMAC and GRF we just use "desirability" signal generated by the environment, i.e., "battle_won" signal from the environment to determine whether an episode has achieved a goal or not. This signal can be identically generated if one set $R_{thr}=R_{max}$. The reason we present the desirability as $R \geq R_{thr}$ is for general definition considering the case where the explicit goal is not presented as in single-agent case. For these cases, we need to define $R_{thr}$ value, where the domain knowledge or preference on the level of reward to achieve should be introduced. However, again, in general cooperative MARL setting where the common goal is explicitly stated, we do not have to determine $R_{thr}$ value. We additionally conduct a parametric study on a single-RL task to see the effect of $R_{thr}$ value, considering the case where desirability is not specifically determined. For further details, please refer to Appendix D.9.

Reviewer comment (Theory):
Is the incentive reward only effective when a trajectory is desirable? Does this mean episodic memory is useful only when a very good trajectory is explored, which can be hard?

Author response:
The incentive is only given to states considered desirable. This mechanism prevents local convergence toward states that have a good return at the early training phase but are not optimal. As the reviewer mentioned, the episodic incentive works after finding desirable trajectories which can be done by standard exploration with $\epsilon$-greedy or some additional incentive such as $r^c$. However, the problem is that even after we find desirable trajectories, the previous work could not fully utilize those good memories. On the contrary, in EMU, thanks to the semantic embedding, we can utilize semantically similar memories and thus encourage some exploration toward states that currently have not achieved goals but have the potential to do so. In this way, EMU can encourage exploration toward desirable states, deemed undesirable without semantic embedding.

Reviewer comment (Experiments):
**Scalability**: Based on the encoder-decoder architecture of the paper, I am assuming that the global state for the environments used is feature-based. It is unclear whether this method will scale to vision-based environments due to **a)** the memory requirements of storing many images, **b)** the optimization difficulty in reconstructing image-based states and **c)** the effectiveness of the introduced reward structure in high-dimensional state spaces. However, I acknowledge that many existing works in this area utilize feature-based observations. Regardless, it would greatly improve the strength of the results of this paper if some gains could be shown in vision-based environments.

Author response:
**A1.** Thanks for the comment. First, we want to mention that our contribution lies majorly on applying episodic control method to cooperative MARL setting in a more efficient and robust way, by introducing a trainable semantic embedding and episodic incentive. As we evaluate EMU on multi-agent systems where high-dimensional state space, such as 322-dimension in MMM2 and 282-dimension in corridor, is already considered during centralized training, we deem our method also scalable to tasks with high-dimensional state spaces. In addition, episodic memory is saved in CPU rather than GPU, where the size RAM can be easily augmented.

If still saving all states is impossible in a certain task or a semantic embedding from original states becomes overhead, one may resort to pre-trained feature extraction model such as ResNet model provided by torch-vision in a certain amount for dimension reduction only, before passing through the proposed semantic embedding.
 
As an example, we implement EMU on the top of DQN model and compare it with original DQN on Atari task. For the EMU (DQN), we adopt some part of pre-trained ResNet18 presented by torch-vision for dimensionality reduction, before passing an input image to semantic embedding. We found a performance gain by adopting EMU on high-dimensional image-based tasks. Please refer to **Appendix D.9** for experimental details and the result.

Reviewer comment (Evaluation):
**Evaluation**: It appears that random projection performs almost equivalently to EmbNet/dCAE when compared using test win rates. The introduction of a new metric (overall win-rate) highlights that EmbNet/dCAE enable faster/more sample-efficient learning. However, improvement on this new metric is not as significant as the original win rate (which is the standard benchmark in the community). I would be curious to see the curve for EMU with random projection added to **Sections 4.1** and **4.2** to better understand the significance of EmbNet/dCAE.

Author response:
**A2.** We partially agree with the reviewer's statement since a long convergence time is one of predicaments in MARL training. Thus, we want to introduce a metric that measures both learning speed and quality. Although their win-rates seem comparable in relatively easy tasks such as 3s\_vs\_5z and 5m\_vs\_6m, adopting random projection instead of semantic embedding degrades the performance as presented in Figure 9 and Figure 22 in the manuscript. Please see EMU (XXX-No-SE) case compared to EMU (XXX). We also conduct additional experiments on EMU (XXX-No-SE) in other tasks. For details, please refer to **Appendix D.11** of the revised manuscript.

Reviewer comment (Experiments):
Results for **1)** in Weaknesses. These set of results are not completely necessary but would be good to see. A well-reasoned argument about why the method should not be difficult to scale will also suffice.

Author response:
**A1.** Thanks for the comment. First, we want to mention that our contribution lies majorly on applying episodic control method to cooperative MARL setting in a more efficient and robust way, by introducing a trainable semantic embedding and episodic incentive. As we evaluate EMU on multi-agent systems where high-dimensional state space, such as 322-dimension in MMM2 and 282-dimension in corridor, is already considered during centralized training, we deem our method also scalable to tasks with high-dimensional state spaces. In addition, episodic memory is saved in CPU rather than GPU, where the size RAM can be easily augmented.

If still saving all states is impossible in a certain task or a semantic embedding from original states becomes overhead, one may resort to pre-trained feature extraction model such as ResNet model provided by torch-vision in a certain amount for dimension reduction only, before passing through the proposed semantic embedding.
 
As an example, we implement EMU on the top of DQN model and compare it with original DQN on Atari task. For the EMU (DQN), we adopt some part of pre-trained ResNet18 presented by torch-vision for dimensionality reduction, before passing an input image to semantic embedding. We found a performance gain by adopting EMU on high-dimensional image-based tasks. Please refer to **Appendix D.9** for experimental details and the result.

Reviewer comment (Experiments):
Results for **2)** in Weaknesses.

Author response:
**A2.** We partially agree with the reviewer's statement since a long convergence time is one of predicaments in MARL training. Thus, we want to introduce a metric that measures both learning speed and quality. Although their win-rates seem comparable in relatively easy tasks such as 3s\_vs\_5z and 5m\_vs\_6m, adopting random projection instead of semantic embedding degrades the performance as presented in Figure 9 and Figure 22 in the manuscript. Please see EMU (XXX-No-SE) case compared to EMU (XXX). We also conduct additional experiments on EMU (XXX-No-SE) in other tasks. For details, please refer to **Appendix D.11** of the revised manuscript.

Reviewer comment (Reproducibility):
It would be helpful to provide details about the state space, action space, environment reward and episode lengths for both SMAC and GRF in the Appendix.

Author response:
**A3.** Thanks for the comments, we present the details about the dimension of state space, action space, reward settings, and episode lengths for both SMAC and GRF in **Appendix D.10**. Please refer to it.

Reviewer comment (Writing):
**Section 3.2** can use more intuition and better writing. It is unclear to me why the episodic inventive for a desirable transition is set to be proportional to the difference between the true value and the predicted value. Is this done to incentivize visits to states where the Q network has not converged?

Author response:
**A4.** More exactly, the proposed episodic incentive $r^p$ is given to states deemed desirable, i.e., feasible to achieve a common goal or $R_{thr}$. The incentive is not provided to undesirable states regardless of their Q-network convergence. Compared to the conventional episodic control, the proposed episodic incentive determines 1) which transition to give an incentive and 2) how much incentive to provide. Determining a desirable transition can be done by checking the desirability of the next state. A remaining issue is how much we incentivize those transitions. Arbitrary incentives make it hard to work well on various tasks since one may need to adjust the corresponding scale factor gradually during training or depending on the level of tasks. 
    
Instead, we developed an incentive whose magnitude is automatically adjusted during training, as $r^p$ proportional to the gap $H(f_{\phi}(s'))-max_{a'}Q_{\theta^-}(s',a')$. This gap compensates the underestimated value of $s'$ compared to $H(f_{\phi}(s'))$. 

Here, $H(f_{\phi}(s')) \rightarrow V^*(s')$ if $s'$ is desirable. However, a direct compensation can be too optimistic, thus we utilize the expected value of it by the count-based estimation with $N_{\xi}(s')/N_{call}(s')$. When $max_{a'}{Q_{\theta^-}}(s',a')$ accurately estimates $V^*(s')$, the original TD-target is preserved as the episodic incentive becomes zero, i.e., $r^p \rightarrow 0$. It is worth noting that the arbitrary magnitude of an incentive could hurt the convergence unless it converges to zero when $max_{a'}{Q_{\theta^-}}(s',a')$ already accurately estimates $V^*(s')$.
    
Thus, the proposed episodic incentive gives an additional reward only to desired transition with the amount of discrepancy between true value and value estimated by target Q-network. In this way, we do not worry about the amount of incentive and a manual control for the corresponding scale factor as in the conventional episodic control. This is another key contribution of this paper in cooperative MARL settings.

Reviewer comment (Theory):
What are the implications of *Theorem 2*?

Author response:
**A5.** Through the proposed episodic incentive, we can generate a gradient signal as the optimal gradient for desired transitions. In other words, $r^p$ is designed to generate the correct gradient signal to the desirable transition, i.e., a transition toward $s'$ such that $\xi(s')=1$.