Paper: A Collaborative Perspective on Exploration in Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
Maintaining $N$ training agents and environments at the same time may increase the computation and space complexity.

Author response:
We admit that the increasing computation and space complexity is the major limitation of our work as we explained in the common question section. In order to mitigate this issue, we implement our code in JAX with vectorized model architectures. In the experiment, using an agent number of size 4, the wall clock running time is around 2 times of training a single agent.
Another potential solution is to use a multi-head version of this algorithm, which is exactly the following question as we will discuss later.

Reviewer comment (Novelty):
The related work part can be further extended.

Author response:
Thanks for the suggestion. We rewrote the related work section.

Reviewer comment (Novelty):
Is this the first work on collaborative exploration in (single-agent) deep RL?

Author response:
No, we are not the first work on this topic. For example, one of the selected baselines, DiCE, also studies this problem by adding a diversity regularization to the loss function. However, our discussions of how and when to collaborate among agents and the instantiation of using the collaborative reward are novel.

Reviewer comment (Reproducibility):
Do the $N$ agents share parameters in their policy or value networks (I guess not, then all agents need to be trained)?

Author response:
No, the current agents do not share parameters. Since the main focus of this work is to study how and when to collaborate among agents, we adopt the most basic model architectures in the experiments.
Using a multi-head variant of the proposed method would be easy to implement where each agent shares the same torso network and uses different heads as the policy/value function.
Experimental results of the multi-head variant are reported in Section 5.8.

Reviewer comment (Evaluation):
Could you explain why the $L_2$ norm of the action difference is a good measure for behavior diversity, rather than some other designs (like difference in $\pi_i(\cdot|s_t)$)?

Author response:
We adopt the L2 norm in the pervious experiments mainly because it is intuitive and the overall results are similar to other more complex metrics. Since the purpose of Table 3 is mainly for providing some qualitative understanding of the proposed method. Therefore, we select the MSE-based metric due to its simplicity.
We followed the reviewers' suggestions and tried some other metrics, i.e., mutual information $I(S;A)$ and KL divergence $D_{KL}(\pi_i, \pi_j)$ to approximate policy diversity in the updated paper.

Reviewer comment (Reproducibility):
It would be great if the authors can release the codes for the baseline algorithms as well. Details on reproducing each figure (or training process) in the paper should also be given in the readme file.

Author response:
We have submitted the code of the proposed method in supplementary material at the time of our original submission. 
Following your suggestion, we will release more code including some baselines, the visualization scripts etc by including a Github link in the final version.

Reviewer comment (Novelty):
Connections with parameter sharing in MARL literature (for example [1, 2, 3, 4]) are completely missing. The general setup of using environment experience from 'multiple' agents to train a single policy is also used in those contexts and connections with explorations as investigated here might be relevant in those contexts as well.

Author response:
Thanks for the reminder. We added discussions of the connections and differences of our work with parameter sharing in MARL in both Section 2 and Appendix A. We also added a multihead variant of CE in Section 5.8, which uses parameter-sharing.

Reviewer comment (Theory):
How does contrastive reward work in early parts of training when initial policies are mostly random?

Author response:
In the early parts of training:
1. When initial policies are mostly random, different agents usually take low-quality actions and receive near zero rewards from the environment.
2. The introduced collaborative intrinsic reward encourages the agents to take actions that are not only novel to itself but also to the other agents. (Please refer to the reply of Q1 to R2). Actually, this is related to your other question on the balancing between intrinsic and task rewards. 
3. When some of the agents sampled good actions and received high rewards, then this information is shared to the other agents via the shared replay buffer.

Reviewer comment (Experiments):
How well would it scale to different kind of environemnts like Atari?

Author response:
We added extra experiments on the Atari games in Appendix D.7. We compare the proposed method to DQN on the Asterix, BeamRider, Breakout, and SpaceInvaders environments. In each environment, we train the agent for 5e6 environmental steps, and we evaluate the agent for every 2e5 steps. We run for 5 random seeds and report the mean evaluation score and standard deviation. We can observe that the proposed CE method also achieves good performances in Atari tasks.

Reviewer comment (Experiments):
Any intuitions on how to balance intrinsic vs task rewards?

Author response:
- In the current algorithm, as the quality of sampled data increases, the scale of the task will gradually outweigh the intrinsic reward.
- A widely used practice is to linearly decay the intrinsic reward coefficient during the training.
- Or we can adopt some more advanced method [1] to tune the intrinsic reward weight automatically.

[1] Yuan, Mingqi, et al. "Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learning." arXiv preprint arXiv:2301.10886 (2023).

Reviewer comment (Novelty):
The paper uses the term collaborative exploration when this subject has been quite studied in the past under the name "concurrent exploration". The paper is also missing a lot of existing work in this area [1-6]. I think it is worth adding these references in the next revision of the paper as section 3.1 could seem like the author came up with concurrent exploration.

Author response:
> Q1: missing related work on concurrent exploration.

Thanks for the suggestion. We first added a paragraph dedicated to concurrent exploration in the related work section and also explain the connection between our work and concurrent exploration in Section 3.1.

Reviewer comment (Evaluation):
It seems like the time complexity for computing the embeddings in section 4.1 is linear in the number of agents and number of states in the replay buffer. This seems detrimental; I assume that we're interested in using a multi-agent method because we care more about wall clock time than sample complexity, in that case the additional cost to compute the embedding may not be worth it. Table 10 shows that the wall clock time doubles with collaborative exploration.

Author response:
> Q2: time complexity for computing the embeddings is linear in the number of agents and number of states in the replay buffer.

Yes.  Denote the memory buffer size as M, embedding dimension as d, and the agent number as N. Then, the time complexity to compute the KNN intrinsic reward is $O(KNMd)$.
To mitigate this issue, we use a memory buffer for each agent with a relatively small size (M = 300).
- In practice, as discussed in the response to Q7 for R1, $N$ does not need to be a large number to strike the balance between complexity and benefits brought to exploration.

Reviewer comment (Writing):
The “multi-agent” phrasing is misleading. Nowhere in A3C, Ape-X, or IMPALA is the term used to refer to parallel instantiations of one agent. Repurposing a well-established terminology should be accompanied by far more solid evidence than a mere footnote.

Author response:
We apologize for the confusion and fully agree with you that the "multi-agent" term is indeed mis-leading. In our original submission, we were aware that there might be a confusion due to borrowing a term from another field, and intended to clarify it with a footnote in the original manuscript. However, it turns our that, as you commented, "repurposing a well-established terminology" does not seem to be a good strategy. Given the confusions it introduced, we fully agree with you on this point and have revised related terms and sections to address this.
In term of coming up with a proper term, as you have suggested, our work is indeed aligned with the line of parallel agent, and we further incorporated collaboration among them, therefore we have changed the term "multi-agent" to "collaborative parallel agents" to reflect both points.

Reviewer comment (Experiments):
Related works are poorly taxonomized. For instance, if MARL research and the proposed method are indeed “very closely related”, how is it that no MARL algorithm is tested against?

Author response:
We have detailed the relationship of the proposed method and parallel agents work [A3C, Apex-X, IMPALA etc] in the respose to Common Question-1. 
In short, our work is related to parallel agent in the sense it also has parallel agents. However, the key difference is that we intent to incorporate explicit collaboration among agent and further investigated the question of how and when to collaborate.
In more detail, our work differs from these related work in the following aspects:
- Our work concentrates on the problem of how to collaborate between agents to improve exploration, while A3C, Ape-X and IMPALA mainly focus on improving data throughput in the parallel agent setting.
- There is no explicit collaboration mechanism in A3C, Ape-X and IMPALA apart from the implicit way via data sharing. In our work, we attempt to answer the question of how to collaborate more effectively beyond data sharing and incorporate explicit collaboration.
We rewrote the related sections to distinguish our work and these parallel agent work more clearly.

Reviewer comment (Experiments):
Despite admitting a resemblance with distributed RL, none of the cited algorithms is set up for comparative evaluation.

Author response:
Thanks for the suggestion. We conducted additional experiments comparing our approach to A2C in the updated Table 1. We adopt the parameter settings from https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail/tree/master, where we run the A2C agent for 1e7 environmental steps. Notably, A2C usually underperforms other baselines, even with 10 times total environmental steps. This aligns with expectations, given that A2C-like agents (A2C/A3C/PPO) typically exhibit low sample efficiency compared with other off-policy algorithms in popular continuous control benchmarks. Our empirical results align with external evidence with on A2C and PPO, e.g.:
- A2C is reported to have lower sample efficiency (compared to e.g. SAC) reported by StableBaselines on MuJoCo: https://github.com/DLR-RM/stable-baselines3/issues/48
- PPO is reported to have similar performances on the DMC benchmark: https://github.com/RyanNavillus/PPO-v3

Reviewer comment (Presentation):
There is a mixup of neighboring lines of research, terminology, and taxonomization that make the paper exceedingly difficult to follow and its contributions hard to assess.

Author response:
We apologize for the inconvenience. This is indeed related to some confusions brought by our mis-usage of some terms etc. We read your comments carefully and categorize your concerns into two categories: (1) presentation related: Q1-Q3; (2) comparison related: Q4-Q6. 
We have addressed each category from several aspects as detailed below and hope the reviewer can reassess the paper based on our response and the revised paper.

Reviewer comment (Novelty):
Agent parallelization works have long shown to be faster and scalable ways to populate state-action visitation matrices, so claiming that most existing works take a single-agent perspective is a complete disregard to several rich lines of research predating this submission.

Author response:
We apologize for the confusion. We have used an imprecise term here.
By "most existing works take a single-agent perspective", we intended to mean that that many existing work do not take an explicit collaborative perspective, either because there is only one agent (where there is no structural support for collaboration), or parallel agents are used without explicit collaborations among them.
We have revised this sentence to make it more precise and rewrote the related sections to clarify the connections and differences of our work with previous works on parallel agents, i.e., Section 2 in the revised paper.

Reviewer comment (Experiments):
How does CE compare against DIAYN?

Author response:
Thanks for the suggestion. We are sorry that we missed the information theory-based baseline, DIAYN, in the original paper. In the previous experiments, we mainly focused on methods that do not change the model architecture of the backbone algorithm, so we overlooked this important baseline which learns a skill conditioned policy $\pi(a \vert s, z)$.
Notably, the experiment setting for DIAYN is different from the other baselines, which first pre-trains a set of skills with the pseudo-reward $r_z(s, a) = \log q_\phi(z \vert s) - \log p(z)$ and then fine-tunes the best-performing skill with the task reward. It's a little bit tricky to make fair comparison for DIAYN with other baselines if we use the same number of total environment interactions. In this supplementary experiment, we ignore this constraint and simply let the DIAYN agent run 2 times of the total environmental steps. We first use 1e6 environmental steps for pre-training and use another 1e6 environmental steps for fine-tuning. We run the experiment using the official code (https://github.com/haarnoja/sac/blob/master/DIAYN.md).
We added the results of DIAYN to Table 1 and Figure 11. From the results, CE outperforms DIAYN in most tasks. DIAYN performs worse than CE mainly because the skills discovered in the unsupervised skill learning stage are not necessarily well aligned with the down stream tasks.

Reviewer comment (Experiments):
How does CE scale with number of agent instances?

Author response:
Thanks for the suggestion. We added results of using 16/32/64 agents in Table 7 in Appendix D.3. The performance of scaling to more agents actually depends on the tasks.
It's notable that there is a trade-off between policy diversity and training stability:
- With more agents, we have more diverse policies and help to solve tasks where exploration is particularly difficult, i.e., the goal-reaching fish-swim task. 
- On the other hand, the off-policyness of the sampled data increases as we have more agents. Given a sampled batch, there are only on average 1/N samples that are collected by each agent. As the number N increase, the problem becomes closer to an offline RL setting, where the optimization becomes more challenging, i.e., the performance variances increase significantly for larger N.
Therefore, using a medium size agent set strikes a good balance of exploration and optimization.

Reviewer comment (Evaluation):
How does the KNN component scale with number of agent instances?

Author response:
- Denote the memory buffer size as M, embedding dimension as d, and the agent number as N. 
- Then, the time complexity to compute the KNN intrinsic reward is $O(KNMd)$, which is linear w.r.t. the agent number.
- In practice, as discussed in the response to Q7, $N$ does not need to be a large number to strike the balance between complexity and benefits brought to exploration.