Experience Selection in Multiagent Deep Reinforcement Learning Experience replay is a crucial technology for offpolicy deep reinforcement learning, which uses a portion of memory as the replay buffer to store previous experience samples for the later policy update. Since each experience sample can be used multiple times, experience replay drastically improves the utilization rate of experience samples. However, how to effectively combine experience replay with multiagent reinforcement learning is still an open challenge. In multiagent reinforcement learning, the decision of the agent needs to consider the dynamic information of the environment as well as the behavior of other agents. If the policies of other agents change, updating the current policy with previous experience samples may deteriorate the agentsu0027 subsequent decisions. Some methods use a smallcapacity replay buffer to store recent experience samples. Although this avoids the problem that the experience sample in the replay buffer is not compatible with the current policy, it will reduce the diversity of experience samples in the replay buffer, that resulting in agents unable to learn the optimal strategy. This paper eases this conflict by enhancing the experience selection mechanism: 1) we use the reservoir retention algorithm to increase the diversity of experience samples in the replay buffer; 2) we use prioritized experience replay to alleviate the problem that the experience sample in the replay buffer is not compatible with the current policy. The experimental results on the covert communication problem confirm the effectiveness of our proposed method.

Experience Sharing Between Cooperative Reinforcement Learning Agents The idea of experience sharing between cooperative agents naturally emerges from our understanding of how humans learn. Our evolution as a species is tightly linked to the ability to exchange learned knowledge with one another. It follows that experience sharing (ES) between autonomous and independent agents could become the key to accelerate learning in cooperative multiagent settings. We investigate if randomly selecting experiences to share can increase the performance of deep reinforcement learning agents, and propose three new methods for selecting experiences to accelerate the learning process. Firstly, we introduce Focused ES, which prioritizes unexplored regions of the state space. Secondly, we present Prioritized ES, in which temporaldifference error is used as a measure of priority. Finally, we devise Focused Prioritized ES, which combines both previous approaches. The methods are empirically validated in a control problem. While sharing randomly selected experiences between two Deep QNetwork agents shows no improvement over a single agent baseline, we show that the proposed ES methods can successfully outperform the baseline. In particular, the Focused ES accelerates learning by a factor of 2, reducing by 51% the number of episodes required to complete the task.