Data-Efﬁcient Exploration with Self Play for Atari
Abstract
Most reinforcement learning (RL) algorithms rely on hand-crafted extrinsic rewards to learn skills.
However, crafting a reward function for each skill is not scalable and results in narrow agents that learn reward-speciﬁc skills.
To alleviate the reliance on reward engineering it is important to develop RL algorithms capable of efﬁciently acquiring skills with no rewards extrinsic to the agent.
While much progress has been made on reward-free exploration in RL, current methods struggle to explore efﬁciently.
Self-play has long been a promising approach for acquiring skills but most successful applications have been in multiagent zero-sum games with extrinsic reward.
In this work, we present SelfPlayer, a data-efﬁcient single-agent self-play exploration algorithm.
SelfPlayer samples hard but achievable goals from the agent’s past by maximizing a symmetric KL divergence between the visitation distributions of two copies of the agent, Alice and Bob.
We show that SelfPlayer outperforms prior leading selfsupervised exploration algorithms such as GoExplore and Curiosity on the data-efﬁcient Atari benchmark.
1. Introduction
Deep Reinforcement Learning (RL) has seen a number of breakthrough advances over the last decade.
From learning to play Atari video games from pixels (Mnih et al., 2015), to mastering the game of Go (Silver et al., 2017b), to learning robotic locomotion (Schulman et al., 2017) and manipulation (Kalashnikov et al., 2018) skills, to competing at grandmaster level on large scale multi-agent video games (Vinyals et al., 2019).
All of these breakthroughs were based on training RL algorithms to maximize an extrinsic reward that was provided by the human designers of
International Conference on Machine Learning, Unsupervised Reinforcement Learning Workshop , 2021.
Copyright 2021 by the author(s).
Atari100k Scores for Unsupervised RL
SelfPlayer vs. Supervised RL these environments, and indeed most RL algorithms today are developed with the assumption that an extrinsic reward function is provided.
However, dense reward shaping is a manual and cumbersome process and many real-world tasks that humans solve on a daily basis either have delayed sparse rewards or no apparent rewards at all.
How can we enable artiﬁcial agents to explore their worlds efﬁciently in a self-supervised manner without access to task-speciﬁc extrinsic rewards?
Perhaps the most common approach to self-supervised exploration in RL is intrinsic motivation (Oudeyer & Kaplan, 2009) where an intrinsic reward function is speciﬁed that does not depend on the extrinsic reward.
The RL agent still optimizes a reward function but the reward function does not use any data extrinsic to the agent.
A popular example of an agent trained with an intrinsic motivation objective is
Curiosity (Pathak et al., 2017) though many other instrinsic motivation objectives exist (Bellemare et al., 2016;
Osband et al., 2016;
Eysenbach et al., 2019).
A less commonly used but alternate approach is through novel goal discovery where an agent sets progressively harder goals for itself dur- ing training and uses a goal-reaching policy to achieve them.
A recent example of this approach is GoExplore (Ecoffet et al., 2020), which uses a count-based heuristic to score previous goals achieved and select challenging ones.
This work falls into the latter category of exploration methods with a focus of automatic goal discovery in the singleagent setting through self-play.
Self-play has long been utilized for effective exploration in multi-agent settings.
It was a key component that enabled state-of-the-art performance in large-scale multi-agent experiments (Silver et al., 2017a;
Vinyals et al., 2019; Berner et al., 2019).
However, given that self-play is deﬁned as a game between two or more agents, it has been more challenging to employ in the single-agent setting.
Recently, it was shown (Sukhbaatar et al., 2018; OpenAI et al., 2021) that self-play can also be utilized for singleagent problems.
In asymmetric self-play, the single agent maintains two copies of itself - Alice and Bob - that play an asymmetric game where Alice tries to achieve goals that are hard for Bob to clone and Bob in turn tries to copy Alice.
Through this adversarial game, an automatic goal-reaching curriculum is built.
While promising, the main shortcoming of this approach is that the self-play reward requires
Monte-Carlo rollouts for Alice’s policy, which results in extremely sample inefﬁcient training.
As such, ASP has only been shown to work beyond toy domains in large-scale experiments from OpenAI that required over 30B training steps to learn effective exploration policies.
In this paper, we propose an alternate approach to singleagent self-play that does not rely on Monte-Carlo rollouts for reward computation.
Our key insight is to leverage previous data collected by Alice and Bob to set goals instead of relying on Monte-Carlo rollouts of the goal-generating agent.
In our formulation, Alice and Bob are symmetric goal-conditioned agents, and they receive goals from an independent goal-setter.
The goal-setter is rewarded for setting goals for Alice that are hard for Bob to achieve and vice versa.
To formalize this idea, we introduce a new objective function for self-play based on the symmetric KL divergence between Alice and Bob’s goal distributions.
To summarize, we present our contributions below:
We introduce SelfPayer a new algorithm for singleagent self-play that instantiates two copies of the same agent, Alice and Bob, and samples challenging goals in hindsight for each agent based on the agent’s prior experience.
SelfPlayer leverages a new objective function for reward-free exploration that is expressed in terms of the symmetric KL divergence between the visitation distributions of different copies of the agent.
We demonstrate that this formulation explores Atari
games
more efﬁciently than leading exploration
ap proaches
for Atari such as Curiosity
(Burda et al., 2019)
and GoExplore
Ecoffet
et al., 2020)
by evaluating performance on the efﬁcient Atari
Atari100k) benchmark
(Kaiser et al., 2020)
2. Related Work
Exploration through Intrinsic Motivation: In recent years, a number of works in RL have aimed to answer this question.
The most common approaches to exploration in
RL have been motivated by the psychological concept of intrinsic motivation (Oudeyer & Kaplan, 2009) wherein an agent seeks to maximize a task-agnostic and hence intrinsic reward function that, if maximized, will help the agent explore its environment more effectively.
Proposals for intrinsic reward heuristics have included state-based counting (Bellemare et al., 2016; Machado et al., 2020), prediction error (Pathak et al., 2017; Burda et al., 2019), uncertainty (Osband et al., 2016; Pathak et al., 2019), and mutual information-based option discovery (Eysenbach et al., 2019; Achiam et al., 2018; Sharma et al., 2020) to name a few.
While intrinsic motivation approaches enable agents to autonomously discover short-horizon skills such as jumping, hopping, and ﬂipping, a limitation is that, in practice, they fail to discover long-horizon skills.
This is particularly evident in hard exploration games like Montezuma’s
Revenge where exploration schemes that rely solely on intrinsic motivation fail to discover far apart rooms (Burda et al., 2019).
Goal-conditioned Exploration: A less frequently employed approach to exploration that overcomes the limitations of intrinsic motivation is exploring through automatic goal discovery.
The automatic goal discovery family of approaches trains a goal-conditioned agent jointly with a goal-setting policy that sets incrementally harder goals over time.
Automatically increasing the goal difﬁculty induces a curriculum and allows the goal-conditioned agent to explore over long-horizons and discover far apart states.
Examples of automatic goal discovery algorithms are GoExplore (Ecoffet et al., 2020), which utilizes a hand-crafted state counting heuristic to set goals, goal generating algorithms (Florensa et al., 2018; Campero et al., 2021; Zhang et al., 2020), and self-play approaches (Sukhbaatar et al., 2018;
OpenAI et al., 2021), which pit two or more agents against each other to compete in a way that induces an automatic goal-reaching curriculum.
Reinforcement Learning with Self-Play: Self-play has been a core feature of several advances in supervised Reinforcement Learning, particularly in two-player game algorithms such as AlphaZero (Silver et al., 2017a) and large-scale multiplayer game algorithms such as AlphaStar (Vinyals et al., 2019) and
Dota
(Berner et al., 2019).
While a key component of many leading multi-player supervised
RL algorithms, self-play has not been adopted to singleagent settings until recently when Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2018) provided a framework for casting single-agent exploration as a multi-agent problem.
In ASP, two copies of the same agent - Alice and Bobcompete against each other to achieve progressively harder goals.
Alice is a state-conditioned agent that is rewarded for ﬁnding paths that are hard for Bob to copy.
Bob is a state and goal-conditioned agent that aims to copy the goals achieved during Alice’s rollout.
A recent extension of ASP (OpenAI et al., 2021) demonstrated that it can scale to simulated robotics manipulation settings but requires enormous compute resources that are only available to a small handful of industrial labs.
To date, no self-supervised self-play exploration approaches have been demonstrated in the Atari setting.
3. Method 3.1. SelfPlayer: Data-Efﬁcient Single-Agent Self-Play
In this paper, we consider the standard setting for goalconditioned RL where an agent acts within a Markov Decision Process, deﬁned as the tuple ( O , G , A , P, γ ) , with the following components: observations o ∈ O =
R n , goals s ∈ G = R n , actions a ∈ A , and state transition distribution, P = P ( o (cid:48) , r | o, a ) , which deﬁnes the task mechanics and rewards.
Actions can either be discrete ( A = J ) or continuous ( A = R m ), though we evaluate on the Atari benchmark (Bellemare et al., 2013) where the action space is discrete.
Without prior knowledge of P , the RL agent’s goal is to use experience to maximize expected rewards,
R = (cid:80) ∞ t =0 γ t r t , under discount factor γ ∈ [0 , 1) .
In Atari the agent receives image-based observations which are a high-dimensional indirect representations of the underlying state.
The above describes the general RL problem setting.
Since this work is concerned with self-supervised exploration, we assume that we cannot access the extrinsic reward function and must explore given only extrinsic reward-independent information in the MDP.
The MDP must still have a reward function but the reward is computed with information intrinsic to the agent such as achieved observations, goals, or actions.
Additionally, in this work goals are deﬁned as embeddings of observations.
Let z ∈
Z be a latent space of image embeddings and G = Z .
Our MDP then consists of the following tuple ( O , Z , A , R unsup , P, γ ) .
To enable self-play in a single agent setting, we instantiate two copies of the agent Alice and Bob similar to the setup in ASP (OpenAI et al., 2021).
The primary aim of self-play approaches is to induce a curriculum that allows the agent to achieve incrementally harder goals at each iteration.
We wish to set goals for Alice and Bob that are possible to achieve yet challenging in order to induce this curriculum.
Given the two agents Alice A and Bob B , a goal g , a goalreaching policy π ( j ) g where j = A, B and a success function
F ( π ( j ) g , g ) = { 0 , 1 } that determines whether Alice or Bob’s policy π ( j ) g can successfully achieve a goal g , we wish to maximize the following reward function for each agent:
[Equation]
In equation 1, Alice is rewarded if she can reach a goal that
Bob cannot and vice versa.
To motivate this objective, we ﬁrst study its properties and show that the Nash equilibrium according to eq.
1 corresponds to a fully explored MDP.
Our primary contribution is not the use of a self-play objective but rather a new approach to approximating it which we describe later in this section.
Deﬁnition 1.
The MDP is considered fully explored if all states of the MDP have been visited, meaning (cid:54) ∃ s ∈ S such that s ∈ D where D is the replay buffer of the exploration agent.
Deﬁnition 2.
Given players i = 1 , . . .
, N and value functions for a particular agent’s policy v iπ , a game is considered to be in a Nash equilibrium if all agents’ value functions are optimal v iπ = v iπ ∗ ∀ i .
This deﬁnition is taken from (P´erolat et al., 2017).
Theorem 1.
If Alice and Bob are in a Nash equilibrium according to equation 1, then the environment has been fully explored.
Proof.
We provide a full proof in the Appendix and provide a sketch of the proof here.
We prove this theorem by contradiction.
Suppose that Alice and Bob are in Nash equilibrium but the MDP is not fully explored.
We can show that there is always a goal ∃ ˆ g ∈ G that Alice can reach but Bob cannot (or vice versa).
Therefore, Alice’s value function v Aπ upon selecting ˆ g for Bob is greater than her current value function, but Alice is in a Nash equilibrium meaning her current value function is optimal.
This is a contradiction.
Therefore the
MDP must be fully explored.
We’ve shown that the Nash equilibrium of eq.
1 corresponds to a fully explored MDP suggesting that this is an appropriate objective for exploration.
We now focus on how to optimize this objective efﬁciently.
In this work, rather than relying on Monte-Carlo rollouts, we instead rely on Alice and Bob’s memory to sample goals from each agent’s past that are hard for the other agent to achieve.
Given Alice and
Bob’s replay buffers D A and D B
we introduce a goal-setter that provides goals for Alice and Bob conditioned on their replay buffers.
The goal-setter is rewarded for setting goals for Alice that are hard for Alice to achieve but easy for Bob z z
Step 1. Explore and store goals in memory
Step 2.
Sample hard but achievable goals
[Figure]
Alice
Bob a ∼ π e ( a | o ) a ∼ π e ( a | o )
[Figure]
and vice versa, hence the goal-setter’s total reward after one rollout for Alice and Bob is R = (cid:80)
i R i
=
R A + R B .
But how do we determine Alice and Bob’s success functions F ( π A , g ) and F ( π B , g ) without relying on estimates from Monte-Carlo rollouts?
Our main insight is that goals that were frequently achieved in the past are likely to be achieved again and goals that were rarely achieved or not achieved at all in the past are likely to be challenging for the agent’s current policy.
This enables us to approximate the self-play reward in terms of distributions over Alice and Bob’s past visitations.
Speciﬁcally, we ﬁt distributions over Alice and Bob’s visitation histories p
A ≡ p ( g |D A ) and p B ≡ p ( g |D B ) .
Therefore, the expected log likelihood of an agent’s j = A, B success is E g ∼ p
j
[log p j ] .
We can then express the new cumulative reward function in terms of the p j = A,B distributions.
[Equation]
which is the symmetric KL divergence between p A and p B .
In this framework, Alice and Bob are normal goalconditioned agents who aim to achieve goals given by a goal-setter.
3.2. Relation to Mutual Information and Entropy
The objective eq. 3 can be expressed in terms of Alice and
BoB’s entropy and mutual information.
[Equation]
The goal-setter is therefore trying to set goals that maximize
Alice and Bob’s individual entropy while minimizing the mutual information between Alice and Bob’s achieved goal distributions.
From eq.
6 it would appear that Alice and Bob will eventually drift off into different mutually exclusive parts of the MDP’s state space, but this is not the case.
When the goal-setter samples goals for Alice those goals are sampled from Bob’s replay buffer and vice versa.
For this reason, each time a goal is set for Alice, the mutual information between Alice’s achieved goal distribution and
Bob’s achieved goal distribution increases.
The goal-setter’s objective is thus to sample goals that will minimally increase
I ( p A , p B ) while maximally increasing H ( p A ) and H ( p B ) .
Practical Implementation
Sampling goals with self-play: We observe that since Alice and Bob are goal-conditioned agents, the goal-setter is the only part of the algorithm that is responsible for maximizing the self-play objective.
Moreover, the goal-setter’s
MDP only has one timestep since it sets a goal and observes its total reward in one step.
This means that we can directly maximize the self-play objective by sampling goals according to eq. 3.
The goal-setter thus functions as a critic.
In our practical implementation, we store discrete representations
Large tunnel maze Four room maze
Random GoExplore SelfPlayer
Random
GoExplore
SelfPlayer z of the input observations o as goals in D j , ﬁt categorical visitation probability distributions to D j where j = A, B , and score the goals in Alice and Bob’s replay buffers according to eq.
3 for each agent.
We then sample goals from
D j according to the scoring function.
The full method is shown in Algorithm 1.
Reaching goal states and exploring: There are a number of ways to reach goal states once a goal is sampled for Alice and Bob.
If given a simulator, the goal-reaching policy can be a reset function that resets the agent at its desired goal.
Since Atari is a simulated environment, this is the strategy used to report the main results in GoExplore (Ecoffet et al., 2020) and we do so as well.
A more general approach for goal reaching is to dispatch a goal-conditioned policy that starts at the current observed state and travels to the desired goal.
Prior works have explored how to craft such goal-reaching policies in the context of Atari by leveraging previous goal-reaching sequences as demonstrations (Hester et al., 2018;
Guo et al., 2020; Ecoffet et al., 2020).
We provide an implementation of an imitation policy in our code and visualizations of goal-reaching rollouts in the supplementary material.
Finally, for exploration we use the same stochastic sticky-action policy used in GoExplore (Ecoffet et al., 2020).
We also tried more sophisticated exploration policies such as Curiosity but found that random exploration with sticky actions performed better (see appendix).
Encoding Visual Observations: Our method relies on the existence of discrete representations of the underlying goal space in order to ﬁt a categorical distribution for sampling goals.
Since we evaluate on the Atari benchmark where observations and goals are pixel images, we wish to encode images into discrete latent codes.
In prior work (Ecoffet et al., 2020) it was observed that downsampling the input pixel image from (64 , 64 , 3) → (11 , 8 , 1) dimensions with numerical values discretized to 8 values is sufﬁcient for
Atari.
Another, more general approach, is to learn a discrete representation space with a Vector Quantized Variational
Autoencoder (VQVAE) (van den Oord et al., 2018).
In this work, we examine self-play with both discrete codes generated through downsampling as well as those learned with VQVAE, opting for the VQVAE2 (Razavi et al., 2019) due to its superior performance on vision tasks than its predecessor.
We run pixel downsampling experiments on
CPUs and VQVAE2 experiments on GeForce RTX Nvidia
GPUs (see appendix for details and hyperparameters).
3.4. Motivational Experiment: Exploring in PointMaze
We motivate SelfPlayer by investigating its performance in a simple gridworld environment that contains a pointmass agent in a maze (PointMaze).
We choose this setting because mazes are closed and hence the underlying MDPs are ﬁnite meaning that maze coverage directly measures the exploration capabilities of the agent.
We compare SelfPlayer to a random exploration policy and a count-based one (GoExplore) in the data-efﬁcient regime by providing only 10k exploration steps.
To score Alice and Bob’s replay buffers according to eq. 3, we ﬁt a categorical distribution to the discrete gridworld states.
We ﬁnd that on large
Tunnel maze, SelfPlayer, GoExplore, and a random policy achieve 35% , 21% and 4% coverage respectively when evaluated over 10 seeds suggesting that in a simple environment
SelfPlayer explores more efﬁciently.
We show qualitative visitation heatmaps in Fig.
Given j = A, B , initialize replay buffers D j , goal reaching polices π g ( a | o, g ) , exploration policies π e ( a | o ) , and goal probability distribu- tions p j ( ·|D j ) .
2: for N epochs do 3:
} do 4: for K trajectories do 5:
Sample categorical distri bution
Bob’s scoring function eq π g g . 6:
Collect exploration rollouts with a
[Table]
π e → ( o t , a t , o t +1 , g t ) length h
Note g t =
z t f ( o t ) where f is an encoder function and z ∈ Z is a latent representation.
D j ← D j ∪ ( o t , a t , o t +
, g t ) 8: update: goal distributions p j ((
D and encoder f
4. Experimental setup and baselines
Environments In this work, we focus on efﬁcient exploration of MDPs.
The Atari arcade environment (Bellemare et al., 2013) is perhaps the most common benchmark for evaluating the performance of RL agents.
Recently, the efﬁcient Atari benchmark (Atari100k) (Kaiser et al., 2020) was introduced in an effort to develop more sample efﬁcient supervised RL algorithms.
Atari100k evaluates performance after 100k agent steps or equivalently 400k frames of game- play since the baselines in this benchmark use an action repeat of 4 (Kaiser et al., 2020).
Evaluation on the Atari100k benchmark has dramatically improved the sample-efﬁciency of pixel-based supervised RL algorithms (Kaiser et al., 2020;
Srinivas et al., 2020; Yarats et al., 2021; Schwarzer et al., 2021).
We utilize the same benchmark to evaluate the data efﬁciency of unsupervised (reward-free) exploration agents and evaluate performance after 400k frames of gameplay for each method.
For SelfPlayer this means that Alice and
Bob individually interact with the environment 200k times for a total of 400k frames.
We evaluate performance on the 48 games used in Curiosity (Burda et al., 2019).
Baselines For baselines we compare to two leading selfsupervised exploration algorithms for Atari – GoExplore (Ecoffet et al., 2020) and Curiosity (Pathak et al., 2017;
Burda et al., 2019).
Curiosity: Curiosity learns a dynamics model ˆ T ( s t +1 | s t , a t ) and incentives the agent to explore with an intrinsic reward that is proportional to the prediction error (cid:15) =
| ˆ s t
+1 − s t +1 | where ˆ s t
+1 ∼ ˆ T .
The dynamics model is learned from data collected from by the agent’s policy which is optimized with proximal policy optimization (PPO) (Schulman et al., 2017).
The agent therefore tends to explore parts of the state space it has not seen before.
We report performance on the
Atari100k benchmark based on data provided by the authors (Burda et al., 2019).
GoExplore: GoExplore is a leading count-based algorithm for exploration in Atari that, like SelfPlayer, is sepa rated into exploration and goal-reaching phases.
During the exploration phase GoExplore stores discrete representations of the input images in a replay buffer and uses a hand-designed count-based score function to sample goal states from the replay buffer.
During the goal-reaching phase, a goal is sampled from the replay buffer and a goal-reaching policy is used to reach the goal state.
GoExplore has two variantsone with human-speciﬁed features to deﬁne states and one operating solely on image inputs.
We compare to the variant that operates solely in image input.
To generate Atari100k scores, we used the code provided by the authors (Ecoffet et al., 2020) which we ported over to our codebase with a few optimizations for faster runtimes.
Note that to make the fairest comparison to the baselines, we use the same exploration and goal-reaching strategies presented in GoExplore (Ecoffet et al., 2020) as well as the same image downsampling procedure.
The only difference is the self-play goal sampling strategy as well as the inclusion of two agents, Alice and Bob.
One drawback of
GoExplore is that the pixel downsampling procedure may not be general.
For this reason, we show in the ablations section that SelfPlayer operating on discrete VQVAE (Razavi et al., 2019) embeddings is competitive with SelfPlayer operating on downsampled pixels, which alleviates generality concerns in terms of the discrete embedding structure.
5. Results 5.1. Main results
A full list of Atari100k scores for SelfPlayer and the baselines is shown in Table 1.
In line with prior work on
Atari (Mnih et al., 2015; Kaiser et al., 2020; Hessel et al., 2017), we use Human Normalized Score (HNS) as our aggregate statistic for evaluation.
With this in mind, we provide a summary of our ﬁndings:
On the Atari100k benchmark SelfPlayer is the state-of-theart unsupervised RL algorithm.
SelfPlayer achieves the best score in 26 games, compared to 7 for the leading unsupervised baseline.
SelfPlayer has a 1.34x higher mean and 2.5x higher median HNS than the next leading unsupervised method and achieves superhuman performance on some games (Breakout, JamesBond, and WizardOfWor).
We also ﬁnd that without any access to extrinsic reward,
SelfPlayer outperforms common supervised RL algorithms such as PPO (Schulman et al., 2017) and Rainbow DQN (Hessel et al., 2017), an is competitive with the data-efﬁcient supervised model-based RL algorithm that introduced the
Atari100k benchmark (Kaiser et al., 2020) (see Fig. 6).
Finally, SelfPlayer does not necessarily rely on Atari-speciﬁc downsampled pixel representations.
SelfPlayer with discrete embeddings from a VQVAE2 (Razavi et al., 2019) trained jointly using Alice and Bob’s replay buffers (VQ-SelfPlayer) is competitive with SelfPlayer trained on downsampled pixels (see Fig. 6).
5.2. Ablations
To further investigate the SelfPlayer algorithm we perform a series of ablations to answer the following questions.
Q1:
Why does SelfPlayer achieve higher extrinsic reward than prior unsupervised RL methods?
Since neither SelfPlayer, GoExplore, or Curiosity have access to extrinsic reward, we investigate why SelfPlayer achieves higher performance in terms of extrinsic reward on Atari100k.
To do so, we compare the number of distinct discrete representations (cells) in a subset of SelfPlayer and
GoExplore runs that have identical downsampling factors.
Cell count is therefore the Atari equivalent of coverage in gridworld, which we previously investigated in Fig.
Cell count plots shown in Fig.
4 show that, as with the gridworld, SelfPayer discovers substantially more cells than
GoExplore which means it has discovered more states in the underlying MDP.
Since Atari games are designed to reward human player for advancing further in the game, the higher scores for SelfPlayer are likely a result of more efﬁcient
[Figure] [Figure]
exploration.
Q2: How does SelfPlayer compare with superivsed RL on Atari100k?
Given the efﬁciency gains achieved by SelfPlayer compared to unsupervised exploration algorithms, we investigate whether SelfPlayer is competitive with supervised RL approaches.
To do so, we compare to the supervised RL baselines presented the original Atari100k benchmark paper (Kaiser et al., 2020) which introduced a data-efﬁcient supervised model-based RL algorithm (SimPLe).
The supervised baselines include PPO (Schulman et al., 2017), Rainbow
DQN (Hessel et al., 2017), and SimPLe.
We extract the scores from all the across the 36 games presented in (Kaiser
[Figure]
et al., 2020) for each baseline, compute the median HNS, and show results in Fig.
We ﬁnd that despite not having access to extrinsic reward, SelfPlayer and VQ-SelfPlayer substantially outerform PPO and Rainbow DQN and are competitive with SimPLe.
Q
: Does SelfPlayer rely on Atari speciﬁc discrete representations
A concern with downsampling pixel observations as discrete encodings of the input image for both SelfPlayer and
GoExplore is that it is an Atari-speciﬁc discretization.
To investigate whether SelfPlayer would work with more general representations we implement a variant of SelfPlayer that operates on discrete embeddings produced by a VQVAE (Razavi et al., 2019) (VQ-SelfPlayer), which is a general method for encoding images from any domain.
To ensure generality, we train the VQVAE2 online as the SelfPlayer agent on data collected by Alice and Bob during exploration and refresh the discrete encodings in the replay buffers after a ﬁxed number of rollouts.
Due to compute constraints, we randomly subsample 16 Atari games and report the median HNS in Fig.
We ﬁnd that VQ-SelfPlayer achieves a slightly higher median HNS than SelfPlayer suggesting that SelfPlayer works comparably well when using more general discrete encodings.
[Table]
Broader Impact and Limitations
The primary beneﬁts of data-efﬁcient exploration methods is enabling autonomous agents to learn skills quickly without supervision.
This could make training of robotic agents, digital assistants
[Table]
and other decision making systems more scalable than the current paradigm where a reward function is hand-designed for each task and domain.
Additionally, efﬁcient exploration could enable advances in other scientiﬁc problems with large search spaces such as molecular docking in drug discovery.
However, since exploration methods like SelfPlayer, GoExplore, and Curiosity are unconstrained a negative potential impact and limitation is the lack of safety guarantees.
For instance, deploying SelfPlayer onto a real robot could have harmful and unintended consequences for the environment and other agents since the algorithm is designed only to explore.
Subsequently the main limitation of our proposed approach SelfPlayer and other exploration methods is that they are most useful in simulated environments as opposed to real world ones.
Another limitation speciﬁc to our approach is the reliance on discrete representations, which are can be challenging to train.
In future work, it would be promising to investigate continuous representation learning approaches as well as other ways of optimizing the self-play objective.