BYOL-Explore : Exploration by Bootstrapped Prediction
Abstract
We present BYOL-Explore , a conceptually simple yet general approach forcuriosity-driven exploration in visually-complex environments.
BYOL-Explorelearns a world representation, the world dynamics, and an exploration policy alltogether by optimizing a single prediction loss in the latent space with no additionalauxiliary objective.
We show that BYOL-Explore is effective in DM-HARD-8 , achallenging partially-observable continuous-action hard-exploration benchmarkwith visually-rich 3 -D environments.
On this benchmark, we solve the majority ofthe tasks purely through augmenting the extrinsic reward with BYOL-Explore ’sintrinsic reward, whereas prior work could only get off the ground with humandemonstrations.
As further evidence of the generality of BYOL-Explore , we showthat it achieves superhuman performance on the ten hardest exploration games in
Atari while having a much simpler design than other competitive agents.
Introduction
Exploration is essential to reinforcement learning (RL) [67], especially when extrinsic rewards aresparse or hard to reach.
In rich environments, the variety of meaningful directions of explorationmakes it impractical to visit everything.
Thus, the question becomes: how can an agent determinewhich parts of the environment are interesting to explore?
One promising paradigm to addressthis challenge is curiosity-driven exploration.
It consists of (i) learning a predictive model of someinformation about the world, called a world model , and (ii) using discrepancies between predictionsof the world model and real experience to build intrinsic rewards [59, 66, 60, 34, 51, 52, 2].
An
RL agent optimizing these intrinsic rewards drives itself towards states where the world model isincorrect or imperfect, generating new trajectories on which the world model can be improved.
Inother words, the properties of the world model influence the quality of the exploration policy, whichin turn gathers new data to shape the world model itself.
Thus, it can be important not to treat learningthe world model and learning the exploratory policy as two separate problems, but instead altogetheras a single joint problem to solve.
In this paper, we present BYOL-Explore , a curiosity-driven exploration algorithm whose appealresides in its conceptual simplicity, generality, and high performance.
BYOL-Explore learns a worldmodel with a self-supervised prediction loss, and uses the same loss to train a curiosity-drivenpolicy, thus using a single learning objective to solve both the problem of building the world model’srepresentation and the curiosity-driven policy.
Our approach builds upon Bootstrap Your Own Latent( BYOL ), a latent-predictive self-supervised method which predicts an older copy of its own latentrepresentation.
This bootstrapping mechanism has already been successfully applied in computer vision [20, 56], graph representation learning [71], and representation learning in RL [24, 62].
However, the latter works focus primarily on using the world-model for representation learning in RLwhereas BYOL-Explore takes this one step further, and not only learns a versatile world model butalso uses the world model’s loss to drive exploration.
We evaluate BYOL-Explore on DM-HARD-8 [22], a suite of 8 complex first-person-view 3 -D taskswith sparse rewards.
These tasks demand efficient exploration since in order to reach the finalgoal and obtain the reward they require completing a sequence of precise, orderly interactions withthe physical objects in the environment, unlikely to happen under a vanilla random explorationstrategy (see Fig.
2 and the videos in supplementary materials).
To show the generality of ourmethod we also evaluate BYOL-Explore on the ten hardest exploration Atari games [5].
In all thesedomains, BYOL-Explore outperforms other prominent curiosity-driven exploration methods, such as
Random Network Distillation ( RND ) [8] and Intrinsic Curiosity Module ( ICM ) [51].
In DM-HARD-8 ,
BYOL-Explore achieves human-level performance in the majority of the tasks using only the extrinsicreward augmented with BYOL-Explore ’s intrinsic reward, whereas previously significant progressrequired human demonstrations [22].
Remarkably, BYOL-Explore achieves this performance usingonly a single world model and a single policy network concurrently trained across all tasks.
Finally,as further evidence of its generality, BYOL-Explore achieves superhuman performance in the tenhardest exploration Atari games [5] while having a simpler design than other competitive agents,such as Agent57 [3, 4] and Go-Explore [14, 15].
Related Work
There is a large body of research in building world models either for planning [66, 63, 27, 26, 61],representation learning [62, 24, 41, 19] or curiosity-driven exploration [59, 68, 60, 34, 51, 52, 2, 63,21, 65].
Most works consider world models that predict the entire observations [58, 48, 16, 19],which necessitates a loss in pixel space when observations are visually complex images.
Someworks have considered predicting latent representations, whether they are random projections [7, 8],or learned representations from a separate model, such as an inverse dynamics model [51] or anauto-encoder [25, 7].
Finally, some RL works [61] have focused on predicting lower-dimensionalquantities such as the extrinsic reward, the action-selection policy, and the value function to build aworld model.
Our BYOL-Explore ’s world model operates in latent space and uses the same loss both for representation and intrinsic reward, simplifying and unifying representation learning and exploration.
BYOL-Explore ’s world model is derived from recent self-supervised representation learning methods [20, 56, 55, 71] and is similar to the ones in self-supervised RL [62, 24].
These previous worksfocused on the benefit of shaping representations for policy learning and have not looked into exploration.
We build on this previous work to show that we can take the impact of a good representationtechnique further and use it to drive exploration.
While our approach belongs to the curiosity-driven exploration paradigm [50, 42, 49, 59, 5, 68, 60,34, 51, 52, 2, 63], other exploration paradigms have also been proposed.
The maximum entropyparadigms try to steer the agent to a desired distribution of states (or state-action pairs) that maximizesthe entropy of visited states [29, 69, 70, 23].
The goal-conditioned paradigm has the agent set itsown goal drive exploration [57, 1, 17, 75, 47, 12, 82, 28, 15, 54, 80, 53].
The reward-free explorationparadigm consists of training an agent to explore the environment such that it would be able toproduce a near-optimal policy for any possible reward function [37, 39, 45, 78, 74, 9, 79, 81].
Method
Our agent has three components: a self-supervised latent-predictive world-model called
BYOL-Explore , a generic reward normalization and prioritization scheme, and an off-the-shelf
RL agent that can optionally share its own representation with BYOL-Explore ’s world model.
3.1 Background and Notation
We consider a discrete-time interaction process [44, 35, 36, 13] between an agent and its environmentwhere, at each time step t ∈ N , the agent receives an observation o t ∈ O and generates an actiona t ∈ A .
We consider an environment with stochastic dynamics p : H ×
A → ∆ O 2 that maps ahistory of past observations-actions and a current action to a probability distribution over futureobservations.
More precisely, the space of past observations-actions is H =
(cid:83) t ∈
N H t where H
=
Oand ∀ t ∈ N ∗ , H t +1 =
H t × A × O .
We consider policies π : H → ∆ A that maps a history of pastobservations-actions to a probability distribution over actions.
Finally, an extrinsic reward functionr e : H ×
A → R maps a history of past observations-actions to a real number.
3.2 Latent-Predictive World Model
BYOL-Explore world model is a multi-step predictive world model operating at the latent level.
It isinspired by the self-supervised learning method BYOL in computer vision and adapted to interactiveenvironments (see Section 3.1).
Similar to BYOL , BYOL-Explore model trains an online networkusing targets generated by an exponential moving average (EMA) target network.
However, BYOLobtains its targets by applying different augmentations to the same observation as the online representation, whereas BYOL-Explore model gets its targets from future observations processed by an
EMA of the online network, with no hand-crafted augmentation.
Also BYOL-Explore model, usesa recurrent neural network (RNN) [33, 11] to build the agent state, i.e., the state of RNN, from thehistory of observations, whereas the original BYOL only uses a feed-forward network for encoding theobservations.
In the remainder of this section, we will explain: (i) how the online network buildsfuture predictions, (ii) how targets for our predictions are obtained through a target network, (iii) theloss used to train the online network, and (iv) how we compute the uncertainties of the world model.
[Figure]
(i) Future Predictions.
The online network is composed of an encoder f
θ that transforms anobservation o t into an observation-representation f θ ( o t ) ∈ R N , where N ∈ N ∗ is the embedding2 We write ∆ Y the set of probability distributions over a set Y . size.
The observation-representation f θ ( o t ) is then fed alongside the previous action a t − 1 to a
RNN cell h
cθ that is referred as the close-loop RNN cell.
It computes a representation b t ∈ R M ofthe history h t ∈ H t seen so far as b t
= h cθ ( b t − 1 , a t − 1 , f θ ( o t )) , where M ∈ N ∗ is the size of thehistory-representation.
Then, the history-representation b t is used to initialize an open-loop RNNcell h oθ that outputs open-loop representations ( b t,k ∈ R M ) K − 1 k =1 as b t,k = h oθ ( b t,k − 1 , a t + k − 1 ) whereb t, 0 = b t and K is the open-loop horizon.
The role of the open-loop RNN cell is to simulate futurehistory-representations while observing only the future actions.
Finally, the open-loop representationb t,k is fed to a predictor g θ to output the open-loop prediction g θ ( b t,k ) ∈ R N at time t + k that playsthe role of our future prediction at time t + k .(ii)
Targets and Target Network.
The target network is an observation encoder f ϕ
whose parameters are an EMA of the online network’s parameters θ .
It outputs targets f ϕ ( o t +
k ) ∈
R N that areused to train the online network.
After each training step, the target network’s weights are updatedvia an EMA update ϕ ← αϕ + (1 − α ) θ where α is the target network EMA parameter.
A sketch ofthe neural architecture is provided in Fig.
1, with more details in App.
A.1.(iii)
Online Network Loss Function.
Suppose our RL agent collected a batch of trajectories 119 (cid:16) ( o j t , a j t ) T − 1 t =0 (cid:17) B − 1 j =0 , where T ∈ N ∗ is the trajectory length and B ∈ N ∗ is the batch size.
Then, theloss L BYOL-Explore ( θ ) to minimize is defined as the average cosine distance between the open-loopfuture predictions g θ ( b jt,k ) and their respective targets f ϕ ( o jt + k ) at time t + k :
[Equation]
where K ( t ) =
min( K, T − 1 − t ) is the valid open-loop horizon for a trajectory of length T and sgis the stop-gradient operator.(iv
) World Model Uncertainties The uncertainty associated to the transition ( o jt , a jt , o
jt
+
) is
the
sum of the corresponding prediction
losses
[Equation]
where 0 ≤ p ≤ T − 2 , 1 ≤ q ≤ K and 0 ≤ t ≤ T − 2 .
This accumulates all the losses correspondingto the world-model uncertainties relative to the observation o jt +1 .
Thus, a timestep receives intrinsicreward based on how difficult its observation was to predict from past partial histories.
Intuition on why BYOL-Explore learns a meaningful representation.
The intuition behind
BYOL-Explore is similar in spirit to the one behind BYOL .
In early training, the target network isinitialized randomly, and so BYOL-Explore ’s online network and the closed-loop RNN are trainedto predict random features of the future.
This encourages the online observation representation tocapture information that is useful to predict the future.
This information is then distilled into thetarget observation encoder network through the EMA slow copy mechanism.
In turn, these featuresbecome targets for the online network and predicting them can further improve the quality of theonline representation.
For further theoretical and empirical insights on why the bootstrap latentmethods learn non-trivial representations see, e.g., [72, 76].
3.3 Reward Normalization and Prioritization Scheme
Reward Normalization.
We use the world model uncertainties ℓ jt as an intrinsic reward.
To counterthe non-stationarity of the uncertainties during training, we adopt the same reward normalizationscheme as RND [8] and divide the raw rewards (( ℓ jt )
T − 2 t =0 )
B − 1 j =0 by an EMA estimate of their standarddeviation σ r .
The normalized rewards are ℓ jt /σ r .
Details are provided in App.
A.1.3.
Reward Prioritization.
In addition to normalizing the rewards, we can optionally prioritize themby optimizing only the rewards with highest uncertainties and nullifying rewards with the lowestuncertainties.
Because of the transient nature of the intrinsic rewards, this allows the agent to focusfirst on parts of the environment where the model is not accurate.
Later on, if the previously nullifiedrewards remain, they will naturally become the ones with highest uncertainties and be optimized.
Thismechanism allows the agent to optimize only the source of high uncertainties and not optimize allsources of uncertainties at once.
To do so, let us denote by µ ℓ/σ r the adjusted EMA mean relative tothe successive batch of normalized rewards (( ℓ jt /σ r ) T − 2 t =0 )
B − 1 j =0 .
We use µ ℓ/σ r as a clipping thresholdseparating high and low-uncertainty rewards.
Then, the clipped and normalized reward that plays therole of intrinsic reward is: r ji,t = max( ℓ jt /σ r − µ ℓ/σ r , 0) ·
3.4 Generic RL Algorithm and Representation Sharing
BYOL-Explore can be used in conjunction with any RL algorithm for training the policy.
In additionto providing an intrinsic reward, BYOL-Explore can further be used to shape the representation learntby the RL agent by directly sharing some components of the BYOL-Explore world model with the
RL model.
For instance, consider a recurrent agent composed of an encoder f ψ , an RNN cell h cψ , apolicy head π ψ and a value head v ψ that are shaped by an RL loss.
Then, we can share the weights θof the BYOL-Explore world model and the weights ψ of the RL model at the level of the encoderand the RNN cell: f ψ = f θ and h cθ =
h cψ and let the joint representation be trained via both the RLloss and BYOL-Explore .
In our experiments, we will show results for both the shared and unsharedsettings.
Architectural details are provided in Appendix A.1.
Experiments
We evaluate the algorithms on benchmark task-suites known to contain hard exploration challenges.
These benchmarks have different properties in terms of the complexity of the observations, partialobservability, and procedural generation, allowing us to test the generality of our approach.
Atari Learning Environment [6].
This is a widely used RL benchmark, comprising of approximately 50 Atari games.
These are 2 -D, fully-observable, (fairly) deterministic environments formost of the games but have a very long optimization horizon (episodes last for an average of 10000steps) and complex observations (preprocessed greyscale images which are 84 × 84 byte arrays).
Weselect the 10 hardest exploration games [5] to conduct our experiments: Alien , Freeway , Gravitar ,
Hero , Montezuma’s Revenge , Pitfall , Private Eye , Qbert , Solaris and Venture .
Hard-Eight Suite [22].
This benchmark comprises of 8 hard exploration tasks, originally builtto emphasize the difficulties encountered by an RL agent when learning from sparse rewards in aprocedurally-generated 3 -D world with partial observability, continuous control, and highly variableinitial conditions.
Each task requires the agent to interact with specific objects in its environmentin order to reach a large apple that provides reward (see Fig. 2).
Being procedurally-generated,properties such as object shapes, colors, and positions are different every episode.
We provide videosin the supplementary materials to ground the difficulty of these tasks.
Note that the current best RLagents that solve these tasks require a small (but non-zero) amount of human expert demonstrations.
Without demonstrations or reward shaping, state-of-the-art deep RL algorithms, such as R2D2 [38],do not get positive reward signal on any of the tasks.
In our case, we train a single RL agent and asingle world model to tackle the 8 tasks all-together, making for a challenging multi-task setting.
4.1 Experimental Setup
At a high level, BYOL-Explore has 4 main hyper-parameters: the target network EMA parameter α ,the open-loop horizon K , choosing to clip rewards and to share the BYOL-Explore representationwith the RL network.
To better understand what part of BYOL-Explore is essential to perform well,we run 4 ablations.
Each ablation corresponds to BYOL-Explore where only one hyper-parameter
[Figure]
set the addition
RND (see App
A details) an
Finally, we run experiments on two different evaluation regimes.
The first regime uses a mixedreward function r t = r e,t
+ λr i,t which is a linear combination of the normalized extrinsic rewardsr e,t and intrinsic rewards computed by the agent r i,t with mixing parameter λ .
This may be the mostimportant regime for a practitioner as we can see if our intrinsic rewards help improve performance,with respect to the extrinsic rewards, compared to the pure RL agent.
The second regime is fullyself-supervised where only the intrinsic reward r i,t is optimized.
This regime gives us a sense of howpure exploration methods perform in complex environments.
Choice of RL algorithm.
We use VMPO [64] as our RL algorithm.
VMPO is an efficient on-policyoptimization method that has achieved strong results across both discrete and continuous controltasks, and is thus applicable to all of the domains we consider.
Further details regarding the RLalgorithm setup and hyperparameters are provided in Appendix A.3.
Performance Metrics.
We evaluate performance in terms of the agent score at a number ofobservations/frames t , Agent score ( t ) , as measured by undiscounted episode return.
The number offrames t corresponds to all the frames generated by all the actors by interacting with the environment,even the skipped ones.
Frames/observations can be skipped if there is an action repeat which is thecase in Atari where the action repeat is of 4 .
We define the highest agent score through training as Agent score =
max t Agent score ( t ) , as donein [18, 3].
We define, for each game, the Human Normalized Score ( HNS ) at number of frame t : HNS ( t ) =
Agent score ( t ) − Random score Human score − Random score as well as the HNS over the whole training: HNS = max t HNS ( t ) .
A HNS higher than 1 means superhuman performance on a specific task.
We similarly define the CHNS Score as HNS clipped between 0 and 1 .
4.2 Atari Results
In these experiments, we set the target EMA rate α = 0 .
99 and open-loop horizon K = 8 .
Weuse λ = 0 .
1 to combine the intrinsic and extrinsic rewards.
We follow the classical 30 randomno-ops evaluation regime [46, 73], and average performance over 10 episodes and over 3 seeds. Thisevaluation regime does not use sticky actions [43].
Fig.
3 (left) shows that BYOL-Explore is almost superhuman on the 10 -hardest exploration games andoutperforms the different baselines of RND , ICM , and pure RL.
Fig.
3 (right) compares BYOL-Explore against its ablations to gain finer insights into our method.
The No clipping ablation performscomparably, showing that the prioritization of intrinsic rewards is not necessary on Atari tasks.
Similarly, the Horizon=1 ablation performs slightly better, indicating that simply predicting one-steplatents is sufficient to explore efficiently on the fully-observable Atari tasks.
The Fixed Targetsablation performs much worse, showing that our approach of predicting learned targets (rather thanfixed random projections) is vital for good performance.
It is also worth noting that all the ablationsexcept Fixed Targets outperform all of our baselines, demonstrating the robustness of our approach.
Finally, because the Horizon=1 ablation was close to superhuman on Atari , we run the sameconfiguration but double the length of the sequences on which we train from 64 to 128 (also doublingmemory requirements while learning).
With this small adjustment, this agent ( BYOL-Explore (big))becomes superhuman on all of the 10 -hardest exploration games.
Purely intrinsic exploration.
To test how
BYOL-Explore behaves when only given intrinsic rewards without any extrinsic signal, we test on thewell-known Montezuma’s Revenge game by settingλ = 0 .
We measure exploratory behavior in terms ofthe number of different rooms of the dungeon the agentis able to explore over its lifetime.
Note that accessinglater rooms requires navigating complex dynamics suchas collecting keys to open doors, avoiding enemies,and carefully traversing rooms filled with traps such astimed lasers.
Figure 4 shows how much room coverageis achieved during training when no extrinsic rewardis used, showing that BYOL-Explore explores furtherthan the best result reported by RND [8].
Importantly, weuse the episodic setting for intrinsic rewards whereasthe published RND results considers the non-episodicsetting for intrinsic rewards — facilitating explorationas the agent is less risk-averse.
Therefore, our settingcould be considered even more challenging.
Our agentexplores more than 20 rooms on average versus 17 with best published RND results.
As expected inthe episodic setting, our RND re-implementation visits even fewer rooms.
However, we can reproducethe published RND results in the episodic setting when using recurrent policies.
Further results.
More fine-grained results are reported in App A.4.1.
We report, in Fig.10 andin Fig.11, the agent scores learning curves for each game.
Tab. 1 and Tab. 2 have agent score atthe end of training.
Finally, Tab. 3 and Tab. 4 show the mean CHNS and different statistics (mean and percentiles) of the HNS across the selected games.
An interesting finding from examining the HNS is that clipping and longer-horizon predictions are critical for very high scores on some gamessuch as Montezuma’s Revenge or Hero .
BYOL-Explore has a median HNS of 331 .
compared tothe No-clipping ablation and the Horizon=1 which have a median HNS of only 181 . 39 and 199 .
80respectively.
Therefore, while clipping is not necessary to get to human-level performance, it is stillcrucial to achieve top performance.
We also provide further results regarding the pure explorationsetting on all 10 games in App.
A.4.2.
4.3 DM-HARD-8 Results
In these experiments, we set the target EMA rate α = 0 .
99 and open-loop horizon K = 10 .
We useλ = 0 .
01 to combine the intrinsic and extrinsic rewards.
In contrast to prior work [22], we performexperiments in the more challenging multi-task regime, training a single agent to solve all eight tasks.
At the beginning of each episode, a task is drawn uniformly at random from the suite.
In Fig.
5 (left) we report the mean CHNS ( t ) across the tasks, averaged over 3 seeds.
We see that
BYOL-Explore outperforms the baselines of RND , ICM , and pure RL by a large margin.
Fig.
5 (right)compares the performance of BYOL-Explore to its various ablations.
Note that the No-clippingablation performs similarly to BYOL-Explore in terms of CHNS .
However, unlike the fully-observable
Atari tasks, the Horizon=1 ablation learns considerably slower and achieves lower final performance(see also our extended ablations on the horizon length in Fig.
15 in App. A.4.4).
We note onceagain that the BYOL-Explore bootstrapping mechanism for learning representations is essential, asconfirmed by the poor performance of the Fixed-targets ablation.
Due to computational limitations,we did not run the No Sharing ablation, as using separate networks requires twice the memory.
We now analyze our method more closely by examining per-task performance.
The full learningcurves for each task can be found in Fig.
6 for BYOL-Explore and the main baselines and in
Appendix A.4.4 (see Fig. 14) for the various ablations.
First, we take note that other curiosity-drivenmethods ( ICM and RND ) cannot get any positive score on the majority of the DM-HARD-8 tasks, evenwith additional hyperparameter tuning and reward prioritizing (see Fig. 17 and Fig. 18 in App. A.4.4).
In contrast, we see that BYOL-Explore achieves strong performance on five out of the eight hardexploration tasks.
Importantly, BYOL-Explore achieves this without human demonstrations, whichwas not the case in prior work [22].
BYOL-Explore even surpasses humans on 4 tasks, namely
Navigate cubes , Throw-across , Baseball , and Wall Sensors (see Tab. 9 in App.
A.4.4 fordetails).
Most impressively, BYOL-Explore can solve Throw-across , which is a challenging taskeven for a skilful human player and was not solvable in prior work without collecting additionalsuccessful human demonstrations [22].
Interestingly, note that on the Navigate Cubes task, both RND and the Fixed-targets ablation achievemaximum performance alongside BYOL-Explore .
We argue that this is because the prediction ofrandom projections (either at the same step as done by RND or multi-step as done by BYOL-Explore )leads to the policy learned performing spatial, navigational exploration — this is the kind of behaviorrequired to explore well on the Navigate Cubes task.
In contrast, the other tasks require exploratorybehavior involving interaction with objects and the use of tools, where both RND and the Fixed-targetsablation fail.
Finally, we observe that two games, namely Remember Sensor and Push Blocks ,are particularly challenging, where all of our considered methods perform poorly.
We hypothesizethat this is due to the larger variety of procedurally generated objects spawned in these levels, and theneed to remember previous cues in the environment leading to a hard credit assignment problem.
Purely intrinsic exploration.
Each of the DM-HARD-8 tasks has complex dynamics and objectinteractions, making it difficult to assess qualitatively the behavior of purely intrinsically motivatedexploration.
Nevertheless, for completeness, we provide results of BYOL-Explore trained only withintrinsic rewards in App.
A.4.4, showing that it does achieve some positive signal on the Drawbridgeand Wall Sensor tasks (see Fig. 19).
Conclusion
We showed that BYOL-Explore is a simple curiosity-driven exploration method that achieves excellent performance on hard exploration tasks with fairly deterministic dynamics.
BYOL-Explore is amulti-step prediction error method at the latent level that relies on recent advances in self-supervisedlearning to train its representation as well as its world-model without any additional loss.
In Atari ,
BYOL-Explore achieves superhuman performance on the 10 -hardest exploration games while beingof much simpler design than other superhuman agents.
Moreover, BYOL-Explore substantiallyoutperforms previous exploration methods on DM-HARD-8 navigation and manipulation tasks in a3 -D, multi-task, partially-observable and procedurally-generated environment.
This shows the generality of our algorithm to handle either 2 -D or 3 -D, single or multi-task, fully or partially-observableenvironments.
In the future, we would like to improve performance in DM-HARD-8 and to demonstrate the generalityof our method by extending it to other domains.
In DM-HARD-8 , we believe we can improveperformance by scaling up the world model and finding better ways to trade off exploration andexploitation.
Beyond DM-HARD-8 , there are opportunities to tackle further challenges, most notablyhighly-stochastic and procedurally-generated environment dynamics such as NetHack [40]. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We are not
including theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] We are not including
theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] The code is
proprietary
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We specify the main training details in the paper and we include a
full list of hyperparameters description in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We report error bars in learning curves of the agent score
for every agent we run.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We include all the information
regarding the compute in the appendix.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use the ALE and
DM-HARD-8 and we cite the creators.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We did not use crowdsourcing.
[Table]
[Table]
[Table]
[Table]
[Table]