S ELF -S UPERVISED E XPLORATION VIA L ATENT B AYESIAN S URPRISE
A BSTRACT
Training with Reinforcement Learning requires a reward function that is used to guide the agent towards achieving its objective.
However, designing smooth and well-behaved rewards is in general not trivial and requires signiﬁcant human engineering efforts.
Generating rewards in self-supervised way, by inspiring the agent with an intrinsic desire to learn and explore the environment, might induce more general behaviors.
In this work, we propose a curiosity-based bonus as intrinsic reward for Reinforcement Learning, computed as the Bayesian surprise with respect to a latent state variable, learnt by reconstructing ﬁxed random features.
We extensively evaluate our model by measuring the agent’s performance in terms of environment exploration, for continuous tasks, and looking at the game scores achieved, for video games.
Our model is computationally cheap and empirically shows state-of-the-art performance on several problems.
Furthermore, experimenting on an environment with stochastic actions, our approach emerged to be the most resilient to simple stochasticity.
Further visualization is available on the project webpage.
I NTRODUCTION
Agents can be trained with Reinforcement Learning (RL) to successfully accomplish tasks by maximising a reward signal, which should likely encourage correct behaviors and penalize wrong actions.
For instance, agents can learn to play video games by maximizing the game score (Mnih et al., 2015) or achieve robotic manipulation tasks, such as solving a Rubik’s cube (OpenAI et al., 2019), by following human-engineered rewards.
However, how to correctly deﬁne reward functions to develop general skills remains an unsolved problem (Amodei et al., 2016).
In contrast to RL agents, humans can learn behaviors without any external rewards, due to the intrinsic motivation that naturally drives them to be active and explore the environment (Larson &
Rusk, 2011). The process of skill acquisition can then be expressed as a selection process over successful exploratory behaviors (Corbetta et al., 2018).
The design of similar mechanisms for RL agents opens to the possibility of training and evaluating agents without external rewards (Matusch et al., 2020), potentially leading to more general strategies of learning.
The idea of instilling intrinsic motivation, or ‘curiosity’, into artiﬁcial agents has raised a large interest in the RL community (Oudeyer et al., 2007; Schmidhuber, 1991).
One common approach consists of learning a model of the environment that is used to generate intrinsic rewards, which can replace or complement the external reward function.
However, what is the best approach to generate such intrinsic bonuses is still unknown.
Several successful approaches modeled intrinsic rewards as the ‘surprisal’ information of the model.
In layman’s terms, this can be described as the difference between the agent’s belief about the environment state and the ground truth, and can be implemented as the model’s predictions error (Achiam & Sastry, 2017; Pathak et al., 2017).
Alternatively, Bayesian surprise (Itti & Baldi, 2006) may also be used as a bonus to incentive exploration, albeit existing work has mostly focused on computationally-expensive approaches, which evaluate surprise in the model’s parameters space (Houthooft et al., 2016).
Contributions.
In this work, we present an intrinsic bonus based on the concept of Bayesian surprise, efﬁciently computed with respect to a latent state variable in the dynamics.
The dynamics learning process is obtained by optimizing a lower bound objective that reconstructs compact features, obtained from the environment’s observations, using a ﬁxed feature model.
The main contributions of this work are as follows: ( i ) a latent dynamics model, which captures the unobserved state of the environment and allows to reconstruct observations in a compact feature space, ( ii ) a new latent Bayesian surprise exploration bonus, representing the information gain with respect to the latent state variable of the dynamics, ( iii ) evaluation of the approach on several continuous-actions robotic simulation tasks and on discrete-actions video games, and comparison with other exploration strategies ( iv ) preliminary assessment of resilience to stochasticity in actions, by comparing to baselines and ablations on an image-prediction environment.
The results empirically show that our L atent B ayesian S urprise ( LBS ) model performs on par and often outperforms state-of-the-art methods, while also being more resilient to stochasticity.
I NTRINSIC M OTIVATION FOR R EINFORCEMENT L EARNING
We focus on bonus-based methods as intrinsic motivation to incentivize exploration in RL.
To foster reader’s understanding, we ﬁrst introduce standard notation and common practices in the area.
Markov Decision Processes.
The RL setting can be formalized as a Partially Observable Markov
Decision Process (POMDP), which is denoted with the tuple M = {S , A , T, R, Ω , O , γ } , where
S is the set of unobserved states, A is the set of actions, T is the state transition function, also referred to as the dynamics of the environment, R is the reward function, which maps transitions into rewards, Ω is the set observations, O is a set of conditional observation probabilities, and γ is a discount factor.
The objective of the RL agent is to maximize the discounted sum of rewards, or return, G t = (cid:80) Tk =
t +1 γ ( k − t − 1) r k , where t indicates the time step at which the return is computed.
Policy Optimization.
In order to maximize the return function, the agent should condition its actions on the basis of the observations coming from the environment.
The policy function π ( a t | o t ) is used to represent the probability of taking action a t when observing o t .
Several policy-optimization algorithms also evaluate two value functions, V ( o t ) and Q ( o t , a t ) , that are used to estimate and predict future rewards with respect to a certain observation or observation-action pair, respectively.
Dynamics Model.
Exploration bonuses as intrinsic rewards can be generated using the surprisal of a model of the environment’s dynamics, which is formalized as p ( s t +1 | o t , a t ) .
In curiosity-driven
RL, it has been common practice to compute predictions on a compact feature space.
For instance, features φ t = f
( o t ) can be computed with the mapping f : R n −→ R m , with m ≤ n , turning the model dynamics into p ( φ t +1 |
o t , a t ) .
To distinguish it from the true state dynamics of the POMDP, we refer to this formulation as Feature Dynamics.
Intrinsic Rewards.
Intrinsic motivation is instilled into the agent by combining the generated intrinsic rewards with the external rewards of the environment.
The combined reward at time step t is represented as: r t = η e r ( e ) t + η i r ( i ) t , where r ( e ) is the external reward, r ( i ) is the intrinsic bonus, and η e and η i are factors adopted to balance external and intrinsic rewards.
How to optimally balance between exploration with intrinsic motivation and exploitation of external rewards is still an unresolved question.
To prevent this issue from affecting our experiments, we assign any external rewards to zero.
This is implemented by setting η e = 0 , focusing the agents solely on the exploratory behaviors inspired by the curiosity mechanism.
L ATENT B AYESIAN S URPRISE
Our model provides intrinsic motivation through a Bayesian surprise signal, which is computed with respect to a latent state variable.
First, we describe how the latent dynamics inference works and how our model is trained by optimizing a variational lower bound on the log-likelihood of future s t s t+ a t o t+ o t (a) POMDP Model Φ t Φ t+ a t o t+ o t (b) Feature Dynamics Model s t s t+ a t o t+ o t Φ t+ Φ t (c) LBS Model (ours) features, then explain how the intrinsic reward signal is obtained, and ﬁnally discuss the intuition behind our work and connections with other methods.
Latent Dynamics.
In a typical POMDP, the model dynamics should predict next states from current observations and actions.
This is represented in Figure 1a, where the agent, after having adopted action a t , reaches a state of the environment that is observed as o t +1 .
The dynamics model infers s t + as p ( s t +1 | o t , a t ) .
Generally, in the curiosity-driven RL literature, hidden true states of the POMDP are overlooked and the state s t is replaced by features φ t that are computed as point estimates.
In this setup, the dynamics of the environment is modeled with p ( φ t +1 | o t , a t ) , as for Figure 1b.
Using ﬁxed deterministic features for exploration, either random or pre-trained, has shown empirically better results than training features concurrently with the dynamics model (Burda et al., 2019a;b), likely because of their greater stability.
However, a Feature Dynamics model based on ﬁxed deterministic features can be dangerous for several reasons: 1) deterministic features do not allow to model uncertainty e.g. stochasticity in observations and transitions; 2) feature reduction can always generate inconsistencies and ambiguities, which may lead to ﬁt ﬁctitious transitions.
To exploit ﬁxed features, and contemporary alleviate the above problems, we introduce the latent random variable s t in the Feature Dynamics model.
As for Figure 1c, the succession of states s t is used to capture the dynamics, in a POMDP fashion, while features are treated as the result of a generation process from their respective observations.
In this setup, the environment dynamics can be modeled with respect to the latent state variable, and denoted as p ( s t +1 | o t , a t ) , and future features can be reconstructed from the latent state, as p ( φ t +1 | s t +1 ) , rather than reconstructing observations.
As illustrated in Figure 2, our LBS model is made of the following components: [Figure]
[Equation]
where the latent dynamics model represents prior beliefs over next states, q ( s t +1 ) is an approximation of the dynamics posterior with respect to the latent state variable, and the features reconstruction module allows to regenerate compact features from the corresponding latent state. [Figure] FixedFeatureModel [Figure]
All the components are jointly trained by maximizing the following variational lower bound on future features log-likelihood:
[Equation]
where β is introduced to incentivize disentanglement in the latent state representation, as in (Higgins et al., 2017).
The derivation of the objective is available in Appendix A.
Reconstructing features φ t +1 , rather than observations, has several advantages.
First, it has been demonstrated by numerous experiments that most of the information contained in the highdimensional observations may be neglectable for exploration (Burda et al., 2019a).
Thus, features reconstruction may allow faster computation, without compromising performance.
Secondly, the use of features in the reconstruction objective avoids any shortcuts in the posterior model.
Bayesian Surprise Bonus.
In information theory, Bayesian surprise is deﬁned as the information gain I obtained about a random variable by observing another random variable.
In the context of our method, we are interested in measuring the amount of information that is gained by the LBS model when facing a new environment’s transition.
After taking action a t while being in the latent state s t , the agent would receive a new observation o t +1 that completes the transition and brings newer information to the latent dynamics model.
Such information gain can be formulated as the KL divergence between the latent dynamics p ( s t +1 ) and its approximate posterior q ( s t +1 |
o t +1 )
and adopted as an intrinsic reward for RL as follows:
[Equation]
The above term can be efﬁciently computed by comparing the predictions of the latent dynamics and of the latent posterior models, in an amortized inference fashion.
Such signal should encourage the agent to collect transitions where the latent dynamics predictions are more uncertain or erroneous.
Connections.
The intuition behind LBS may be interpreted as the intersection of two known ideas in the exploration landscape: one consists of computing features using a ﬁxed feature network and then using the error prediction of a ‘distillation’ model as curiosity (Burda et al., 2019b), the other comes from the promising approach of using Bayesian surprise as an exploration signal (Houthooft et al., 2016).
The introduction of a latent random variable allows to combine the two ideas, while also allowing to model stochasticity and feature’s inadequacies in a probabilistic fashion.
In the LBS model, the latent posterior can be seen as a distillation process from next observations to next features, through the latent state.
Instead, the latent dynamics allows to compute prior beliefs, enabling the computation of Bayesian surprise in an efﬁcient way, by exploiting amortized inference.
E XPERIMENTS
The aim of the experiments is to compare the performance of our LBS model and of its latent
Bayesian surprise bonus against other approaches for self-supervised exploration in RL.
Environments.
Results are presented with respect to two sets of environments: one of continuous control tasks and one with discrete-action games.
The continuous control tasks include the classic
Mountain Car environment (Moore, 1990), the Mujoco-based Half-Cheetah environment (Todorov et al., 2012), and the Ant Maze environment used in (Shyam et al., 2019).
The discrete-action games include 8 video games from the Atari Learning Environment (ALE; Bellemare et al.
and
the
Super Mario Bros. game, which is a popular NES platform game.
For all tasks, we update the policy using the Proximal Policy Optimization algorithm (PPO; Schulman et al.
Implementation Details.
The LBS model components are implemented as follows: the latent dynamics and posterior models are deep neural networks with variational outputs, representing multivariate gaussian distributions; the feature reconstruction model is implemented as a multi-layer perceptron (MLP).
Parameter sharing is enabled between the posterior and the prior, by using the last layer of the latent dynamics, concatenated with new observations, as an input to the posterior.
To easily obtain ﬁxed features from observations, we follow the trend that empirically found random features to be working well for exploration (Burda et al., 2019a; Pathak et al., 2019; Burda et al., 2019b).
Fixed features φ t are obtained by processing observations o t with a randomly initialised network that is never updated.
This network is either represented by a single linear layer, for continuous control tasks, or by a small convolutional network, for vision-based tasks.
Further details on the implementation and the experiments can be found in Appendix B and C.
4. C ONTINUOUS C ONTROL
In our continuous control experiments, we discretize the observation-space into bins and compare the number of bins explored, in terms of coverage percentage, against the following baselines:
• Intrinsic Curiosity Model (ICM; Pathak et al.
(2017)): intrinsic rewards are computed as the mean-squared error (MSE) between network’s predictions in feature space and the true features.
Observation are processed into features using a feature network, trained jointly with the dynamics model to optimize an inverse-dynamics objective.
• Random Network Distillation (RND; Burda et al.
(2019b)): features are obtained with a ﬁxed randomly initialized network.
Intrinsic rewards for each transition are computed as the prediction errors between next-observation features and the output of a distillation network, which is trained to match the outputs of the random feature network.
• Variational Information Maximizing Exploration (VIME; Houthooft et al.
(2016)): the dynamics is modeled as a Bayesian neural network (BNN; Bishop (1997)).
Intrinsic rewards for single transitions are shaped as the information gain computed with respect to the BNN’s parameters before and after updating the network, using the transition’s values.
• Random : a naive agent that explores by performing a series of random actions.
The training curves are presented in Figure 3, averaging over runs with eight different random seeds.
Mountain Car.
In MountainCar, the limited observation space is discretized into 100 bins.
As shown in Figure 3, VIME is the best performing approach, with LBS and ICM following, slightly behind.
While the ﬁrst three models could reach around 85% of the visitable bins, RND struggles behind with about 50% of visited bins, which is still 35% better than Random exploration ( ∼ 15%).
Ant Maze.
In the Ant Maze environment, the agent can explore up to seven bins, corresponding to different aisles of a maze.
LBS and ICM perform best in this environment, eventually reaching the end of the maze in all runs.
VIME ( ∼ 95%) and RND ( ∼ 91%) perform comparably, achieving a solid result against the coverage achieved by the Random baseline ( ∼ 68%).
Half-Cheetah.
In the Half-Cheetah environment, the observation space is discretized into 100 bins.
Performance in this task see LBS reaching the highest number of bins, with around 53% of coverage.
They follow ICM ( ∼ 47%) and VIME ( ∼ 39%).
RND lacks behind by slightly outperforming the
Random baseline ( ∼ 25% vs ∼ 17%).
4. A RCADE G AMES
For the arcade games, the environments chosen are designed in a way that either requires the player to explore in order to succeed, e.g. Qbert, or to survive as long as possible to avoid boredom, e.g.
Pong.
For this reason, agents are trained only with curiosity, but evaluated on the game score they achieve in one episode, or on the distance traveled from the initial position (only Super Mario Bros.).
We follow the setup of (Burda et al., 2019a) and compare against their baselines, which use MSE prediction error in feature space as the intrinsic motivation signal.
The difference between the Variational Autoencoder, or VAE model , the Random Features, or RF model , and the Inverse Dynamics
Features, or IDF model , lies in the features adopted.
The VAE model trains an autoencoder, as in (Kingma & Welling, 2014), concurrently with the dynamics model; the RF model uses a randomly initialized network; the IDF model trains features that allows to model inverse dynamics
A comparison against the above baselines, using different features on high-dimensionality observations, should demonstrate the relevance of our latent Bayesian surprise bonus, compared to errorprediction bonuses, regardless of the features set adopted.
The training curves are presented in
Figure 4, averaging over runs with three different
Atari.
The results in the selected Atari games are favorable towards the LBS model.
It achieves the best ﬁnal score in 4 out 8 games: Pong, Seaquest, Breakout and Qbert, with a large margin for the latter twos; and performs comparably in all other games, apart from Montezuma Revenge.
Interestingly, in Breakout, the only two levels available have exactly the same game mechanics and graphics, so that when the agent breaks the last brick of Level 1, Level 2 starts exactly as a game reset, as depicted in Figure 5a.
For a curiosity-driven RL agent, this is totally disruptive as the model tends to assign low intrinsic rewards to the start of Level 2, which provides well-known observations.
[Figure]
[Figure]
[Figure]
This fact is exacerbated by the curves of the best scores, in Figure 5b, which shows that LBS was able to reach high scores, but eventually kept an average score below the Level 1’s threshold (432) 2 .
As for Montezuma Revenge, we present in Figure 6 the max scores achieved by all methods, showing that also IDF and LBS models could actually reach non-zero scores during training.
It is unclear why the ﬁnal scores of the latter got to zero, and further analysis should be conducted about this.
However, concerns about the use of Montezuma Revenge as a benchmark for exploration methods have already been raised in the past (Ta¨ıga et al., 2019), as the games mostly requires other kind of systematic approaches to reach higher scores, rather than vanilla exploration (Badia et al., 2020).
Super Mario Bros.
The ﬁnal distance reached in the Mario’s game are comparable for LBS, RF and VAE models, with ICM behind.
It may be of interest, that different runs of our models found different paths to solve World 1-1.
One model went down from the pipe at the start of the level, reaching the end by using a shortcut, while the other two ran and jumped across all the level.
4. S TOCHASTIC E NVIRONMENT
In this environment, as in Pathak et al.
(2019), we use the Noisy MNIST dataset to perform an experiment on stochastic actions.
Taking examples from the test set of MNIST, we built a ﬁctitious dynamics that always starts either from an image of a zero or a one, as in Figure 7a: a 0-image always transitions to a 1-image, while a 1-image transitions into a number between two and nine.
We assess the performance in terms of the ratio between the intrinsic motivation provided for transitions starting from 1-images and transitions starting from 0-images.
A well-behaved exploration model should eventually understand that results associated with the more stochastic 1-image transitions belong to the same category, and lose interest with respect to them.
Thus, the correct expected behavior is that the ratio should eventually lean to values close to the unity.
In Figure 7b, we compare LBS to ICM and RND and observe that LBS is the only model that eventually captures the stochasticity in the transitions starting from 1-images, eventually lowering the ratio at convergence.
We further benchmark against two ablations of LBS: LBS - means only , which takes only the means from the latent dynamics and posterior outputs, computing both the training loss and the intrinsic rewards accordingly; LBS - no latent , which gets rid of the latent state and instead uses stochastic random features with ﬁxed variance.
In the latter, the training loss and the intrinsic rewards are both collapsed into a KL divergence term between the dynamics model and the true features posterior.
As shown in Figure 7c, both the latent state and the use of a random variable as a latent are important components of LBS that help capturing stochasticity in the transitions.
R ELATED W ORK
Reinforcement Learning.
Value-based methods in RL use the Q-value function to choose the best action in discrete settings (Mnih et al., 2015; Hessel et al., 2018).
However, the Q-value approach cannot scale well to high-dimensional outputs or continuous environments.
Policy Optimization techniques solve these problems by directly optimizing the policy, by either learning online, using samples collected from the policy (Schulman et al., 2015; 2017), or ofﬂine, reusing the experience stored in a replay buffer (Lillicrap et al., 2016; Haarnoja et al., 2018).
Latent Dynamics.
In complex environments, the use of latent dynamics models has proven successful for control and long-term planning, either by using VAEs to model locally-linear latent states (Watter et al., 2015), or by using recurrent world models in POMDPs (Buesing et al., 2018; Hafner et al., 2019; 2020).
Intrinsic Motivation.
Several exploration strategies have been presented in Section 4, concerning entropy maximization in the transitions (Pathak et al., 2017), with respect to future observations (Burda et al., 2019b) or model’s parameters (Houthooft et al., 2016).
In Disagreement (Pathak et al., 2019), they train an ensemble of dynamics models and use the variance in their predictions as intrinsic motivation.
In VDM (Bai et al., 2020), they model a latent variable to reconstruct features (using a different reconstruction model from ours) and use the average surprisal over several sampled reconstructions as an exploration bonus.
Models that use alternative approaches to modelling the environment’s dynamics are based on pseudo-counts (Bellemare et al., 2016;
Ostrovski et al., 2017; Tang et al., 2017), which use density estimations techinques to explore less seen areas of the environment, Randomized Prior Functions (Osband et al., 2018), applying statistical bootstrapping and ensembles to the Q-value function model, or Noisy Nets (Fortunato et al., 2018), applying noise to the value-function network’s layers.
Planning Exploration.
Recent breakthroughs concerning exploration in RL have also focused on using the learned environment dynamics to plan to explore.
This is the case of (Shyam et al., 2019) and (Ratzlaff et al., 2020), where they use imaginary rollouts from their dynamics models to plan exploratory behaviors, and (Sekar et al., 2020), where they combine a model-based planner in latent space (Hafner et al., 2020) with the Disagreement exploration strategy (Pathak et al., 2019).
D ISCUSSION
In this work, we introduced the LBS model to generate intrinsic rewards as the Bayesian suprise over a latent state variable.
Our method has proven successful in several continuous-control and discreteaction settings, outperforming or performing on par with state-of-the-art methods for self-supervised exploration in RL.
The experiments in low-dimensional continuous-control tasks, which evaluate performance over an artiﬁcial metrics on the coverage of the observation space, have shown that our method provides more in-depth exploration than other methods.
In particular, VIME seems to shine in lowdimensional environments such as Mountain Car but is struggling with higher-dimensional observations.
RND generally performed worst in these low-dimensional experiments.
In our arcade games results, we provided additional insights about exploration in some video games, such as Breakout, Montezuma Revenge and Super Mario Bros., identifying issues to be considered for future works or highlighting interesting exploratory behaviors that emerged from our runs.
Future benchmarks using these games would likely beneﬁt from this information.
We also tested LBS to be resilient to stochasticity in the dynamics, in a simple ﬁctitious task.
We believe stochasticity is an important limitation that still affects exploration methods and future work should focus on understanding to which extent limitations apply and how to overcome them.
As our Bayesian surprise proved successful in several environments, we are also interested in applying it for planning exploration, by further exploiting our latent dynamics model.
A natural implementation of this, could for instance adopt recurrent world models (Ha & Schmidhuber, 2018) or latent overshooting (Hafner et al., 2019) to compute Bayesian surprise with respect to latent state predictions in the more distant future. A CKNOWLEDGMENTS
This research received funding from the Flemish Government (AI Research Program)
A L ATENT D YNAMICS L OWER B OUND
The LBS model’s loss is obtained by maximizing a variational lower bound (ELBO) on future features log-likelihood.
The ELBO derivation employs marginalization and Jensen’s inequality, as shown:
[Equation]
B I MPLEMENTATION D ETAILS
Mountain Car.
The agent is spawned at the bottom of a valley and can move across two hills, by adopting an adequate velocity action.
The observation space is two-dimensional, and limited to the following ranges: [-1.2, 0.6] for the position and [-0.07, 0.07] for the velocity.
We divide each range in ten equally-sized ranges and generate 100 bins from their possible combinations.
Ant Maze.
A robotic ant should navigate a ’C-shaped’ maze, where 7 visitable bins have been identiﬁed by heavily discretizing the observation space (Shyam et al., 2019).
Half-Cheetah.
Tasks using the HalfCheetah environment usually concern training a bipedal walker robot to learn skills such as walking, running, ﬂipping backward or forward.
For this reason, we chose to evaluate exploration performance in terms of x-axis velocity and angular velocity, empirically choosing ranges of [-9.0, 9.0] and [-18.0, 18.0], respectively.
Each range is divided into ten smaller subranges and exploration is measured as the percentage of the bins visited among thebins, representing all visitable combinations of subranges.
Model Details.
In the LBS modules, the variational layers are implemented as linear layers that output the means and standard deviations of a distribution.
Standard deviations are obtained by applying a softplus operator to the outputs of the network.
The reparametrization trick is used to propagate gradients through the sampling from the posterior.
For vision-based tasks, observations are ﬁrst processed by small convolutional networks, which are trained as part of the dynamics and of the posterior models.
As for the model’s loss, we found β = 0 .
01 to be working well across all tasks.
We used 512-dimensional hidden layers, latent state and features, for vision-based tasks.
For continuous-control experiments, we instead use 32-dimensional hidden layers, latent state and features’ dimensionality is the same as the observation’s dimensionality.
Normalization.
Observation normalization is applied differently for different kinds of environment.
In continuous control tasks, observations are normalized by maintaining running statistics.
For video games, following the setup in (Burda et al., 2019a), observation’s mean and standard deviation values are computed on 10,000 steps collected using a random agent, and are kept ﬁxed during training.
Hyper-parameters.
In all experiments, the same learning rate, number of updates and mini-batches have been used both for the policy and the model training.
The policy value loss coefﬁcient is ﬁxed to 0 .
5 and the entropy coefﬁcient to 0 .
For continuous-control experiments, we used a learning rate of 3 e − 4 , 10 updates per episode, and 32 minibatches.
From one environment, we collected 2048 steps per epoch.
For the arcade games experiments, we used a learning rate of 1 e − 4 , 3 updates per episode, and 8 minibatches.
Fromparallel environments, we collected 128 steps from each, per epoch.
C A DDITIONAL R ESULTS
In this Section
[Table]
we provide the ﬁnal performance in terms of coverage for the continuous control experiments (Table 1) and the ﬁnal scores for the video games experiments (Table 2).
We also compare to Human performance, reporting the scores in (Mnih et al., 2015), to show that curiositydriven playing of video games can in some cases be in the same scale with human performance (e
[Table]
g.
BeamRider, Qbert), or even outperform it (e.g. Breakout).
It should be also considered that in all
Atari games, the max length of an episode was limited to 4500 steps and that performance is shown with respect to 400M frames for training, as in (Burda et al., 2019a).
Thus, the scores here shown may likely be increased with longer training or removing the episodic steps limitation
[Table]
[Table]
[Table]
Qbert
[Table] [Table] [Table]
Riverraid
[Table]
Seaquest
[Table] [Table]
SpaceInvaders [Table] [Table] [Table]
[Table]
Super Mario Bros.
[Table] [Table]
Finally, we report the training curves for the episode lenghts (Figure 8) and the best scores achieved (Figure 9), in all arcade games experiments.