I MPLICIT G ENERATIVE M ODELING FOR E FFICIENT
E
XPLORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Efﬁcient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse.
A commonly used approach for exploring such environments is to introduce some “intrinsic” reward.
In this work, we focus on model uncertainty estimation as an intrinsic reward for efﬁcient exploration.
In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent’s belief of the environment dynamics.
Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration.
We design a training algorithm for our generative model based on the amortized Stein Variational Gradient
Descent.
In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models.
In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efﬁciency in exploration.
I NTRODUCTION
Reinforcement learning (RL) has enjoyed recent success in a variety of applications, including superhuman performance in Atari games (Mnih et al., 2013), robotic control (Lillicrap et al., 2015), image-based control tasks (Hafner et al., 2019), and playing the game of Go (Silver et al., 2016).
Despite these achievements, many recent RL techniques still suffer from poor sample efﬁciency.
Agents are often trained for millions, or even billions of simulation steps before achieving a reasonable performance (Burda et al., 2018a).
This lack of statistical efﬁciency makes it difﬁcult to apply
RL to real-world tasks, as the cost of acting in the real world is far greater than in a simulator.
It is then a problem of utmost importance, to design agents that make efﬁcient use of collected data.
According to (Sutton & Barto, 2018), there are three key aspects in building a data-efﬁcient agent for reinforcement learning: generalization, exploration, and long-term consequence awareness.
In this work, we focus on the efﬁcient exploration aspect.
In particular, we focus on those challenging environments with sparse external rewards, where exploration is commonly driven by some sort of intrinsic reward.
It is observed (Osband et al., 2017; 2018) that a Bayesian uncertainty 1 estimate plays an important role in efﬁcient exploration in deep RL, but is unfortunately not appropriately addressed in the majority of state-of-the-art RL algorithms.
In this work, we introduce a new framework of Bayesian uncertainty modeling for intrinsic rewardbased exploration in RL.
Our framework characterizes the (epistemic) uncertainty in the agent’s belief of the environment dynamics in a non-parametric way to enable ﬂexibility and expressiveness.
The main component of our framework is a network generator, each draw of which is a neural network that serves as the dynamic function for RL.
Multiple draws then approximate a posterior of the dynamic model and the variance in future state prediction based on this posterior is used as an intrinsic reward for exploration.
Recently, it has been shown (Ratzlaff & Fuxin, 2019) that training such kind of generators can be done in classiﬁcation problems and the resulting draws of networks can represent a rich distribution of networks that perform approximately equally well on the classiﬁcation task.
For our goal of training this generator for the dynamic function, we propose an algorithm to optimize the KL divergence between the implicit distribution (represented by draws from the generator) and the true posterior of the dynamic model (given the agent’s experience) via the amortized Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016;
Feng et al., 2017).
Comparing with recent works (Pathak et al., 2019; Shyam et al., 2019) that maintain an ensemble of dynamic models and use the divergence or disagreement among them as an intrinsic reward for exploration, our implicit modeling of the posterior has several advantages: Firstly, it is a more ﬂexible framework for approximating the model posterior comparing with ensemble-based approximation where the number of particles is ﬁxed.
Secondly, it is based on the principle of amortized
SVGD (Feng et al., 2017), where the KL divergence between the implicit posterior and the true posterior is directly minimized in a nonparametric sense, and further projected to a ﬁnite-dimensional parameter update.
This is in contrast with existing ensemble-based methods that count on the random initialization and/or bootstrapped experience sampling for the ensemble to approximate the posterior.
Thirdly, it is more memory efﬁcient given that our method stores and updates only parameters of the generator, in contrast with parameters of every member network of the ensemble.
In our experiments, we compare our approach with several state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches that also leverage the uncertainty in dynamic models.
In all the tasks we have tested, our implementation consistently outperforms competing methods regarding data efﬁciency in exploration.
In summary, our contributions are:
• We propose a new framework for implicitly approximating the posterior of network parameters where the uncertainty of the network function can be used as an intrinsic reward for efﬁcient exploration in RL.
• We design an amortized SVGD-based training algorithm for the proposed framework and apply it to approximate the implicit distribution of the dynamic model of the environment.
• We test our implementation on three challenging exploration tasks and compare with three state-of-the-art intrinsic reward-based methods, two of which are also based on uncertainty in dynamic models.
The consistent superior performance of our method demonstrates the effectiveness of the proposed framework in estimating Bayesian uncertainty in the dynamic model for efﬁcient exploration.
P ROBLEM S ETUP AND B ACKGROUND
Consider a Markov Decision Process (MDP) represented as ( S , A , P, r, ρ 0 ) , where S is the state space, A is the action space.
P : S ×
A × S → [0 , 1] is the unknown dynamics model, specifying the probability of transitioning to next state s (cid:48) from current state s by taking action a , as P ( s (cid:48) | s, a ) .
r : S ×
A → R is the reward function, ρ 0 : S → [0 , 1] is the distribution of initial states.
A policy is a function π : S × A → [0 , 1] , which outputs a distribution over the action space for given state s .
2. E XPLORATION IN R EINFORCEMENT L EARNING
In online decision-making problems, such as multi-arm bandits and reinforcement learning, a fundamental dilemma in an agent’s choice is exploitation versus exploration.
Exploitation refers to making the best decision given current information, while exploration refers to gathering more information about the environment.
In standard reinforcement learning setting where the agent receives an external reward for each transition step, common recipes for exploration/exploitation trade-off include naive methods such as (cid:15) -greedy (Sutton & Barto, 2018) and optimistic initialization (Lai & Robbins, 1985), posterior guided methods such as upper conﬁdence bounds (Auer, 2002;
Dani et al., 2008) and Thompson sampling (Thompson, 1933).
In the situation we focus on, where external rewards are sparse or disregarded, the above trade-off narrows down to the pure exploration problem of efﬁciently accumulating information about the environment.
The common approach is to explore in a task-agnostic way under some “intrinsic” reward.
An exploration policy can then be trained in the standard RL way where dense rewards are available.
Existing methods construct intrinsic rewards from visitation frequency of the state (Bellemare et al., 2016), prediction error of the dynamic model as “curiosity” (Pathak et al., 2017), diversity of visited states (Eysenbach et al., 2018), etc.
2. D YNAMIC M ODEL U NCERTAINTY AS I NTRINSIC R EWARD
Following the guiding principle of modeling Bayesian uncertainty in online decision making, two recent methods (Pathak et al., 2019; Shyam et al., 2019) train an ensemble of dynamic models and use the variation/information gain as an intrinsic reward for exploration.
In this work, we follow the similar idea of exploiting the uncertainty in the the dynamic model, but emphasize on the implicit posterior modeling in contrast with directly training an ensemble of dynamic models.
Let f : S ×
A → S denote a model of the environment dynamics (usually represented by a neural network) we want to learn based on the agent experience D .
We design a generator module G which takes a random draw from the normal distribution and outputs a sample vector of parameters θ that determines f (denoted as f θ ).
If samples from G represent the posterior distribution p ( f θ | D ) , then given ( s t , a t ) , the uncertainty in the output of the dynamics model can be computed by the following variance among a set of samples { θ i } mi =1 from G , and used as an intrinsic reward r in for learning an exploration policy,
[Equation]
In learning the exploration policy, this intrinsic reward can be computed with either actual rollouts in the environment or simulated rollouts generated by the estimeted dynamic model.
P OSTERIOR A PPROXIMATION VIA A MORTIZED SVGD
In this section, we introduce the core component of our exploration agent, the dynamic model generator G .
In the following subsections, we ﬁrst introduce the design of this generator and then describe its training algorithm in detail.
A summary of our algorithm is given in the last subsection.
3. I MPLICIT P OSTERIOR G ENERATOR
As shown in Fig.
1, the dynamic model is deﬁned as a N -layer neural network function f
θ ( s, a ) , with input (state, action) pair ( s, a ) and model parameters θ = ( θ 1 , · · · , θ N ) , where θ j represents network parameters of the j -th layer.
The generator module G consists of exactly N layer-wise generators, { G 1 , · · · , G N } , where each G j takes a random noise vector z j ∈ R d and outputs the corresponding parameter vector θ j = G j
( z j ; η j ) , where η j are the parameters of G j .
Note that z j ’s are generated independently from a d -dimensional standard normal distribution, rather than jointly.
As mentioned in § 1, this framework has advantages in ﬂexibility and efﬁciency, comparing with ensemble-based methods (Shyam et al., 2019; Pathak et al., 2019), since it maintains only parameters of the N generators, i.e., η = ( η 1 , · · · , η N ) , and enables drawing an arbitrary number of sample networks to approximate the posterior of the dynamic model.
3. T RAINING WITH A MORTIZED S TEIN V ARIATIONAL G RADIENT D ESCENT
We now introduce the training algorithm of the generator module G .
Assuming that the true posterior of the dynamic model given agent’s experience D is p ( f |D ) , and the implicit distribution of f θ captured by G is q ( f θ ) .
We want q ( f θ ) to be as close as possible to p ( f |D ) , such closeness is commonly measured by the KL divergence D KL [ q ( f θ ) (cid:107) p ( f |D )] .
Traditional approach for ﬁnding q that minimizes D KL [ q ( f θ ) (cid:107) p ( f |D )] is variational inference, where an evidence lower bound (ELBO) is maximized.
Recently, a nonparametric variational inference framework, Stein Variational Gradient
Descent (SVGD) (Liu & Wang, 2016), was proposed, which represents q with a set of particles rather than making any parametric assumptions, and approximates the functional gradient descent w.r.t.
D KL [ q ( f θ ) (cid:107) p ( f |D )] by iterative particle evolvement.
We apply SVGD to our sampled network functions, and follow the idea of amortized SVGD (Feng et al., 2017) to project the functional gradients to the parameter space of η by back-propagation through the generators.
Given a set of dynamic functions { f θ i } mi =1 sampled from G , SVGD updates each function by
[Equation]
where (cid:15) is a step size, and φ ∗ is the function in the unit ball of a reproducing kernel Hilbert space (RKHS)
H that maximally decreases the KL divergence between the distribution q represented by { f θ i } mi =1 and the target posterior p ,
[Equation]
This optimization problem has a closed form solution,
[Equation]
where k ( · , · ) is the positive deﬁnite kernel associated with the RKHS.
We use a Gaussian kernel for our implementation.
The log-likelihood term for f θ corresponds to the negation of the regression loss of future state prediction for all transitions in D , i.e., −L ( f θ ) =
− (cid:80) ( s,a,s (cid:48) )
∈D L ( f θ ( s, a ) , s (cid:48) ) .
Given that f θ is determined by θ , the corresponding SVGD update rule for each sampled θ i is,
[Equation]
where
[Equation]
Given that θ i ’s are generated by G ( z ; η ) , the update rule for η can be obtained by by the chain rule,
[Equation]
where φ ∗ ( θ i ) can be computed by (2) using empirical expectation from sampled batch { θ i } mi =1 ,
[Equation]
Initialize Generator G η , parameters T, m
Initialize Policy π , Experience buffer D while True do while episode not done : do
[Equation]
←
[Equation]
(0 , I d ) η ← evaluate (3), (4) on D
D π ← D or (cid:101) D ∼ MDP ( f Θ )
R π ← r in ( f θ , s, a | ( s, a ) ∼ D π ) by (1) π ← update policy on ( D π , R π )
D T ← rollout π for T steps
D ← D ∪ D T end end
Algorithm 1 shows our procedure in psuedocode.
Starting with a buffer D of random transitions, our algorithm samples a set of dynamic models
f Θ = { f θ i } from the generator G , and updates the generator parameters η using amortized SVGD (3) and (4).
For policy update, the intrinsic reward (1) is evaluated on either the actual experience D or the simulated experience (cid:101) D generated by f θ i .
The exploration policy is then updated using a modelfree RL algorithm on the collected experience
D π and intrinsic rewards R π .
The updated exploration policy is then used to rollout in the environment for T steps so that new transitions are collected and added to the buffer D for subsequent iterations.
3. S UMMARY OF THE E XPLORATION A LGORITHM
To condense what we have proposed so far, we summarize in Algorithm 1 the procedure used to train the generator of dynamic models and the exploration policies.
We repeat the process, with the agent acting in the environment under the exploration policy and collecting new experience.
E XPERIMENTS
In this section we conduct experiments to compare our approach to existing state-of-the-art in efﬁcient exploration with intrinsic reward.
For our propose, only the task-agnostic setting is considered, where the agent explores the environment irrespective of the downstream task.
Task agnostic exploration is essential when external rewards are sparse and there is large uncertainty in the environment.
[Figure]
As a sanity check, we ﬁrst follow MAX (Shyam et al., 2019) to evaluate our method on a stochastic version of the toy environment NChain.
As shown in Figure 2, the chain is a ﬁnite sequence of
N states.
Each episode starts from state 1 and lasts for N + 9 steps.
For each step, the agent can move forward to the next state in the chain or backward to the previous state.
Attempting to move off the edge of the chain results in the agent staying still.
Reward is only afforded to the agent at the edge states: 0 .
01 for reaching state 0 , and 1 .
for reaching state N − 1 .
In addition, there is uncertainty built into the environment: each state is designated as a ﬂip-state with probability 0 .
When acting from a ﬂip-state, the agent’s actions are reversed, i.e., moving forward will result in movement backward, and vice-versa.
Given the (initially) random dynamics and a sufﬁciently long chain, we expect an agent using an (cid:15) -greedy exploration strategy to exploit only the small reward of state 0 .
In contrast, agents with exploration policies which actively reduce uncertainty can efﬁciently discover every state in the chain.
Figure 7 shows that our agent navigates the chain in less thanepisodes, while indeed, the (cid:15) -greedy agent (double DQN) does not make meaningful progress.
For completeness, we also evaluate the agents introduced in the following sections and show the results in the appendix.
4. C ONTINUOUS C ONTROL E NVIRONMENTS
We also consider three challenging continuous control tasks in which efﬁcient exploration is known to be difﬁcult.
In each environment, the dynamics are nonlinear and cannot be solved with simpler (efﬁcient) tabular approaches.
As stated above, external reward is completely removed; the agent is motivated purely by the uncertainty in its belief of the environment.
Experimental setup
To validate the effectiveness of our method, we compare with several state-of-the-art formulations of intrinsic reward.
Speciﬁcally, we conduct experiments comparing the following methods:
• ( Ours )
The proposed intrinsic reward, using the estimated variance of an implicit distribution of the dynamic model.
• ( ICM ) Error between predicted next state and observed next state (Pathak et al., 2017).
• ( Disagreement ) Variance of predictions from an ensemble of dynamic models (Pathak et al., 2019).
• ( MAX ) Jensen-Renyi information gain of the dynamic function (Shyam et al., 2019).
• ( Random )
Pure random exploration as a naive baseline.
Implementation details
Given our goal is to compare the performance across different intrinsic rewards, we ﬁx the model architecture, training pipeline, and hyperparameters across all methods.
The dynamic models arelayer fully connected neural networks.
For the purpose of computing the information gain, dynamic models for MAX predict both mean and variance of the next state, while for other methods, Dynamic models predict only the mean.
Our generator as well as the dynamic models for other methods are optimized using Adam (Kingma & Ba, 2014) with a learning rate of 1 e − 4 .
To learn exploration policies, we use the Soft Actor Critic (Haarnoja et al., 2018) algorithm for all methods.
For MAX ,
ICM , and Disagreement , we use ensembles of 32 dynamic models respectively to compute the intrinsic reward.
Since our method trains a generator of dynamic models instead, we ﬁx the number of models we sample from the generator at m = 32 for a fair comparison.
Further implementation details can be found in the supplementary material.
4.2. A CROBOT C ONTROL
[Figure]
Our ﬁrst environment is a modiﬁed continuous control version of the Acrobot.
As shown in Figure 4(a), the Acrobot environment begins with a hanging down pendulum which consists of two links connected by an actuated joint.
Normally, an action a ∈ {− 1 , 0 , 1 } applies or not ( a = 0 ) a unit force on the joint in the left or right direction.
We modify the environment such that a continuous action a ∈ [ − 1 , 1] applies a force F = | a | in the corresponding direction.
To focus on efﬁcient exploration, we test the ability of each exploration method to sweep the entire lower hemisphere: positioning the acrobot completely horizontal towards both (left and right) directions.
Given this is a relatively simple task and can be solved by random exploration, as shown in
Figure 4(b), all four intrinsic reward methods solve it within just hundreds of steps and our method is the most efﬁcient one.
The takeaway here is that in relatively simple environments where there might be little room for improvement over state-of-the-art, our method still achieves a better performance due to its ﬂexibility and efﬁciency in approximating the model posterior.
We will see in subsequent experiments that this observation scales well with more difﬁcult environments.
4.2. A NT M AZE N AVIGATION
Next, we evaluate on the Ant Maze environment (Figure 5(a)).
In this control task, the agent provides torques to each of the 8 joints of the ant.
The provided observation contains the pose of the torso as well as the angles and velocities of each joint.
The agent’s performance is measured by the percentage of the U-shaped maze explored during evaluation.
Figure 5(b) shows the result of each method over 5 seeds.
Our agent consistently navigates to the end of the maze at the time when other methods have only explored 60% or less.
We show how state visitation frequencies progress through training in ﬁgures 5(c)-5(f).
While MAX (Shyam et al., 2019) also navigates the maze, the more advanced uncertainty modeling of our method allows our agent to better estimate the state novelty, which leads to a considerably quicker exploration.
[Figure]
) is able to navigate to the end of the maze much faster than any other method.
Figures (c-f) show the behavior of the agent at different stages of training.
Points are color-coded with blue points occurring at the beginning of the episode, and red points at the end.
4.2. R OBOTIC M ANIPULATION
The ﬁnal task is an exploration task in a robotic manipulation environment, HandManipulateBlock.
As shown in Figure 6(a), a robotic hand is given a palm-sized block for manipulation.
The agent has actuation control of the 20 joints that make up the hand, and its exploration performance is measured by the percentage of possible rotations of the cube that the agent performs.
3 In particular, the state of the cube is represented by Cartesian coordinates along with a quaternion to represent the rotation.
We transform the quaternion to Euler angles and discretize the resulting state space by 45 degree intervals.
The agent is evaluated based on how many of the 512 total states are visited.
This task is far more challenging than previous tasks, having a larger state space and action space.
Additionally, states are difﬁcult more difﬁcult to reach than the Ant Maze environment; requiring manipulation of 20 joints instead of 8.
In order to explore in this environment, an agent must also learn how to rotate the block without dropping it.
Figure 6(b) shows the performance of each method over 5 seeds.
This environment proved very challenging for all methods, none succeeded in exploring more than half of the state space.
When placed in a complicated environment where the task is not clear, we want our agents to explore as fast as possible, in order to master the dynamics of the environment.
For this environment, we can see that our method indeed performs the best by a clear margin, regarding exploration efﬁciency. [Figure]
R ELATED W ORK
Efﬁcient Exploration remains a major challenge in deep reinforcement learning (Fortunato et al., 2017; Burda et al., 2018b;
Eysenbach et al., 2018; Burda et al., 2018a), and there is no consensus on the correct way to explore an environment.
One practical guiding principle for efﬁcient exploration is the reduction of agent’s epistemic uncertainty of the environment (Chaloner & Verdinelli, 1995;
Osband et al., 2017).
Osband et al.
(2016) uses a bootstrap ensemble of DQNs, where the predictions of the ensemble are used as an estimate of the agent’s uncertainty over the value function.
Osband et al.
(2018) proposed to augment the predictions of a DQN agent by adding the contribution from a prior to the value estimate.
In contrast to our method, these approaches seek to estimate the uncertainty in the value function , while we focus on exploration with intrinsic reward by estimating the uncertainty of the dynamic model .
Fortunato et al.
(2017) add parameterized noise to the agent’s weights, to induce state-dependant exploration beyond (cid:15) -greedy or entropy bonus.
Methods for constructing intrinsic rewards for exploration have become the subject of increased study.
One well-known approach is to use the prediction error of an inverse dynamics model as an intrinsic reward (Pathak et al., 2017; Schmidhuber, 1991).
Schmidhuber (1991) and Sun et al.
(2011) proposed using the learning progress of the agent as an intrinsic reward.
Count based methods (Bellemare et al., 2016;
Ostrovski et al., 2017) give a reward proportional to the visitation count of a state.
Houthooft et al.
(2016) formulate exploration as a variational inference problem, and use Bayesian neural networks (BNN) to maintain the agent’s belief over the transition dynamics. The
BNN predictions are used to estimate a form of Bayesian information gain called compression improvement.
The variational approach is also explored in Mohamed & Rezende (2015); Gregor et al. (2016); Salge et al.
(2014), who proposed using intrinsic rewards based on a variational lower bound on empowerment; the mutual information between an action and the induced next state.
This reward is used to learn a set of discriminative low-level skills.
The most closely-related work to ours are two recent methods (Pathak et al., 2019; Shyam et al., 2019) that compute intrinsic rewards from an ensemble of dynamic models.
Disagreement among the ensemble members in next-state predictions is computed as an intrinsic reward.
Shyam et al.
(2019) also uses active exploration (Schmidhuber, 2003; Chua et al., 2018), in which the agent is trained in a surrogate MDP, to maximize intrinsic reward before acting in the real environment.
Our method follows the similar idea of exploiting the uncertainty in the dynamic model, but instead suggests an implicit generative modeling of the posterior of the dynamic function, which enables a more ﬂexible approximation of the posterior uncertainty with better sample efﬁciency.
There has been a wealth of research on nonparametric particle-based variational inference methods (Liu & Wang, 2016; Dai et al., 2016; Ambrogioni et al., 2018), where a certain number of particles are maintained to represent the variational distribution, and updated by solving an optimization problem.
Notably, we make use of the amortized SVGD (Feng et al., 2017) to optimize our generator for approximately sampling from the posterior of the dynamic model.
C ONCLUSION
In this work, we introduced a novel method for representing the agent’s uncertainty of the environment dynamics.
We formulated an intrinsic reward based on the uncertainty given by an approximate posterior of the dynamic model to enable efﬁcient exploration in difﬁcult environments, Through experiments in control, navigation, and manipulation, we demonstrated that our method is consistently more sample efﬁcient than the baseline methods.
Future work includes investigating the efﬁcacy of learning an approximate posterior of the agent’s value or policy model, as well as more efﬁcient sampling techniques.
A A PPENDIX A. I MPLEMENTATION D ETAILS FOR C ONTINUOUS E NVIRONMENTS
Here we describe in more detail the various implementation choices we used for our method as well as for the baselines.
Toy Chain Environment
The chain environment was implemented based on the NChain-v0 gym environment.
We altered
NChain-v0 to contain 40 states instead of 10 to reduce the possibility of solving the environment with random actions.
We also modiﬁed the stochastic ’slipping’ state behavior by ﬁxing the behavior of the states respect to reversing an action.
For both our method and MAX , we use ensembles of 5 deterministic neural networks with 4 layers, each are 256 units wide with tanh nonlinearities.
As usual, our ensembles are sampled from the generator at each timestep, while MAX uses a static ensemble.
We generate each layer in the target network with generators composed of two hidden layers, 64 units each with ReLU nonlinearities.
Both models are trained by minimizing the regression loss on the observed data.
We optimize using Adam with a learning rate of 10 − 4 , and weight decay of 10 − 6 .
We use Monte Carlo Tree Search (MCTS) to ﬁnd exploration policies for use in the environment.
We build the tree with 25 iterations of 10 random trajectories, and UCB-1 as the selection criteria.
Crucially, during rollouts, we query the dynamic models instead of the simulator, and we compute the corresponding intrinsic reward.
For MAX we use the Jensen Shannon divergence while our method uses the variance in the predictions of our samples.
There is a small discrepancy between the numbers reported in the MAX paper for the chain environment.
This is due to using
UCB-1 as the selection criteria instead of Thompson sampling as used in the MAX paper.
We take actions in the environment based on the children with the highest value.
The tree is then discarded after one step, after which, the dynamic models are ﬁt for 10 additional epochs.
Continuous Control Environments
For each method, we kept the implementation details consistent across the environments Acrobot,
Ant Maze, and Block Manipulation.
The common details of each exploration method are as follows.
Each method uses (or samples) ensembles to approximate environment dynamics.
Models in the ensemble are composed of 32 networks with 4 hidden layers, 512 units wide with ReLU nonlinearities, except for MAX which uses swish 4 .
ICM , Disagreement , and our method use ensembles of deterministic models, while MAX uses probabilistic networks which output a Gaussian distribution over next states.
The approximate dynamic models (ensembles/generators) are optimized with Adam, using a minibatch size of 256, a learning rate of 10 − 4 , and weight decay of 10 − 4 .
For our dynamic model, each layer generator is composed of two hidden layers, 64 units wide and
ReLU nonlinearity.
The output dimensionality of each generator is equal to the product of the input and output dimensionality of the corresponding layer in the dynamic model.
To sample one dynamic model, each generator takes as input an independent draw from z ∼ Z where Z = N (0 , I 32 ) where
I is a 32 dimensional identity matrix.
We sample ensembles of arbitrary size m by instead providing a batch { z } mi =1 as input.
To train the generator such that we can sample accurate transition models we update according to Equation 4; we compute the regression error on the data, as well as the repulsive term using an appropriate kernel.
For all experiments we used a standard Gaussian kernel k
( θ, θ i ) =
exp (cid:0) − 1 h || θ −
θ
i
|| 22 (cid:1) where h is the median of the pairwise distances between sampled particles { θ } mi
=1 .
Because we sample functions f θ instead of data points, the pairwise distance is computed by using the likelihood of the data x under the model: log f θ ( x ) .
For MAX , we used the code provided from (Shyam et al., 2019).
Each member in the approximate dynamic model ensemble is a probabilistic neural network that predicts a Gaussian distribution (with diagonal covariance) over next states.
The exploration policies are trained with SAC, given an experience buffer of rollouts ¯ D = { s, a, s (cid:48) } ∪ Rπ performed by dynamic models, where R π is the intrinsic reward: the Jensen-Renyi divergence between next state predictions of the dynamic model.
the policy trained with SAC acts in the environment to maximize the intrinsic reward, and in doing so collects additional transitions that serve as training data for the dynamic models for the subsequent training phase.
For Disagreement (Pathak et al., 2019), we follow the author’s implementation, changing minimal details.
The intrinsic reward is formulated as the predictive variance of the approximate dynamic model, where the model is given by a bootstrap ensemble.
In this work, we report results using two versions of this method.
The proposed intrinsic reward speciﬁcally is formulated in a manner quite similar to our own, however, an ensemble is used instead of a distribution for the approximate posterior.
In section § 4 we report results only using the intrinsic reward , instead of the full proposed method which makes use of a differentiable reward function, which treats the reward as a supervised loss.
We do this because the form of the approximate dynamic model does not preclude the use of different policy optimization techniques.
Nonetheless, in the next section § A.2 we report results using the full method as proposed, on each continuous control experiment.
The Bayesian approach is extended in (Houthooft et al., 2016)
A. A DDITIONAL R ESULTS
Here we report additional results on both our toy chain task. We also perform additional comparisons with Disagreement , including the original policy optimization method with a differentiable reward function as in (Pathak et al., 2019).
A.2. E XTENDED NC HAIN R ESULTS
Here we show the results of each method on the NChain task. We used the same experimental setup as in section 4.1 for our experiments. For both ICM and Disagreement we again use MCTS with UCB-1 to ﬁnd exploration policies, with the same hyperparameters used for our method and MAX .
We ﬁnd that reducing uncertainty is a crucial component to be able to explore the chain.
ICM only reduces prediction error, so it is not interested in states it can predict correctly.
The only source of reward for ICM is in the ﬂipped directions of some states, so its possible to initialize chains where
ICM cannot ﬁnd novel states, causing high variance between runs.
We found that Disagreement performs better than ICM
but not as well as MAX or our method.
We suspect that this is due to a weakness in the ensemble diversity, lessening the effect of the intrinsic reward for this task.
The ensemble can easily overﬁt to a toy environment like NChain, reducing its ability to explore.
MAX may avoid overﬁtting to the chain due to using ensembles of stochastic neural networks.
Our method uses amortized SVGD to promote model diversity, and we use an estimate of the uncertainty in our dynamic model to explore new states.
A.2. E XTENDED D ISAGREEMENT R ESULTS
We repeat our main experiments, comparing our method to both disagreement purely as an intrinsic reward, as well as the full method using the differentiable reward function for policy optimization.
For the full method we use the author’s ofﬁcial implementation for the following experiments.
This is in contrast to the method reported in the main text where we only implement the intrinsic reward.
Figures 8, 9, and 10 show results on the Acrobot, Ant Maze, and Block Manipulation environments, respectively.
In each ﬁgure, lines correspond to the mean of ﬁve seeds, and shaded regions denote ± one standard deviation.
In each experiment, we can see that treating the intrinsic reward as a supervised loss (gray) improves on the baseline disagreement intrinsic reward (green).
However, our method (red) remains the most sample efﬁcient in these experiments.