L EARNING TO P LAN O PTIMISTICALLY : U NCERTAINTY -G UIDED D EEP E XPLORATION VIA L ATENT M ODEL E NSEMBLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning complex behaviors through interaction requires coordinated long-term planning.
Random exploration and novelty search lack task-centric guidance and waste effort on non-informative interactions.
Instead, decision making should target samples with the potential to optimize performance far into the future, while only reducing uncertainty where conducive to this objective.
This paper presents latent optimistic value exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards.
We combine ﬁnitehorizon rollouts from a latent model with value function estimates to predict inﬁnitehorizon returns and recover associated uncertainty through ensembling.
Policy training then proceeds on an upper conﬁdence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance.
We apply LOVE to visual control tasks in continuous state-action spaces and demonstrate improved sample complexity on a selection of benchmarking tasks.
I NTRODUCTION
The ability to learn complex behaviors through interaction will enable the autonomous deployment of various robotic systems in the real world.
Reinforcement learning (RL) provides a key framework for realizing these capabilities, but efﬁciency of the learning process remains a prevalent concern.
Reallife applications yield complex planning problems due to high-dimensional environment states, which are further exacerbated by the agent’s continuous actions space.
For RL to enable real-world autonomy, it therefore becomes crucial to determine efﬁcient representations of the underlying planning problem, while formulating interaction strategies capable of exploring the resulting representation efﬁciently.
In traditional controls, planning problems are commonly formulated based on the underlying statespace representation.
This may inhibit efﬁcient learning when the environment states are highdimensional or their dynamics are susceptible to non-smooth events such as singularities and discontinuities (Schrittwieser et al., 2019;
Hwangbo et al., 2019; Yang et al., 2019).
It may then be desirable for the agent to abstract a latent representation that facilitates efﬁcient learning (Ha & Schmidhuber, 2018; Zhang et al., 2019; Lee et al., 2019).
The latent representation may then be leveraged either in a model-free or model-based setting.
Model-free techniques estimate state-values directly from observed data to distill a policy mapping.
Model-based techniques learn an explicit representation of the environment that is leveraged in generating ﬁctitious interactions and enable policy learning in imagination (Hafner et al., 2019a).
While the former reduces potential sources of bias, the latter offers a structured representation encoding deeper insights into underlying environment behavior.
The agent should leverage the chosen representation to efﬁciently identify and explore informative interactions.
We provide a motivational one-dimensional example of a potential action-value mapping in Figure 1 (left).
The true function and its samples are visualized in red with the true maximum denoted by the green dot.
Relying only on the predicted mean can bias policy learning towards local optima (orange dot; Sutton & Barto (2018)), while added stochasticity can waste samples on un-informative interactions.
Auxiliary information-gain objectives integrate predicted uncertainty, however, uncertain environment behavior does not equate to potential for improvement (pink dot).
It is desirable to focus exploration on interactions that harbor potential for improving overall performance.
Combining mean performance estimates with the associated uncertainty into an upper conﬁdence
Exploration based on maximization of the predicted mean can exploit local optima (orange dot), while information-gain bonuses may focus on uncertain regions with little potential of improvement (
Explicitly considering uncertainty over expected performance can help focus exploration high bound (UCB) objective provides a concise method of doing so (blue dot
Auer et
; Krause &
Ong (2011)).
The underlying uncertainty can be explicitly represented by maintaining an ensemble of hypothesis on environment behavior (Osband et al., 2016; Lakshminarayanan et al., 2017).
Figure(right) demonstrates this selective uncertainty reduction by showcasing forward predictions of a ensembled model on two motion patterns of a walker agent.
The expected high-reward walking behavior has been sufﬁciently explored and model hypotheses strongly agree (top), while little effort has been extended to reduce uncertainty over the expected low-reward falling behavior (bottom).
This paper demonstrates that exploring interactions through imagined positive futures can yield information-dense sampling and data-efﬁcient learning.
We present latent optimistic value exploration (LOVE), an algorithm that leverages optimism in the face of uncertain long-term rewards in guiding exploration.
Potential futures are imagined by an ensemble of latent variable models and their predicted inﬁnite-horizon performance is obtained in combination with associated value function estimates.
Training on a UCB objective over imagined futures yields a policy that behaves inherently optimistic and focuses on interactions with the potential to improve performance under the current world model.
This provides a concise, differentiable framework for driving deep exploration while not relying on stochasticity.
We present empirical results on challenging visual control tasks that highlight the necessity for deep exploration in scenarios with sparse reward signals, and demonstrate improved sample-efﬁciency on a selection of benchmarking environments from the DeepMind Control Suite (Tassa et al., 2018).
We compare to both Dreamer (Hafner et al., 2019a), the current state-of-the-art model-based agent, and DrQ (Kostrikov et al., 2020), a concurrent model-free approach.
R ELATED WORK
Problem representation Model-free approaches learn a policy by directly estimating performance from interaction data.
While their asymptotic performance previously came at the cost of sample complexity (Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018), recent advances in representation learning through contrastive methods and data augmentation have improved their efﬁciency (Srinivas et al., 2020; Laskin et al., 2020; Kostrikov et al., 2020).
However, their implicit representation of the world can make generalization of learned behaviors under changing task speciﬁcations difﬁcult.
Model-based techniques leverage a structured representation of their environment that enables them to imagine potential interactions.
The nature of the problem hereby dictates model complexity, ranging from linear (Levine & Abbeel, 2014; Kumar et al., 2016), over Gaussian process models (Deisenroth & Rasmussen, 2011; Kamthe & Deisenroth, 2018) to deep neural networks (Chua et al., 2018; Clavera et al., 2018).
In high-dimensional environments, latent variable models can provide concise representations that improve efﬁciency of the learning process (Watter et al., 2015;
Ha & Schmidhuber, 2018; Lee et al., 2019; Hafner et al., 2019a).
Planning interactions Model-based approaches leverage their representation of the world in predicting the performance of action sequences.
The agent may then either solve a model predictive control-style optimization (Nagabandi et al., 2018; Chua et al., 2018; Hafner et al., 2019b) or train a policy in simulation (Kurutach et al., 2018; Clavera et al., 2018).
The resulting ﬁnite-horizon formulations can be extended by value function estimates to approximate an inﬁnite-horizon planning problem (Lowrey et al., 2019; Hafner et al., 2019a; Seyde et al., 2020).
When considering learned models, ensembling the model predictions may further be leveraged in debiasing the actor strategy during training (Kurutach et al., 2018;
Chua et al., 2018; Clavera et al., 2018; Seyde et al., 2020).
Both explicit and implicit model rollouts in combination with value estimates can also be utilized to accelerate model-free learning (Oh et al., 2017;
Feinberg et al., 2018; Buckman et al., 2018).
Directed exploration Directed exploration can improve over random exploration by focusing on information-dense interactions (Schmidhuber, 2010).
These methods are commonly driven by uncertainty estimates.
Information gain techniques deﬁne an auxiliary objective that encourages exploration of unexpected environment behavior or model disagreement and have been applied in discrete (Stadie et al., 2015; Ostrovski et al., 2017; Pathak et al., 2017) and continuous actions spaces (Still & Precup, 2012; Houthooft et al., 2016; Henaff, 2019).
When driving interactions to improve knowledge of the environment dynamics, effort may be wasted on uncertain interactions that are tangential to the objective.
Alternatively, exploration can be driven by uncertainty over expected performance as encoded by value functions (Osband et al., 2016; 2017; Chen et al., 2017;
O’Donoghue et al., 2018; Lee et al., 2020), multi-step imagined returns (Depeweg et al., 2018; Henaff et al., 2019) or their combination (Lowrey et al., 2019; Schrittwieser et al., 2019; Seyde et al., 2020).
Model-ensemble agents Related work on ensemble agents has demonstrated impressive results.
We note key differences to our approach.
ME-TRPO (Kurutach et al., 2018) leverages a dynamics ensemble to debias policy optimization on ﬁnite-horizon returns under a known reward function and random exploration.
MAX (Shyam et al., 2019) and Amos et al. (2018) explore via ﬁnite-horizon uncertainty in a state and latent dynamics ensemble, respectively. RP1 (Ball et al., 2020) explores in reward space via ﬁnite-horizon returns, but assumes access to the nominal reward function and full proprioceptive feedback. Seyde et al.
(2020) also leverage full proprioception and embed optimism into the value function, which prohibits adjustment of the exploration trade-off during policy learning and limits transferability.
Exploring uncertain dynamics samples interactions orthogonal to task completion and ﬁnite-horizon objectives limit exploration locally, while full-observability and access to the reward function are strong assumptions.
We learn latent dynamics, reward and value functions in partially observable settings to explore uncertainty over inﬁnite-horizon returns.
This enables backpropagation through imagined trajectories to recover analytic policy gradients, while offering a ﬂexible framework to guide exploration based on expected potential for long-term improvement.
P RELIMINARIES
In the following, we ﬁrst introduce the general problem deﬁnition and the associated nomenclature.
We then provide an overview of the representation learning framework used to abstract environment behavior, which leverages the recurrent state space model (RSSM) proposed by Hafner et al. (2019a).
3. P ROBLEM FORMULATION
We formulate the underlying optimization problem as a partially observable Markov decision process (POMDP) deﬁned by the tuple {X , A , T, R, Ω , O , γ } , where X , A , O denote the state, action and observation space, respectively, T :
X × A → X signiﬁes the transition mapping, R :
X ×
A → R the reward mapping, Ω: X → O the observation mapping, and γ ∈ [0 , 1) is the discount factor.
We deﬁne x t and a t to be the state and action at time t , respectively, and use the notation r t =
R ( x t , a t ) .
Let π φ ( a t | o t ) denote a policy parameterized by φ and deﬁne the discounted inﬁnite horizon return
G t = (cid:80) ∞ τ
= t γ τ − t R ( x τ , a τ ) , where
x t +1 ∼ T ( x t +1 | x t , a t ) and a t ∼ π φ ( a t | o t ) .
The goal is then to learn the optimal policy maximizing G t under unknown nominal dynamics and reward mappings.
3. P LANNING FROM PIXELS
We build on the world model introduced in Hafner et al.
(2019b)
and reﬁned in Hafner et al. (2019a).
High-dimensional image observations are ﬁrst embedded into a low-dimensional latent space using a neural network encoder.
An RSSM then serves as a probabilistic transition model and deﬁnes the model state s .
Together, the encoder and RSSM deﬁne the representation model.
The agent therefore abstracts observation
o t of environment state x t into model state s t , which is leveraged for planning.
Consistency of the learned representations is enforced by minimizing the reconstruction error of a decoder network in the observation model and the ability to predict rewards of the reward model.
For details, we refer the reader to Hafner et al. (2019a), and provide their deﬁnitions of the models as
Representation model:
[Equation]
where p and q denote distributions in latent space, with θ as their joint parameterization.
The action model π φ ( a t | s t ) is then trained to optimize the predicted return of imagined world model rollouts.
The world model is only rolled-out over a ﬁnite horizon H , but complemented by predictions from the value model v ψ ( s t ) at the terminal state s t + H to approximate the inﬁnite horizon return.
The policy and value function are trained jointly using policy iteration on the objective functions
[Equation]
respectively.
Here, V λ ( s τ ) represents an exponentially recency-weighted average of the k -step value estimates V kN ( s τ ) along the trajectory to stabilize the learning (Sutton & Barto, 2018), such that
[Equation]
with h = min( τ + k, t + H ) .
U NCERTAINTY - GUIDED LATENT EXPLORATION
The world model introduced in Section 3.2 can be leveraged in generating ﬁctitious interactions for the policy to train on.
However, the learned model will exhibit bias in uncertain regions where insufﬁcient samples are available.
Training on imagined model rollouts then propagates simulation bias into the policy.
Here, we address model bias by ensembling our belief over environment behavior.
We can leverage the underlying epistemic uncertainty in formulating a UCB objective for policy learning that focuses exploration on regions with high predicted potential for improvement.
4. M ODEL LEARNING WITH UNCERTAINTY ESTIMATION
The model parameters are only weakly constrained in regions where interaction data is scarce and random inﬂuences have a detrimental effect on prediction performance.
In order to prevent the agent from learning to exploit these model mismatches, we consider predictions from an ensemble.
Individual predictions will align in regions of high data support and diverge in regions of low support.
The ensemble mean then serves as a debiased estimator of environment behavior, while the underlying epistemic uncertainty is approximated via model disagreement (Lakshminarayanan et al., 2017).
We consider an ensemble of M latent-space particles.
Each particle is represented by a unique pairing of a transition model, reward model and value model to yield,
Particle ensemble:
[Equation]
The encoder remains shared between the individual representation models.
This ensures that all particles operate over the same compact latent space, while the internal transition dynamics retain the ability of expressing distinct hypothesis over environment behavior.
For each particle i , we then deﬁne the predicted inﬁnite-horizon trajectory return as
[Equation]
where V λ,i ( s τ ) is computed via Eq.
(3) with the particle’s individual transition, reward and value models.
Distinctness of the particles is encouraged by varying the initial network weights between ensemble members and shufﬂing the batch order during training.
Predicted trajectory returns with corresponding uncertainty estimates are then obtained by considering the ensemble mean and variance,
[Equation]
4. P OLICY LEARNING WITH DIRECTED EXPLORATION
The policy learning objective in Eq.
(2) could be replaced by the ensemble mean in Eq.
This would reduce model bias in the policy, but require an auxiliary objective to ensure sufﬁcient exploration.
We consider exploration to be desirable when it reduces uncertainty over realizable task performance.
The trajectory return variance in Eq.
(6) encodes uncertainty over long-term performance of actions.
In combination with the expected mean, we recover estimated performance bounds.
During data acquisition, we explicitly leverage the epistemic uncertainty in identifying interactions with potential for improvement and deﬁne the acquisition policy objective via an upper conﬁdence bound (UCB) as
[Equation]
where the scalar variable β quantiﬁes the exploration-exploitation trade-off.
For β < 0 we recover a safe-interaction objective, while β > 0 translates to an inherent optimism that uncertainty harbors potential for improvement.
Here, we learn an optimistic policy π φ aq that is intrinsically capable of deep exploration and focuses interactions on regions with high information-density.
Furthermore, in the absence of dense reward signals, the acquisition policy can leverage prediction uncertainty in driving exploration.
This behavior is not limited to within the preview window, as the value function ensemble projects long-term uncertainty into the ﬁnite-horizon model rollouts.
While training in imagination, we leverage the optimistic policy to update our belief in regions that the acquisition policy intends to visit.
In parallel, we train an evaluation policy π φ ev that aims to select the optimal actions under our current belief.
The evaluation policy optimizes the expected mean return ( β = 0 ).
4. L ATENT OPTIMISTIC VALUE EXPLORATION (LOVE)
In the following, we provide a high-level description of the proposed algorithm, LOVE, together with implementation details of the overall training process and the associated pseudo-code in Algorithm 1.
Summary
The proposed algorithm leverages an ensemble of latent variable models in combination with value function estimates to predict inﬁnite-horizon trajectory performance and associated uncertainty.
The acquisition policy is then trained on a UCB objective to imagine positive futures and focus exploration on interactions with high predicted potential for long-term improvement.
The ensemble members are constrained to operate over the same latent space to encourage learning of abstract representations conducive to the objective, while ensuring consistency between predictions.
Implementation The algorithm proceeds in two alternating phases.
In the online phase, the agent leverages its acquisition policy to explore interactions optimistically and resulting transitions are appended to memory D .
In the ofﬂine phase, the agent ﬁrst updates its belief about environment behavior and then adjusts its policy accordingly.
The representation learning step extends the procedure introduced in Hafner et al. (2019a) to ensemble learning and groups each model with a unique value function estimator into a particle.
The batch order is varied between particles during training to ensure variability in the gradient updates and to prevent collapse into a single mode.
The policy learning step combines particle predictions to generate the value targets of Eq.
(5) by simulating ensemble rollouts from the same initial conditions.
The trajectory return statistics of
Eq.
(6) are combined into the UCB objective of Eq.
(7), which the acquisition policy optimizes. We provide an Online
Ofﬂine ModelUpdate Policy Update
E XPERIMENTS
We provide results for training agents using LOVE on a selection of visual control tasks.
First, we illustrate intrinsic exploration in the absence of reward signals on a classic continuous control task.
We then benchmark performance on a selection of environments from the DeepMind Control Suite (Tassa et al., 2018).
We use a single set of parameter values throughout, as detailed in Appendix A.
5. C ASE STUDY : EXPLORATION IN THE ABSENCE OF REWARDS
We consider a planar bug trap environment in which the agent starts inside an enclosure and needs to exit through a narrow passage.
The environment does not provide any reward signals to guide planning and an escape needs to arise implicitly through exploration.
The agent is represented as a pointmass under continuous actuation-limited acceleration control.
The ratio of agent diameter to passage width is 0 .
9 and collisions are modelled as inelastic with the coefﬁcient of restitution set to 0 .
The relative size strictly limits the range of policies allowing for entry into the passage, while the low coefﬁcient of restitution encourages directed exploration by reducing the presence of random motion patterns.
The agent always continues from the terminal conﬁguration of the previous episode.
We consider two variations of the same environment distinct in the location of the passage and the relative initial position of the agent.
The two initial conﬁgurations are provided in the ﬁrst column of
Figure 2.
The agent observes a top-down view and interacts with the environment for 100 episodes.
We compare performance of LOVE with both Dreamer and LVE, a variation of LOVE that does not leverage optimism ( β = 0 ).
The resulting occupancy maps are provided in the remaining panels of Figure 2.
We note that LOVE escapes the bug trap in both scenarios to explore a much larger fraction of the state space than either LVE or Dreamer.
This can be attributed to LOVE’s ability to explicitly consider reward uncertainty beyond the preview horizon, which optimistically drives deep exploration in the absence of reward signals (column 2).
Removing this long-term optimism by only guiding interactions through mean performance estimates can lead to prediction collapse and the agent assuming the absence of reward without explicitly querying the environment for conﬁrmation (column 3).
We observe similar behavior for the Dreamer agent, which employs a single latent model and leverages random noise for exploration (column 4).
Repeating both scenarios on 3 random seeds conﬁrms this trend (column 5), where the associated occupancy maps are provided in Appendix E.
5. B ENCHMARKING : PERFORMANCE ON D EEP M IND C ONTROL S UITE
The previous section highlighted the ability of LOVE to guide exploration even in the absence of proper reward feedback.
In the following, we benchmark performance on a variety of visual control tasks from the DeepMind Control Suite.
Each environment features (64 , 64) RGB image observations and a continuous action space, while reward signals vary from dense to sparse.
Each episode initializes the agent in a random conﬁguration and runs for 1000 environment steps with an action repeat of 2 .
We use the same set of parameters throughout all our experiments, mirroring the base parameters of Dreamer to ensure a fair comparison.
However, we do not use random exploration noise and set the corresponding magnitude to 0 .
We furthermore use an ensemble of M = 5 latent variable models, an initial UCB trade-off parameter of β = 0 . 0 , a per episode UCB growth rate of δ = 0 . 001 , and alter the policy learning rate and training steps to account for the more complex UCB objective, requiring propagation of both updated mean and uncertainty estimates into the policy. The changes to the policy learning rate and training steps are also applied to the Dreamer agent to yield ∆ Dreamer, as we found this modiﬁcation to generally affect performance favourably (see Appendix I).
We furthermore introduce an exploration baseline, ∆ Dreamer + Curious, inspired by the curiosity bonus in Sekar et al.
(2020) that combines ∆ Dreamer with an ensemble of RSSM dynamics.
Figure 3 provides results for benchmarking performance over 300 episodes.
Performance is evaluated on 5 seeds, where solid lines indicate the mean and shaded areas correspond to one standard deviation.
The numeric data is provided in Table 1.
We note that LOVE improves performance over both ∆ Dreamer and ∆ Dreamer + Curious on the majority of experiments.
This suggests that LOVE’s combination of latent model ensembling with directed exploration can aid in identifying interactions conducive to solving the task.
Ensembling reduces propagation of model-speciﬁc biases into the policy.
Optimistic exploration of uncertain long-term returns focuses sampling on regions with promising performance estimates while ignoring uncertain regions that are tangential to task completion.
We ablate the performance of LOVE on the UCB trade-off parameter β and compare against LVE ( β = 0 ).
We note that while LOVE outperforms LVE on the majority of tasks, both reach similar terminal performance in several instances.
However, LOVE provides clear improvements on the fullysparse Cartpole Swingup and the partially-sparse Hopper Hop tasks.
Both environments initialize the agent in conﬁgurations that provide no reward feedback, thereby forcing the agent to actively explore.
LOVE leverages uncertainty-guided exploration and gains an advantage under these conditions.
Similarly, this can explain the performance parity on the Pendulum Swingup task.
While the task only provides sparse reward feedback, random environment initializations offer sufﬁcient visitation [Figure] to the non-zero reward states and therefore remove the necessity for active exploration.
LOVE also improves performance on the Walker Walk and Walker Run tasks.
These environments require learning of stable locomotion patterns under complex system dynamics, where directed exploration helps to efﬁciently identify tweaks to the gait.
Similar to the bug trap environment of Section 5.1, we observe that optimistic exploration is especially favoured by objectives that provide sparse reward feedback, while enabling efﬁcient discovery of tweaks to motion patterns under complex dynamics.
We additionally compare to Data Regularized Q (DrQ), a concurrent model-free approach.
DrQ updates its actor-critic models online giving it an advantage over both LOVE and Dreamer.
We note that LOVE performs favourably on the majority of environments with signiﬁcant differences on the sparse tasks (Pendulum, Cartpole Sparse) and the locomotion tasks (Hopper, Walker).
DrQ outperforms LOVE on Finger Spin and Cartpole with dense rewards.
On these tasks, learning an explicit world model may actually be detrimental to attaining performance quickly.
The former task features high-frequency behaviors that may induce aliasing, while the latter task allows the agent to leave the frame yielding transitions with no visual feedback.
We additionally provide converged performance results for pixel-based D4PG (Barth-Maron et al., 2018) and proprioception-based A3C (Mnih et al., 2016) at 10 8 environment steps to put the results into perspective. 5. B ENCHMARKING : EXPLORING SPARSE REWARD TASKS To further analyze exploration in sparse reward settings, we introduce three additional tasks. The
Cheetah Run and Walker Walk tasks are modiﬁed with a reward threshold below which rewards are zeroed ( 0 . 25 and 0 .
7 , respectively; rescaled to [0, 1]).
Whereas terminal performance on the dense task versions was similar, sparse reward feedback favors LOVE in Figure 4 (left). Particularly on the modiﬁed Cheetah Run task a strong separation can be observed. In the maze environment, a planar pointmass is controlled via continuous forces to navigate a maze in the absence of reward feedback. Motions need to arise from an intrinsic motivation to explore and we test for this property by letting the agent interact with the environment for 100 k timesteps.
The occupancy maps in Figure 4 (right) suggest that optimism over returns can set its own exploration goals in domains without reward feedback.
These observations further underline the potential of combining model ensembling for policy debiasing with optimism over inﬁnite-horizon returns for guided deep exploration.
[Figure]
300k steps
LOVE
LVE ∆ Dreamer + Curious
DrQ
A3C
D4PG
Cartpole Dense 676 ±644 ±433 ±550 ±781 ±558 ±862 ±
Cartpole Sparse 694 ±245 ±389 ±242 ±231 ±180 ±482 ±
Cheetah Run 699 ±690 ±658 ±715 ±533 ±214 ±524 ±
Finger Spin 569 ±640 ±335 ±299 ±898 ±129 ±986 ±
Hopper Hop 223 ±139 ±124 ±89 ±151 ±1 ±242 ±
Pendulum 635 ±865 ±576 ±168 ±399 ±49 ±681 ±
Walker Run 568 ±511 ±431 ±426 ±338 ±192 ±567 ±
Walker Walk 955 ±927 ±
[Table]
Best performance in bold.
Our method of policy debiasing via latent ensembling in combination with optimistic exploration improves performance over ∆ Dreamer on all tasks.
It further outperforms a variation of ∆ Dreamer that explores using a curiosity bonus and compares favourably to the concurrent DrQ agent.
Proprioceptive A3C and pixel-based D4PG after 10 8 steps for reference.
Cartpole and Pendulum refer to the Swingup tasks, while Cheetah Sparse and Walker Sparse refer to modiﬁed versions of the Run and Walk tasks with reward threshold 0 .
25 and 0 . 7 , respectively.  C ONCLUSION We propose latent optimistic value exploration (LOVE), a model-based deep reinforcement learning algorithm that leverages long-term optimism in directing exploration for continuous visual control.
LOVE leverages ﬁnite-horizon rollouts of a latent model ensemble in combination with value function estimates to predict long-term performance of candidate action sequences.
The ensemble predictions are then combined into an upper conﬁdence bound objective for policy training.
Training on this objective yields a policy that optimistically explores interactions that have the potential of improving task performance while ignoring interactions that are uncertain but tangential to task completion.
We evaluate LOVE experimentally regarding its exploration capabilities and performance on benchmarking tasks.
In the absence of reward signals, LOVE demonstrates an intrinsic motivation to explore interactions based on their information density.
Empirical results on various tasks from the DeepMind
Control Suite demonstrate LOVE’s competitive performance and ability to focus exploration on interactions that are conducive to the task.
LOVE demonstrates improved sample efﬁciency over the current state-of-the-art model-based Dreamer agent, and compared favourably to the concurrent model-free DrQ agent.
Future work will consider learning more efﬁcient representations without assigning model capacity to image reconstruction.
Application of concurrent methods in contrastive learning and data augmentation for reinforcement learning will likely further improve performance.
A P ARAMETERS AND IMPLEMENTATION
We use as single set of parameters throughout all experimental evaluations.
The general model architecture follows Hafner et al.
(2019a), where the variational autoencoder from Ha & Schmidhuber (2018) is combined with the RSSM from Hafner et al.
(2019b).
We extend their default parameters by the ensemble size M = 5 , the initial UCB trade-off parameter β ini = 0 . 0 , and the per-episode linear
UCB growth rate δ = 0 .
The learning rates for the model, the value function and the policy are 6 × 10 − 4 , 8 × 10 − 5 , 2 × 10 − 4 , respectively, and updates are computed with the Adam optimizer (Kingma & Ba, 2014).
Throughout all experiments, the online phase consists of 1000 environment interactions with an action repeat of 2 , while the ofﬂine phase consists of 200 learning updates.
An overview of the range of hyper-parameter values that were investigated is provided in Table 2.
Not all possible pairings were considered and suitable combinations were determined by inspection, while the best pairing was selected empirically.
Our implementation builds on Dreamer ( https: //github.com/danijar/dreamer ) and the remaining parameters are set to their default values.
Experiments were conducted on 4 CPU cores in combination with 1 GPU (NVIDIA V100).
[Table]
B N ETWORK A RCHITECTURES
The base network architectures employed throughout this paper are provided in Table 3.
Each particle is assigned a distinct instance of its associated models.
In the following, we brieﬂy comment on how the two parts of the transition model interact and provide further insights into the remaining models.
Transition model
The transition model follows the recurrent state space model (RSSM) architecture presented in Hafner et al.
(2019a;b).
The RSSM propagates model states consisting of a deterministic and a stochastic component, respectively denoted by s t,d and s t,s at time t .
The stochastic component s t,s is represented as a diagonal Gaussian distribution.
The transition model then leverages the imagine 1-step method to predict priors for the associated mean and standard deviation, ( µ priort,s , σ priort,s ), based on the previous model state and applied action.
In the presence of observations, the observe 1-step method can be leveraged to convert prior estimates into posterior estimates, ( µ post t,s , σ post t,s ).
The transition model may then propagate posteriors based on a context sequence using both the imagine 1-step and observe 1-step methods, from which interactions can be imagined by propagating prior estimates based on the imagine 1-step method.
Each particle uses a transition model that follows the presented network architecture, but possesses distinct parameters.
Encoder model The encoder parameterization follows the architectural choices presented in Ha &
Schmidhuber (2018).
The encoder generates embeddings based on 64 × 64 RGB image observations.
Observation model The observation model follows the decoder architecture presented in Ha &
Schmidhuber (2018).
The image observations are reconstructed from the associated model states s τ .
Reward and value model Rewards and values are both predicted as scalar values from fullyconnected networks that operate on the associated model states s τ , similar to Hafner et al. (2019a).
Each particle uses a pairing of a reward model and a value model with distinct sets of parameters.
Action model
The action model follows Hafner et al.
(2019a), where the mean µ a is rescaled and passed through a tanh to allow action saturation.
It is combined with a softplus standard deviation based on σ a and the resulting Normal distribution is squashed via a tanh (see Haarnoja et al.
[Table] [Table]
C P REDICTION UNCERTAINTY
We provide an illustrative visualization of how the prediction uncertainty in the ensemble evolves during model training.
The ensemble is provided with context from a sequence of 5 consecutive images and then predicts forward in an open loop fashion for 15 steps (preview horizon).
The ground truth sequence is compared to ensemble predictions after 10 , 150 , and 300 episodes of agent training.
Figures 5 and 6 show two different motion patterns for the Walker Walk task.
The motion in Figurecan be described as a regular walking pattern.
At the beginning of model training, the agent will have mostly observed itself falling to the ground and, in combination with a poorly trained policy, the ensemble predictions place the agent on the ground in a variety of conﬁgurations.
After 150 episodes, short-term uncertainty has been signiﬁcantly reduced, while considerable uncertainty remains at the end of the preview window.
After 300 episodes, the ensemble predictions align with the ground truth sequence.
The agent therefore focused on reducing uncertainty over this desirable motion pattern.
This can be contrasted with the results of Figure 6, where uncertainty over an irregular falling pattern remains even after 300 episodes.
The falling motion is undesirable, and while the ensemble predictions agree on a fall being imminent, no signiﬁcant amount of effort was spent on identifying exactly how the agent would fall.
We can observe similar results on the Cheetah Run task for a running motion pattern in Figure 7 and a falling motion pattern in Figure 8.
However, the lower complexity Cheetah dynamics seem to allow for more precise predictions than on the Walker task.
[Figure]
[Figure]
[Figure]
[Figure]
The regular walking pattern is well-explored and only induces little deviation in the ensemble.
This desirable and
[Figure]
Prediction
Ground Truth
Episode
Episode Episode
Ground Truth
Context - - - -
Prediction
[Figure]
Episode
Episode
[Figure]
Context - - - -
Prediction
Ground
[Figure] [Figure]
D B ASELINES
The baseline performance data for DrQ was taken from Kostrikov et al.
(2020), the ones for D4PG and A3C from Tassa et al.
(2018), while the data for Dreamer was generated by running the ofﬁcial
TensorFlow 2 implementation of Hafner et al. (2019a).
It should be noted that both DrQ and D4PG use 84 × 84 image observations, whereas LOVE and Dreamer use 64 × 64 image observations.
Larger resolution provides more ﬁne-grained information, which potentially translates to improved planning.
Furthermore, DrQ continuously reﬁnes its policy online, while the other algorithms only do so ofﬂine.
E B UGTRAP EXTENDED
We provide additional occupancy maps for the bug trap environment in Figure 9.
The environment provides no reward feedback and assesses the agent’s ability to actively search for informative feedback through intrinsic motivation.
Furthermore, the environment geometry makes exploration of the outside area difﬁcult.
In the absence of useful mean performance estimates, LOVE leverages uncertainty-guided exploration to query interactions.
This allows for escaping in 5 out of 6 trials and achieving the largest area coverage (column 2).
LVE does not leverage uncertainty estimates and only escapes during 3 trials (column 3), while displaying a highly reduced area coverage (rows 1 and 3).
Similarly, random exploration allows the Dreamer agent to only escape in 2 instances (column 4).
F A BLATION STUDY : D REAMER
We compare performance to ∆ Dreamer, a variation that uses our changes to the default parameters.
Figure 10 indicates that performance improves on several tasks, while deteriorating on Finger Spin.
LOVE outperforms ∆ Dreamer on the majority of tasks.
It can thus be concluded that increased information propagation generally affects performance favourably.
However, relying on a single model can propagate simulation bias into the policy and in turn impede efﬁcient learning.
This could serve as an explanation for the unchanged performance on the not fully observable Cartpole Swingup tasks, as well as the deteriorating performance on the high-frequency Finger Spin task
[Figure]
The changes improve performance of Dreamer on some environments, while signiﬁcantly decreasing performance on outperforms ∆ G A BLATION STUDY : PLANNING HORIZON We investigate increasing the planning horizon used during latent imagination. Longer horizons shift performance estimation from values towards rewards, which can be advantageous when the value function has not been sufﬁciently learned.
Prediction quality over long horizons relies on accurate dynamics rollouts.
Figure 11 indicates that on average the longer horizon reduces performance. [Figure] i [Figure]
o
[Figure]
t u r n
[Figure]
i
[Figure]
t u r n
[Figure]
H A BLATION STUDY : β SCHEDULE
We investigate variations of the beta schedule, initializing either with a negative (initially pessimistic) or a positive value (initially optimistic).
The former variation penalizes uncertainty in the beginning and then transitions to become optimistic, while the latter seeks out uncertainty from the start.
Based on Figure 12, we notice that terminal performance is mostly similar.
The initially pessimistic agent exhibits reduced performance on the sparse pendulum, where it only explores well after it transitions to optimism (Episode 100)
[Figure]
and improved performance on the challenging Hopper task
[Figure]
where initial pessimism potentially guards against local optima.
Our choice of parameters tries to mitigate unfounded optimism during initialization (initial value 0) [Figure] while encouraging exploration throughout the course of training (linear increase).
Here, we chose the same values for all tasks
[Figure]
but one could imagine task-speciﬁc choices (negative beta for safe-RL, positive beta for optimistic exploration)
[Figure]
I A BLATION STUDY : ENSEMBLE SIZE
We investigate variation of the ensemble size.
A smaller ensemble generates uncertainty estimates that are more susceptible to bias in the individual ensemble members and may even generate misleading estimates.
Figure 13 demonstrates that a smaller ensemble (M=2) impacts performance unfavorably
[Figure]