Lagrangian Model Based Reinforcement Learning
Abstract
We are interested in reinforcement learning (RL) for physical systems.
One of the drawbacks of traditional RL algorithms has been their poor sample efﬁciency.
One approach to improve it is model-based RL.
In our algorithm, we learn a model of the environment, essentially its transition dynamics and reward function, use it to generate imaginary trajectories and backpropagate through them to update the policy, exploiting the differentiability of the model.
Intuitively, learning more accurate models should lead to better performance.
Recently, there has been growing interest in developing better deep neural network based dynamics models for physical systems, through better inductive biases.
We focus on systems undergoing rigid body motion.
We compare two versions of our model-based RL algorithm, one which uses a standard deep neural network based dynamics model and the other which uses a Lagrangian Neural Network based dynamics model, which utilizes the structure of the underlying physics.
We ﬁnd that, in environments that are not sensitive to initial conditions, both versions achieve similar average-return, while the physics-informed version achieves better sample efﬁciency.
Whereas, in environments that are sensitive to initial conditions, the physics-informed version achieves signiﬁcantly better average-return and sample efﬁciency.
In these latter environments, our physics-informed model-based RL approach achieves better average-return than Soft Actor-Critic, a state-of-the-art model-free RL algorithm.
Introduction
We are interested in reinforcement learning (RL) for physical systems.
RL can solve sequential decision making problems through trial and error.
In recent years, RL has been combined with powerful function approximators such as deep neural networks to solve complex continuous control tasks (Lillicrap et al., 2015; Schulman et al., 2017; Fujimoto et al., 2018;
Haarnoja et al., 2018).
However, there remain some critical challenges in RL today (Dulac-Arnold et al., 2021).
One of them is sample efﬁciency.
Traditional RL algorithms require a lot of interactions with the environment to learn successful policies.
In the case of physical systems, collecting such large amounts of training data using actual hardware is not practical.
One solution is to train in simulation.
However, for many physical systems, developing realistic simulations is hard.
Another solution is to improve the sample efﬁciency.
One approach to improve it is model-based RL, where we learn a model of the environment, essentially its transition dynamics and reward function, use it to generate imaginary trajectories and use that data to update the policy.
This way we need fewer interactions with the actual environment.
Some model-based RL algorithms use the model just to generate additional data and update the policy using a model-free algorithm (Sutton, 1991; Janner et al., 2019).
Other model-based RL algorithms exploit the differentiability of the model and backpropagate through the imaginary trajectories to update the policy (Deisenroth and Rasmussen, 2011; Heess et al., 2015;
Clavera et al., 2020; Hafner et al., 2019, 2020).
We adopt the second approach.
Intuitively, learning more accurate models should lead to better performance.
Recently there has been growing interest in developing better deep neural network based dynamics models for physical systems through better inductive biases (Lutter et al., 2019a; Greydanus et al., 2019;
Lutter et al., 2019b; Zhong et al., 2019, 2020; Cranmer et al., 2020; Finzi et al., 2020; Zhong et al., 2021a).
These physics-informed models predict the future states more accurately compared to standard deep neural networks.
They also obey the underlying physical laws, such as conservation of energy better.
Previous studies in this area have mostly focused on improving dynamics learning.
Some studies have learnt a physics-informed model and used it for inverse dynamics control (Lutter et al., 2019a) and energy based control (Lutter et al., 2019b; Zhong et al., 2019).
We focus on systems undergoing rigid body motion.
The environments considered are shown in Figure 1.
In this work, we assume that there is no friction or contacts.
In future work, we plan to include these effects as well.
We provide more details about the environments, such as the task to be accomplished, which joints are acutated, etc, in Appendix A.
We develop our own simulations from ﬁrst principles.
[Figure]
We compare two versions of our model-based RL algorithm, one which uses a standard deep neural network based dynamics model and the other which uses a Lagrangian Neural Network based dynamics model, which utilizes the structure of the underlying physics.
Our contributions are as follows,
• Our ﬁrst contribution is combining physics-informed models and model-based RL.
To the best of our knowledge, we are the ﬁrst to do so.
• Our second contribution is showing in which environments we can signiﬁcantly beneﬁt by learning physics-informed models.
In environments that are not sensitive to initial conditions, both versions of our algorithm achieve similar average-return, while the physics-informed version achieves better sample efﬁciency.
Whereas, in environments that are sensitive to initial conditions, the physics-informed version achieves signiﬁcantly better average-return and sample efﬁciency.
Thus, in environments that are sensitive to initial conditions, we can signiﬁcantly beneﬁt by learning physics-informed models.
There are two factors behind this, one is that numerical errors accumulate fast in these environments, and the other is that while standard deep neural networks struggle to learn accurate dynamics models in these environments, physics-informed models are able to do so due to their inductive biases.
Sensitivity to initial conditions is characterized by a large ﬁnite-time maximal Lyapunov exponent.
The main factors that cause it are underactuation, which leads to relatively unrestrained motion, and the system’s high inherent sensitivity to initial conditions, which is characterized by a large long-term maximal Lyapunov exponent under free fall.
• Our third contribution is showing when model accuracy matters in model-based RL, in general.
We show that model accuracy matters a lot in environments that are sensitive to initial conditions, as numerical errors accumulate fast in these environments.
Whereas, model accuracy matters only to some extent in environments that are not sensitive to initial conditions, as numerical errors accumulate slowly in these environments.
• Our fourth contribution is showing that physics-informed model-based RL can exceed the asymptotic performance of model-free RL.
We compare our model-based approaches against Soft Actor-Critic (SAC) (Haarnoja et al., 2018), a state-of-the-art model-free RL algorithm.
In the environments that are sensitive to initial conditions, due to factors such as underactuation, high variance in the agent trajectories and poor predictability, we require a lot of samples to learn.
SAC
is affected by this issue, while our model-based RL approaches overcome this issue by generating sufﬁcient imaginary data.
Hence, our physics-informed model-based RL approach achieves better average-return than SAC in these environments.
Model-Based RL
Our model-based RL algorithm essentially iterates over three steps.
First is the environment interaction step, where we use the current policy to interact with the environment and gather data.
Second is the model learning step, where we use the gathered data to learn the dynamics and reward models.
Third is the behaviour learning step, where we use the learned model to generate imaginary trajectories and backpropagate through them to update the policy.
We discuss the model learning and behaviour learning steps in detail below.
2.1 Model Learning
In the model learning step, we learn the dynamics and reward models.
In dynamics learning, we want to predict the next state, given the current state and action.
We model these systems using
Lagrangian mechanics.
Hence, the state consists of generalized coordinates q , which describe the conﬁguration of the system, and generalized velocities ˙ q , which are the time derivatives of q .
The action is the motor torque τ .
So, in dynamics learning, we essentially want to learn the transformation ( q t , ˙
q t , τ t )
→ ( q t +1 , ˙ q t +1 ) .
The most straightforward solution is to train a standard deep neural network.
We refer to this approach as DNN.
This is shown in Figure 2(a).
Another approach is to utilize the structure of the underlying Lagrangian mechanics.
This approach builds upon recent work such as Deep Lagrangian Networks (DeLaN) (Lutter et al., 2019a), DeLaN for energy control (Lutter et al., 2019b) and Lagrangian Neural Networks (Cranmer et al., 2020).
We detail this approach here.
In Lagrangian mechanics, the Lagrangian is a scalar quantity deﬁned as
L ( q , ˙ q , t ) = T ( q , ˙ q ) − V ( q ) , where T ( q , ˙ q ) is the kinetic energy and V ( q ) is the potential energy.
The Lagrangian equations of motion are given by, d dt ∂ L ∂ ˙
q − ∂ L ∂ q = τ .
For systems undergoing rigid body motion, the kinetic energy is 12 ˙ q T M ( q ) ˙ q , where M ( q ) is the mass matrix, which is symmetric and positive deﬁnite.
Hence, the Lagrangian equations of motion become,
[Equation]
Here, C ( q , ˙ q ) ˙ q represents the centripetal / Coriolis forces and G ( q ) represents the conservative forces (e. g., gravity).
We use one network to learn the potential energy function V ( q ) and another network to learn a lower triangular matrix L ( q ) , using which we compute the mass matrix as
M ( q ) =
L ( q ) L T ( q ) .
We then compute C ( q , ˙ q ) ˙
q and G ( q ) as shown in Equation 1.
Rearranging
Equation 1, we get the acceleration as ¨ q =
M − 1 ( q ) ( τ − C ( q , ˙ q ) ˙
q − G ( q )) .
We then numerically integrate the state derivative (˙ q , ¨ q ) over one time step using second-order Runge-Kutta to compute the next state.
We refer to this approach as LNN, short for Lagrangian Neural Network.
The entire process is shown in Figure 2(b).
In both the DNN and LNN approaches to dynamics learning, we use the L1 error between the predicted state and the ground truth as the loss for training.
In reward learning, we want to learn the reward function.
In general, the reward is a function of the current state, action and the next state.
In our case, the reward only depends on the next state.
Hence, we train a network to map the next state to the reward.
We use the L1 error between the predicted reward and the ground truth as the loss for training.
2.2 Behaviour Learning
In the behaviour learning step, we use the learned model to generate imaginary trajectories and backpropagate through them to update the policy.
We build upon the Dreamer algorithm (Hafner et al., 2019, 2020).
We adopt an actor-critic approach.
The critic aims to predict the expected discounted return from a given state.
We train the critic to regress the λ -return (Schulman et al., 2015).
We stabilize the critic training by computing the λ -return using a target network that is updated every [Figure] critic updates.
The critic loss function is given by L ( w ) =
E [ [Equation] w [Equation]
− sg [Equation]
V
[Equation]
λ
[Equation]
)) ) 2 ] , where,
[Equation]
We stop the gradients around the target (denoted by the sg ( . ) function), as is typical in the literature.
We use a stochastic actor.
The actor aims to output actions that lead to states that maximize the expected discounted return.
We train the actor to maximize the same λ -return that was computed to train the critic.
We add an entropy term to the actor objective to encourage exploration.
The overall actor loss function is given by L ( θ ) = − E
[ (cid:80) T − 1 t =
[ V (cid:48) λ ( s t ) − η log π
[Equation] | s t ; θ ) ]] .
To backpropagate through sampled actions, we use the reparameterization trick (Kingma and Welling, 2013).
The actor network outputs the mean µ and standard deviation σ of a Gaussian distribution, from which we obtain the action as a t = tanh ( µ θ ( s t ) + σ θ ( s t ) · (cid:15) ) , where (cid:15) ∼ N (0 , I ) .
We summarize our overall model-based RL algorithm in Algorithm 1.
Initialize networks with random weights.
Execute random actions for K episodes to initialize replay buffer.
for each episode do //
Model Learning for N times do
Draw mini batch of transitions from replay buffer.
Fit dynamics and reward models.
end //
Behaviour Learning for N 2 times do
Draw mini batch of states from
From state, imagine a
Backpropagate to , critic. end //
Environment Interaction Interact with environment for one using
Add to buffer. end
2.3 Experiments
We train two versions of our model-based RL algorithm, one which uses the DNN approach for dynamics learning and the other which uses the LNN approach.
We refer to them as MBRL-DNN and MBRL-LNN respectively.
In both versions, we use an imagination horizon of 16 time steps.
In addition, we train a state-of-the-art model-free RL algorithm, Soft Actor-Critic (SAC) (Haarnoja et al., 2018), to serve as a baseline.
We train each algorithm on ﬁve random seeds.
Results and Analysis
We record the results from the experiments in Table 1.
The training curves are shown in Figure 3.
We summarize the results below,
• Across environments, MBRL-LNN achieves signiﬁcantly lower dynamics error than MBRL-DNN.
The reward error for both methods are similar.
In Reacher, Pendulum and Cartpole, all the methods, i e., MBRL-DNN, MBRL-LNN and SAC, successfully solve the task and achieve similar average-return.
• In Cart-2-pole, again, all the methods successfully solve the task.
However, in terms of averagereturn, MBRL-LNN > SAC > MBRL-DNN.
In Acrobot, Cart-3-pole and Acro-3-bot, only MBRL-LNN and SAC successfully solve the task, while MBRL-DNN is unsuccessful.
Again, in terms of average-return, MBRL-LNN > SAC > MBRL-DNN.
• Across environments, MBRL-LNN achieves similar or better average return than MBRL-DNN and SAC, while requiring fewer samples.
In other words, MBRL-LNN achieves better sample efﬁciency.
We try to understand the results better.
Reacher, Pendulum and Cartpole appear to be simple environments where all methods perform well.
Cart-2-pole, Acrobot, Cart-3-pole and Acro-3-bot appear to be more challenging environments, where there is a noticeable difference in the performance of different methods.
We try to understand what makes these latter environments more challenging and why certain methods perform better in these environments than others.
3.1 Environment Dynamics
As a ﬁrst step, we try to characterize the underlying dynamics of each environment by calculating their Lyapunov exponents.
Lyapunov exponents measure the rate of separation of trajectories which start from nearby initial states.
If the initial separation vector is u 0 and the separation vector at time t is u t , then the Lyapunov exponent is deﬁned as λ = 1 t log (cid:107) u t (cid:107) (cid:107) u 0 (cid:107) .
The rate of separation varies based on the orientation of the initial separation vector.
In general, there is a spectrum of Lyapunov exponents, equal in number to the state space dimension.
It is common to refer to the largest among them as the maximal Lyapunov exponent (MLE).
The rate of separation is maximum along the direction [Table] associated with the MLE.
An arbitrary initial separation vector will typically contain some component in this direction, and because of the exponential growth rate, this component will dominate the evolution.
The MLE is a measure of the system’s sensitivity to initial conditions.
It also represents how quickly numerical errors will accumulate.
Thus, it is also a measure of the predictability.
Finite-Time MLE :
We are interested in knowing how nearby trajectories diverge over ﬁnite-time, under a trained RL policy.
Hence, we compute the ﬁnite-time MLE.
We consider the period from the start of an episode, till the agent reaches the goal state, which typically takes several hundred time steps.
The simplest method to compute the ﬁnite-time MLE is to evolve two trajectories starting from nearby initial states.
A more efﬁcient and reliable method is to use the so-called variational equation (Skokos, 2010), which linearizes the dynamics to explicitly compute how separation vectors evolve with time.
The original variational equation is deﬁned only for autonomous systems.
We modify it to accomodate a control policy.
Let the dynamics in the presence of control inputs be described by ˙ s
= f
( s , a ) , where a = π ( s ) .
The modiﬁed variational equation is given by, ˙ u
= d f d s u
= ( ∂ f ∂ s + ∂ f ∂ a d a d s ) u .
We consider a random initial separation vector of unit length.
We then jointly integrate the system dynamics along with the modiﬁed variational equation to compute how the separation vector evolves with time.
We renormalize the separation vector periodically and each time, use the Lyapunov exponent deﬁnition to compute a fresh estimate of the ﬁnite-time MLE and update its running average.
We continue this process until the agent reaches the goal state.
We compute the ﬁnite-time MLE for policies of MBRL-DNN, MBRL-LNN and SAC, and average the results.
We record the results in
Table 1.
We ﬁnd that Reacher, Pendulum and Cartpole have a relatively small, negative, ﬁnite-time
MLE.
This implies that they are less sensitive to initial conditions and more predictable.
Whereas,
Cart-2-pole, Acrobot, Cart-3-pole and Acro-3-bot have a relatively large, positive, ﬁnite-time MLE.
This implies that they are more sensitive to initial conditions and less predictable.
The ﬁnite-time
MLE is a function of the system dynamics as well as the policy.
Free Fall MLE : To get an idea of the system’s inherent sensitivity to initial conditions, independent of any control policy, we compute the long-term MLE under free fall.
We call this the free fall MLE.
We follow the same procedure we used to compute the ﬁnite-time MLE, with some minor changes.
One change is that, we use the original variational equation.
Let the dynamics in the absence of control inputs be described by ˙ s
= f ( s ) .
Then, the variational equation is given by ˙ u
= d f d s u .
The other change is that, we continue the MLE computation process for a sufﬁciently long time, until the running average converges.
We record the results in Table 1.
We ﬁnd that Reacher, Pendulum and Cartpole have a relatively small, positive, free fall MLE, implying low inherent sensitivity to initial conditions.
Whereas, Cart-2-pole, Acrobot, Cart-3-pole and Acro-3-bot have a relatively large, positive, free fall MLE, implying high inherent sensitivity to initial conditions.
3.2 Actuation
In Appendix A, we describe which joints are acutated in each environment.
Reacher and Pendulum are fully actuated.
The rest of the environments are underactuated.
The level of underactuation varies.
Cartpole and Cart-2-pole are somewhat less underactuated, while Acrobot, Cart-3-pole and
Acro-3-bot are somewhat more underactuated.
Underactuation makes it harder to learn successful policies.
It also leads to relatively unrestrained motion.
3.3 Imaginary Trajectories
Next, we assess the quality of the imaginary trajectories produced by MBRL-DNN and MBRL-LNN.
We generate 10,000 imaginary trajectories of length 16 time steps and also generate the corresponding ground truth trajectories.
First, we assess the accuracy of the imaginary trajectories.
For each imaginary trajectory, we compute the state error (L1 error between predicted state and true state) at each time step.
We also compute the trajectory error, which is the sum of the state errors along the trajectory.
We plot the average state error as a function of time in Figure 4.
We record the average trajectory error in Table 1.
We ﬁnd that, MBRL-LNN has a low trajectory error across environments.
MBRL-DNN’s trajectory error varies.
In Reacher, Pendulum, Cartpole and Cart-2-pole, it has a moderate trajectory error.
In Acrobot, Cart-3-pole and Acro-3-bot, it has a high trajectory error.
Across environments, MBRL-LNN has lower trajectory error than MBRL-DNN, i. e, MBRL-LNN’s imaginary trajectories are more accurate.
Next, we assess how well the imaginary trajectories respect the underlying physical laws, such as conservation of energy.
In our environments, the work done by external non-conservative forces, i. e, the actuators, is non-zero.
Hence, the total energy of the system will change.
The change in total energy must equal the work done by external non-conservative forces, for energy to be conserved.
We deﬁne the energy error as the absolute difference between the two quantities.
For each imaginary trajectory, we compute the energy error at each time step.
We plot the average energy error as a function of time in Figure 5.
We ﬁnd that, across environments, MBRL-LNN has lower energy error than MBRL-DNN, i. e, MBRL-LNN’s imaginary trajectories conserve energy better.
3.4 Discussion
Cart-2-pole, Acrobot, Cart-3-pole and Acro-3-bot are underactuated and have high inherent sensitivity to initial conditions, which is characterized by a large free fall MLE.
Underactuation makes it harder to learn successful policies.
It also leads to relatively unrestrained motion.
Such unrestrained motion in a system with a large free fall MLE leads to sensitivity to initial conditions, which is characterized by a large ﬁnite-time MLE.
Sensitivity to initial conditions leads to high variance in the agent trajectories.
It also leads to poor predictability, which affects critic learning, as it is concerned with predicting the discounted return, and hence also affects actor learning, as our methods follow an actor-critic approach.
Under these conditions, we need a lot of samples to learn.
SAC is successful in all four of these environments.
However, it is affected by the issue of a lot of samples being needed to learn.
MBRL-DNN and MBRL-LNN overcome this issue by generating a lot of imaginary data, which allows them to perform many more policy updates.
This is the main reason why SAC achieves lower average-return than MBRL-LNN in these environments.
Sensitivity to initial conditions makes it challenging for standard deep neural networks to learn accurate dynamics models.
In Acrobot, Cart-3-pole and Acro-3-bot, MBRL-DNN has a large dynamics error.
As numerical errors accumulate fast in these environments, its imaginary trajectories have high trajectory error.
Hence, it is unable to learn good policies and is unsuccessful in these environments.
Cart-2-pole is a borderline case.
Here, MBRL-DNN achieves a moderate dynamics error.
Hence, even though numerical errors accumulate fast, its imaginary trajectories only have moderate trajectory error.
Hence, it is able to learn a reasonably good policy and is successful.
In all four of these environments, MBRL-LNN achieves a small dynamics error due to its inductive biases.
Hence, even though numerical errors accumulate fast, its imaginary trajectories have low trajectory error.
Hence, it is able to learn good policies and is successful in these environments.
Thus,
MBRL-LNN achieves better average-return and sample efﬁciency than both MBRL-DNN and SAC in these environments, because it is able to generate a lot of high quality imaginary data and perform many policy updates using it.
In contrast, Reacher, Pendulum and Cartpole are well-actuated and/or have low inherent sensitivity to initial conditions, which is characterized by a small free fall MLE.
Hence, they are less sensitive to initial conditions, which is characterized by a small ﬁnite-time MLE.
In these environments,
MBRL-LNN has a low dynamics error, MBRL-DNN has a moderate dynamics error and numerical errors accumulate slow.
Hence, MBRL-LNN has a low trajectory error, MBRL-DNN has a moderate trajectory error and both methods are able to learn good policies and are successful, and so is SAC.
In these environments, MBRL-LNN achieves similar average-return to MBRL-DNN and SAC, but achieves better sample efﬁciency.
Conclusion
We apply model-based RL to systems undergoing rigid body motion.
In our algorithm, we learn a model of the environment, use it to generate imaginary trajectories and backpropagate through them to update the policy.
We compare two versions of our algorithm, one which uses a standard deep neural network based dynamics model and the other which uses a Lagrangian Neural Network based dynamics model, which utilizes the structure of the underlying physics.
In environments that are not sensitive to initial conditions, both versions of our algorithm achieve similar average-return, while the physics-informed version achieves better sample efﬁciency.
Whereas, in environments that are sensitive to initial conditions, the physics-informed version achieves signiﬁcantly better averagereturn and sample efﬁciency.
Thus, in environments that are sensitive to initial conditions, we can signiﬁcantly beneﬁt by learning physics-informed models.
There are two factors behind this, one is that numerical errors accumulate fast in these environments, and the other is that while standard deep neural networks struggle to learn accurate dynamics models in these environments, physics-informed models are able to do so due to their inductive biases.
Sensitivity to initial conditions is characterized by a large ﬁnite-time maximal Lyapunov exponent.
The main factors that cause it are underactuation, which leads to relatively unrestrained motion, and the system’s high inherent sensitivity to initial conditions, which is characterized by a large long-term maximal Lyapunov exponent under free fall.
We show when model accuracy matters in model-based RL, in general.
We show that model accuracy matters a lot in environments that are sensitive to initial conditions, as numerical errors accumulate fast in these environments.
Whereas, model accuracy matters only to some extent in environments that are not sensitive to initial conditions, as numerical errors accumulate slowly in these environments.
We show that physics-informed model-based RL can exceed the asymptotic performance of modelfree RL.
We compare our model-based approaches against Soft Actor-Critic (SAC), a state-of-the-art model-free RL algorithm.
In the environments that are sensitive to initial conditions, due to factors such as underactuation, high variance in the agent trajectories and poor predictability, we require a lot of samples to learn.
SAC is affected by this issue, while our model-based RL approaches overcome this issue by generating sufﬁcient imaginary data.
Hence, our physics-informed model-based RL approach achieves better average-return than SAC in these environments.
Future Work
In future work, we plan to move towards robotics tasks.
We plan to consider friction, as it is present in most real world robots.
As far as learning friction, there are two approaches.
In the black box approach, the friction is learnt directly from data using a neural network (Zhong et al., 2020).
In the white box approach, an analytic friction model is used and its parameters alone are learnt from data (Lutter et al., 2019b).
We plan to pursue the white box approach.
In future work, we also plan to consider contact dynamics, as it is present in many robotics tasks like manipulation and locomotion.
There are predominantly two approaches to simulate contact dynamics, the linear complementarity approach (LCP)
(Stewart and Trinkle, 1996; Anitescu and Potra, 1997) and the convex optimization approach (Todorov et al., 2012).
LCP was the standard approach for a long time.
Convex optimization has similar accuracy and is more efﬁcient.
We plan to adopt the latter approach and use differentiable convex optimization layers (Agrawal et al., 2019) to learn differentiable contact models which can extend LNNs.
There has been some recent work in this direction by Zhong et al., 2021b.
However they assume that the robot’s environment is known and use it to compute the active contacts and their jacobians.
We plan to do away with this assumption by equipping the robot with sensors to perceive its environment.
Upon learning a physics-informed model that captures rigid body dynamics, friction and contact dynamics, we plan to train a model-based RL algorithm.
Appendix A Environment Details
The task to be accomplished in each environment is as follows,
• Reacher : Control a fully-actuated two-link manipulator in the horizontal plane to reach a ﬁxed target location.
• Pendulum : Swing up and balance a simple pendulum.
• Cartpole : Swing up and balance an unactuated pole by applying forces to a cart at its base.
• Cart-2-pole :
One extra pole is added to Cartpole.
Again, only the cart is actuated.
• Acrobot : Control a two-link manipulator to swing up and balance.
Only the second pole is actuated.
• Cart-3-pole : One extra pole is added to Cart-2-pole.
Only the cart and the third pole are actuated.
• Acro-3-bot : One extra pole is added to Acrobot.
Only the ﬁrst and the third poles are actuated.
In all the environments, there are no terminal states.
Each episode consists of 1000 time steps.
The total reward over an episode is in the range [0 , 1000] .
B Implementation Details
Table 2 and Table 3 list the hyperparameters and network architectures used in our model-based RL experiments.
For SAC, we use the same hyperparameters and network architectures as the original
SAC paper (Haarnoja et al., 2018)
[Figure]
[Table]
C Long-Horizon Imaginary Trajectories
Although during behaviour learning we use an imagination horizon of only 16 time steps, in Figuresand 7, we plot the average state error and energy error as a function of time for imaginary trajectories of horizon 100 time steps, to see how far into the future our models can predict accurately.
We ﬁnd that, across environments, MBRL-LNN is able to accurately predict quite far into the future, while
MBRL-DNN is unable to.