Lagrangian Model Based Reinforcement Learning
Abstract
We apply reinforcement learning (RL) to robotics tasks.
One of the drawbacks of traditional RL algorithms has been their poor sample efﬁciency.
In robotics, collecting large amounts of training data using actual robots is not practical.
One approach to improve the sample efﬁciency is model-based RL.
In our model-based
RL algorithm, we learn a model of the environment, essentially its transition dynamics and reward function, use it to generate imaginary trajectories and backpropagate through them to update the policy, exploiting the differentiability of the model.
Intuitively, learning better environment models should lead to better model-based RL performance.
Recently, there has been growing interest in developing better deep neural network based dynamics models for physical systems, through better inductive biases.
We focus on robotic systems undergoing rigid body motion.
We utilize the structure of rigid body dynamics to learn a Lagrangian Neural Network and use it to train our model-based RL algorithm.
To the best of our knowledge, we are the ﬁrst to explore such an approach.
While it is intuitive that such physics-informed dynamics models will improve model-based RL performance, it is not apparent in which environments we will signiﬁcantly beneﬁt from such an approach.
We show that, in environments with underactuation and high inherent sensitivity to initial conditions, characterized by a large maximal Lyapunov exponent under freefall, we are likely to signiﬁcantly beneﬁt from learning physics-informed dynamics models.
In such environments, our Lagrangian model-based RL approach achieves better average-return and sample efﬁciency than a version of our model-based RL algorithm that uses a standard deep neural network based dynamics model, as well as Soft Actor-Critic, a state-of-the-art model-free RL algorithm.
Introduction
We apply reinforcement learning (RL) to robotics tasks.
RL can solve sequential decision making problems through trial and error.
In recent years, RL has been combined with powerful function approximators such as deep neural networks to successfully solve complex robotics tasks with high-dimensional, continuous state and action spaces (Lillicrap et al., 2015; Schulman et al., 2017;
Haarnoja et al., 2018a; Fujimoto et al., 2018; Haarnoja et al., 2018b; Andrychowicz et al., 2020).
However, there remain some critical challenges that must be addressed, to take more RL based robotic systems to the real world (Dulac-Arnold et al., 2021).
One of these challenges is sample efﬁciency.
One of the drawbacks of traditional RL algorithms has been their poor sample efﬁciency – they require a large number of interactions with the environment to learn successful policies.
In robotics, collecting such large amounts of training data using actual robots is not practical.
One solution is to train in simulation.
However, in many robotics tasks, developing realistic simulations is hard.
Another solution is to improve the sample efﬁciency.
One approach to improve the sample efﬁciency is model-based RL, where we learn a model of the environment, essentially its transition dynamics and reward function, use it to generate imaginary trajectories and use them to update the policy.
In our model-based RL algorithm, we exploit the differentiability of the model and backpropagate through the imaginary trajectories to update the policy, similar to Dreamer (Hafner et al., 2019, 2020).
Intuitively, learning better environment models should lead to better model-based RL performance.
Recently there has been growing interest in developing better deep neural network based dynamics models for physical systems through better inductive biases (Lutter et al., 2019a; Greydanus et al., 2019;
Lutter et al., 2019b; Zhong et al., 2019, 2020; Cranmer et al., 2020; Finzi et al., 2020;
Zhong et al., 2021a).
These physics-informed models predict the future states more accurately compared to standard deep neural networks.
They also obey the underlying physical laws, such as conservation of energy better.
Previous studies in this area have mostly focused on improving dynamics learning.
Some studies have learnt a physics-informed dynamics model and used it for inverse dynamics control (Lutter et al., 2019a) and energy based control (Lutter et al., 2019b; Zhong et al., 2019).
We learn a physics-informed dynamics model and use it to train a model-based RL algorithm.
To the best of our knowledge, we are the ﬁrst to explore such an approach.
We focus on robotic systems undergoing rigid body motion.
We utilize the structure of rigid body dynamics to learn a Lagrangian Neural Network (Lutter et al., 2019a,b; Cranmer et al., 2020) and use it to train our model-based RL algorithm.
While it is intuitive that such physics-informed dynamics models will improve model-based RL performance, it is not apparent in which environments we will signiﬁcantly beneﬁt from such an approach.
We show that, in environments with underactuation and high inherent sensitivity to initial conditions, characterized by a large maximal Lyapunov exponent under freefall, we are likely to signiﬁcantly beneﬁt from learning physics-informed dynamics models.
In such environments, our Lagrangian model-based RL approach achieves better average-return and sample efﬁciency than a version of our model-based RL algorithm that uses a standard deep neural network based dynamics model, as well as Soft Actor-Critic (Haarnoja et al., 2018b), a state-of-the-art model-free RL algorithm.
Environments Considered
We focus on robotic systems undergoing rigid body motion.
The environments considered are shown in Figure 1.
In this work, we assume that there is no friction or contacts.
In future work, we plan to include these effects as well.
We provide more details about the environments, such as the task to be accomplished, which joints are acutated, etc, in Appendix A.
We develop our own simulations from ﬁrst principles.
[Figure]
Model-Based RL
In model-based RL, the training essentially iterates over three steps.
First is the environment interaction step, where we use the current policy to interact with the environment and gather data.
Second is the model learning step, where we use the gathered data to learn the dynamics and reward models.
Third is the behaviour learning step, where we use the learned model to generate imaginary trajectories and use them to update the policy.
We discuss the model learning and behaviour learning steps in detail below.
3.1 Model Learning
In the model learning step, we learn the dynamics and reward models.
In dynamics learning, we want to predict the next state, given the current state and action.
We model these systems using
Lagrangian mechanics.
Hence, the state consists of generalized coordinates q , which describe the conﬁguration of the system, and generalized velocities ˙ q , which are the time derivatives of q .
The action is the motor torque τ .
So, in dynamics learning, we essentially want to learn the transformation ( q t , ˙
q t , τ t )
→ ( q t +1 , ˙ q t +1 ) .
The most straightforward solution is to train a standard deep neural network.
We refer to this approach as DNN.
This is shown in Figure 2(a).
Another approach is to utilize the structure of the underlying Lagrangian mechanics.
This approach builds upon recent work such as Deep Lagrangian Networks (DeLaN) (Lutter et al., 2019a), DeLaN for energy control (Lutter et al., 2019b) and Lagrangian Neural Networks (Cranmer et al., 2020).
We detail this approach below.
In Lagrangian mechanics, the Lagrangian is a scalar quantity deﬁned as L ( q , ˙ q , t ) =
T ( q , ˙ q ) − V ( q ) , where T ( q , ˙ q ) is the kinetic energy and V ( q ) is the potential energy.
The Lagrangian equations of motion are given by,
[Equation]
For systems undergoing rigid body motion, the kinetic energy is given by T ( q , ˙ q ) =
[Equation]
˙
[Equation]
) ˙
q , where M ( q ) is the mass matrix, which is symmetric and positive deﬁnite.
Hence, the Lagrangian is given by L ( q , ˙ q , t )
= 1 2 ˙ q T M ( q ) ˙
q − V ( q ) .
Substituting this expression into Equation 1, we get the rigid body equations of motion,
[Equation]
Here, C ( q , ˙ q ) ˙ q represents the centripetal / Coriolis forces and G ( q ) represents the conservative forces (e. g., gravity).
Rearranging Equation 2 we get,
[Equation]
By integrating the state derivative (˙ q , ¨ q ) over one time step, we can obtain the next state, (cid:20)
[Equation]
We use one network to learn the potential energy function V ( q ) and another network to learn a lower triangular matrix L ( q ) , using which we compute the mass matrix as M ( q )
=
L ( q ) L T ( q ) .
We then compute C ( q , ˙ q ) ˙
q and G ( q ) as per Equation 2 and ¨ q as per Equation 3.
Then, as per Equation 4, we integrate (˙ q , ¨ q ) over one time step, numerically, using second-order Runge-Kutta, to compute the next state.
We refer to this approach as LNN, short for Lagrangian Neural Network.
The entire process is shown in Figure 2(b).
In both the DNN and LNN approaches to dynamics learning, we use the L1 error between the predicted state and the ground truth as the loss for training.
In reward learning, we want to learn the reward function.
In general, the reward is a function of the current state, action and the next state.
In our case, the reward only depends on the next state.
Hence, we train a network to map the next state to the reward.
We use the L1 error between the predicted reward and the ground truth as the loss for training.
[Figure]
3.2 Behaviour Learning
In the behaviour learning step, we use the learned model to generate imaginary trajectories and use them to update the policy.
Some model-based RL algorithms use the model just to generate additional data.
To update the policy, they use a model-free algorithm.
Algorithms such as Dyna (Sutton, 1991) and MBPO (Janner et al., 2019) adopt this approach.
Other model-based RL algorithms exploit the differentiability of the model and backpropagate through the imaginary trajectories to update the policy.
Algorithms such as PILCO (Deisenroth and Rasmussen, 2011), SVG (Heess et al., 2015),
Model-Augmented Actor-Critic (Clavera et al., 2020) and Dreamer (Hafner et al., 2019, 2020) adopt this approach.
We follow the latter approach.
We build upon the Dreamer algorithm (Hafner et al., 2019, 2020).
We adopt an actor-critic approach.
The critic aims to predict the expected discounted return from a given state.
We train the critic to regress the λ -return (Sutton and Barto, 2018; Schulman et al., 2015).
We stabilize the critic training by computing the λ -return using a target network that is updated every 100 critic updates.
The critic loss function is given by L ( w ) =
E [ (cid:80) T − 1 t =0 12 ( V ( s t ; w ) − sg ( V (cid:48) λ ( s t )) ) 2 ] , where, V (cid:48) λ ( s t ) = (cid:26)
[Equation]
We stop the gradients around the target (denoted by the sg ( . ) function), as is typical in the literature.
We use a stochastic actor.
The actor aims to output actions that lead to states that maximize the expected discounted return.
We train the actor to maximize the same λ -return that was computed to train the critic.
We add an entropy term to the actor objective to encourage exploration.
The overall actor loss function is given by L ( θ ) = − E
{+[ (cid:80) T − 1 t =
+}
[ V (cid:48) λ ( s t ) − η log π ( a t | s t ; θ ) ]] .
To backpropagate through sampled actions, we use the reparameterization trick (Kingma and Welling, 2013).
The actor network outputs the mean µ and standard deviation σ of a Gaussian distribution, from which we obtain the action as a t = tanh ( µ θ ( s t ) + σ θ ( s t ) · (cid:15) ) , where (cid:15) ∼ N (0 , I ) .
We summarize our overall model-based RL algorithm in Algorithm 1.
Initialize networks with random weights.
Execute random actions for K episodes to initialize replay buffer.
for each episode do //
Model Learning for N 1 times do
Draw mini batch of transitions from replay buffer.
Fit dynamics and reward models.
end for //
Behaviour Learning for N 2 times do
Draw mini batch of states from replay buffer.
From each state, imagine a trajectory of length T .
Backpropagate through imaginary trajectories to update actor, critic.
Every 100 updates, copy critic weights into critic target.
end for // Environment Interaction
Interact with environment for one episode using current policy.
Add transitions to replay buffer.
end for
3.3 Experiments
We train two versions of our model-based RL algorithm, one which uses the DNN approach for dynamics learning and the other which uses the LNN approach.
We refer to them as MBRL-DNN and MBRL-LNN respectively.
In both versions, we use an imagination horizon of 16 time steps.
In addition, we train a state-of-the-art model-free RL algorithm, Soft Actor-Critic (SAC) (Haarnoja et al., 2018b), to serve as a baseline.
We train each algorithm on ﬁve random seeds.
Results
We record the results from the experiments in Table 1.
The training curves are shown in Figure 3.
We summarize the results below.
• Across environments, MBRL-LNN achieves signiﬁcantly lower dynamics error than MBRL-DNN.
The reward error for both methods are similar.
• In Reacher, Pendulum and Cartpole, all the methods, i e., MBRL-DNN, MBRL-LNN and SAC, successfully solve the task and achieve similar average-return.
• In Cart-2-pole, again, all the methods successfully solve the task.
However, in terms of averagereturn, MBRL-LNN > SAC > MBRL-DNN.
In Acrobot, Cart-3-pole and Acro-3-bot, only MBRL-LNN and SAC successfully solve the task, while MBRL-DNN is unsuccessful.
Again, in terms of average-return, MBRL-LNN > SAC > MBRL-DNN.
• Across environments, MBRL-LNN achieves similar or better average return than MBRL-DNN and SAC, while requiring fewer samples.
In other words, MBRL-LNN achieves better sample efﬁciency.
Analysis
We try to understand the results better.
Reacher, Pendulum and Cartpole appear to be simple environments where all methods perform well.
Cart-2-pole, Acrobot, Cart-3-pole and Acro-3-bot appear to be more challenging environments, where there is a noticeable difference in the performance of different methods.
We try to understand what makes these latter environments more challenging and why certain methods perform better in these environments than others.
5.1 State Space and Action Space
In Appendix A, we list the dimensions of the state space and action space of all the environments.
Broadly speaking, higher-dimensional state spaces are more challenging.
Actuation matters too.
Reacher and Pendulum are fully actuated.
The rest of the environments are underactuated.
The level of underactuation varies.
Cartpole and Cart-2-pole are somewhat less underactuated, while Acrobot,
Cart-3-pole and Acro-3-bot are somewhat more underactuated.
Underactuation makes it harder to learn successful policies, hence we need more samples.
5.2 Environment Dynamics
We try to characterize the underlying dynamics of each environment by calculating their Lyapunov exponents.
Lyapunov exponents measure the rate of separation of trajectories which start from nearby initial states.
If the initial separation vector is u 0 and the separation vector at time t is u t , then the Lyapunov exponent is deﬁned as λ = 1 t log (cid:107) u t (cid:107) (cid:107) u 0 (cid:107) .
The rate of separation varies based on the orientation of the initial separation vector.
In general, there is a spectrum of Lyapunov exponents, equal in number to the dimensionality of the state space, each associated with a direction.
It is common to refer to the largest Lyapunov exponent as the maximal Lyapunov exponent (MLE).
The rate of separation is maximum along the direction associated with the MLE.
An arbitrary initial separation vector will typically contain some component in this direction, and because of the exponential growth rate, this component will dominate the evolution.
The MLE is a measure of the system’s sensitivity to initial conditions.
It also represents how quickly numerical errors will accumulate.
Thus, it is also a measure of the predictability of a system.
The standard version of the MLE is concerned with the maximal growth rate in the long-term, i. e., t → ∞ .
We refer to this as the long-term MLE.
Under a trained RL policy, the separation between two trajectories starting from nearby initial states will become zero in the long-term, as they will be driven to the same goal state.
Hence, the long-term MLE under a trained RL policy is simply −∞ .
Freefall MLE : To get an idea of the system’s inherent sensitivity to initial conditions, independent of any control policy, we set the control inputs to zero and compute the long-term MLE.
We refer to this as the freefall MLE.
All environments except Reacher are set in the vertical plane.
In these environments, we start near the goal state (vertically upright) and let the system fall freely under just [Table] the inﬂuence of gravity.
In Reacher, which is set in the horizontal plane, we start at a random state and let the system freely evolve.
In all cases, we use the true environment dynamics.
The simplest method to compute the freefall MLE is to evolve two trajectories starting from nearby initial states for a sufﬁciently long time, monitor their separation and use the Lyapunov exponent deﬁnition.
A more efﬁcient and reliable method is to use the so-called variational equation (Skokos, 2010), which linearizes the dynamics to explicitly compute how separation vectors evolve with time.
Let the true dynamics in the absence of control inputs be described by ˙ s
= f ( s ) .
Then, the variational equation is given by ˙ u
= d f d s u .
We follow the procedure outlined in Skokos, 2010 to compute the freefall MLE.
We describe the procedure here.
We consider a random initial separation vector of unit length.
We then jointly integrate the system dynamics along with the variational equation to compute how the separation vector evolves with time.
One issue is that, the variational equation was derived by linearizing the dynamics, hence it is valid only for small separations.
However, in practice, the separation vector can grow large.
Hence, we renormalize the separation vector periodically, without changing its direction.
Each time, just before renormalizing, we use Equation ??
to compute a fresh estimate of the freefall MLE and update its running average.
We continue this process for a sufﬁciently long time, until the running average converges.
We record the result in Table 1.
We ﬁnd that Reacher, Pendulum and Cartpole have a relatively small, positive, freefall MLE, implying low inherent sensitivity to initial conditions.
Whereas, Cart-2-pole, Acrobot, Cart-3-pole and Acro-3-bot have a relatively large, positive, freefall MLE, implying high inherent sensitivity to initial conditions.
We turn our focus back to computing the MLE under a trained RL policy, as this is our actual use case.
We now compute the ﬁnite-time MLE, i. e., the maximal growth rate over ﬁnite time.
We consider two ﬁnite time periods, (i) “Transient” :
Here, we consider the period from the start of an episode, till the agent reaches the goal state, which typically takes several hundred time steps, and (ii)
“Short-term” : Here, we consider a period of 16 time steps, which is the imagination horizon used in our model-based RL experiments.
Transient MLE :
We compute the ﬁnite-time MLE of the true environment dynamics over the transient period, under a trained RL policy (MBRL-DNN / MBRL-LNN / SAC).
We refer to this as the transient MLE.
The transient MLE is a measure of the system’s sensitivity to initial conditions and predictability, over the transient period, under the given policy.
It is a measure of the stability of the system’s transient dynamics, under the given policy.
We follow the same procedure we used to compute the freefall MLE, with some minor changes.
One change is that, we tweak the standard variational equation to accomodate a control policy.
Let the true dynamics in the presence of control inputs be described by ˙ s
= f
( s , a ) , where a = π ( s ) .
Now, the variational equation is given by,
[Equation]
u .
The other change is that, we start from the default initial state and stop the MLE computation once the agent reaches the goal state.
We compute the transient MLE of the true environment dynamics, under trained policies of MBRL-DNN, MBRL-LNN and SAC, and average the results.
We record the average transient MLE in Table 1.
We ﬁnd that Reacher, Pendulum and Cartpole have a relatively small, negative, transient MLE.
This implies that they have relatively stable transient dynamics, i. e., they are less sensitive to initial conditions and more predictable, over the transient period.
Whereas, Cart-2-pole, Acrobot, Cart-3pole and Acro-3-bot have a relatively large, positive, transient MLE.
This implies that they have relatively unstable transient dynamics, i. e., they are more sensitive to initial conditions and less predictable, over the transient period.
These systems exhibit unstable transient dynamics due to their high inherent sensitivity to initial conditions and also due to underactuation, which leads to relatively unrestrained motion.
Short-term MLE :
In model-based RL, we generate imaginary trajectories that predict the near future.
Hence, the short-term predictability becomes important.
We compute the ﬁnite-time MLE of the learnt dynamics model (MBRL-DNN / MBRL-LNN) over a period of 16 time steps, while following the corresponding policy.
We refer to this as the short-term MLE.
The short-term MLE along with the dynamics error determine the short-term predictability.
We adapt the procedure outlined in Skokos, 2010 to compute the short-term MLE.
We describe the procedure here.
Let the learnt dynamics model be described by s t
= F ( s t − 1 , a t − 1 ) , where a t − 1 =
π ( s t − 1 ) .
How s t will change for a small change in s 0 is given by d s t d s 0 =
d s t d s t − 1 d s t − 1 d s t − 2 · · · d s 1 d s
=
J t .
We deﬁne the short-term MLE as 1 T (cid:80) t =
T t =
[Equation]
Here, T = 16 is the length of the trajectory, ∆ t is the sampling time and u is a random vector.
We generate 10,000 imaginary trajectories of length 16 time steps and compute the average short-term
MLE of MBRL-DNN and MBRL-LNN.
We record the results in Table 1.
We ﬁnd that in Reacher,
Pendulum and Cartpole, MBRL-DNN has a relatively small short-term MLE, implying numerical errors will accumulate slowly in the short-term.
Cart-2-pole is a borderline case.
Here, MBRL-DNN has a moderate short-term MLE, implying numerical errors will accumulate moderately in the shortterm.
In Acrobot, Cart-3-pole and Acro-3-bot, MBRL-DNN has a relatively large short-term MLE, implying numerical errors will accumulate fast even in the short-term.
In contrast, MBRL-LNN has a relatively small short-term MLE across environments, implying numerical errors will accumulate slowly in the short-term always.
5.3 Imaginary Trajectories
Next, we assess the quality of the imaginary trajectories produced by MBRL-DNN and MBRL-LNN.
We use the 10,000 imaginary trajectories of length 16 time steps we generated earlier to compute the short-term MLE.
We generate the corresponding ground truth trajectories.
First, we assess the accuracy of the imaginary trajectories.
For each imaginary trajectory, we compute the state error (L1 error between the predicted state and the true state) at each time step.
We also compute the trajectory error, which is the sum of the state errors along the trajectory.
We plot the average state error as a function of time in Figure 4.
We record the average trajectory error in Table 1.
We ﬁnd that, MBRL-LNN has a low trajectory error across environments, while MBRL-DNN’s trajectory error varies.
In Reacher, Pendulum and Cartpole, MBRL-DNN has a low trajectory error.
In Cart-2-pole, MBRL-DNN has a moderate trajectory error.
In Acrobot, Cart-3-pole and Acro-3-bot,
MBRL-DNN has a high trajectory error.
Across environments, MBRL-LNN has lower trajectory error than MBRL-DNN, i. e, MBRL-LNN’s imaginary trajectories are more accurate.
Next, we assess how well the imaginary trajectories respect the underlying physical laws, such as conservation of energy.
In our environments, the work done by external non-conservative forces, i. e, the actuators, is non-zero.
Hence, the total energy of the system will change.
The change in total energy must equal the work done by external non-conservative forces, for energy to be conserved.
We deﬁne the energy error as the L1 error between the change in total energy and the work done by external non-conservative forces.
For each imaginary trajectory, we compute the energy error at each time step.
We plot the average energy error as a function of time in Figure 5.
We ﬁnd that, across environments, MBRL-LNN has lower energy error than MBRL-DNN, i. e, MBRL-LNN’s imaginary trajectories conserve energy better.
5.4 Discussion
Underactuation and high inherent sensitivity to initial conditions, characterized by a large freefall
MLE, are the main factors that make Cart-2-pole, Acrobot, Cart-3-pole and Acro-3-bot challenging.
Underactuation, in itself, makes it harder to learn successful policies, hence we need more samples.
Underactuation also leads to relatively unrestrained motion.
This condition combined with high inherent sensitivity to initial conditions leads to unstable transient dynamics, which is characterized by a large transient MLE.
Unstable transient dynamics leads to sensitivity to initial conditions over the transient period, which leads to high variance in the agent trajectories.
Unstable transient dynamics also leads to poor predictability over the transient period, which affects critic learning, as it is concerned with estimating the future return, and hence also affects actor learning, as our methods follow an actor-critic approach.
Under these conditions, we need a lot of samples to learn.
Unstable transient dynamics also makes it challenging for standard deep neural networks to learn accurate dynamics models.
SAC is successful in all four of these environments.
However, it is unable to overcome the issue of a lot of samples being needed to learn.
MBRL-DNN and MBRL-LNN overcome this issue by generating a lot of imaginary data, which allows them to perform many more policy updates.
This is the main reason why SAC achieves lower average-return than MBRL-LNN in these environments.
In Acrobot, Cart-3-pole and Acro-3-bot, due to unstable transient dynamics, MBRL-DNN has a large dynamics error and short-term MLE, due to which its imaginary trajectories have high trajectory error.
Hence, it is unable to learn good policies and is unsuccessful in these environments.
Cart-2-pole is a borderline case.
Here, MBRL-DNN achieves a moderate dynamics error and short-term MLE, despite the unstable transient dynamics.
Hence its imaginary trajectories only have moderate trajectory error and as a result, it is able to learn a reasonably good policy and is successful.
In all four of these environments, MBRL-LNN learns accurate dynamics models despite the unstable transient dynamics due to its inductive biases.
Hence, it has a small dynamics error and short-term
MLE, due to which its imaginary trajectories have low trajectory error.
Hence, it is able to learn good policies and is successful.
Thus, MBRL-LNN achieves better average-return and sample efﬁciency than both MBRL-DNN and
SAC in these environments, because it is able to generate a lot of high quality imaginary data and perform many policy updates using it.
Conclusion
We apply model-based RL to robotic systems undergoing rigid body motion.
We utilize the structure of rigid body dynamics to learn a Lagrangian Neural Network and use it to train our model-based RL algorithm.
While it is intuitive that such physics-informed dynamics models will improve modelbased RL performance, it is not apparent in which environments we will signiﬁcantly beneﬁt from such an approach.
We show that, in environments with underactuation and high inherent sensitivity to initial conditions, characterized by a large maximal Lyapunov exponent under freefall, we are likely to signiﬁcantly beneﬁt from learning physics-informed dynamics models.
In such environments, our
Lagrangian model-based RL approach achieves better average-return and sample efﬁciency than a version of our model-based RL algorithm that uses a standard deep neural network based dynamics model, as well as Soft Actor-Critic, a state-of-the-art model-free RL algorithm.
Future Work
In future work, we plan to consider friction, as it is present in most real world robots.
As far as learning friction, there are two approaches.
In the black box approach, the friction is learnt directly from data using a neural network (Zhong et al., 2020).
In the white box approach, an analytic friction model is used and its parameters alone are learnt from data (Lutter et al., 2019b).
For e. g, in robotic arms, the actuator friction dominates and it can be modelled as a combination of static friction, viscous friction and stiction, which depend on the generalized velocity ˙ q and a few coefﬁcients of friction.
These coefﬁcients alone are learnt from data.
We plan to pursue the white box approach.
In future work, we also plan to consider contact dynamics, as it is present in many real world robotics tasks like robotic manipulation and legged locomotion.
Simulation of contact dynamics is an area of active research, unlike simulation of smooth multi-joint dynamics, which is well established.
There are predominantly two approaches to simulate contact dynamics, the linear complementarity approach (LCP) (Stewart and Trinkle, 1996; Anitescu and Potra, 1997) and the convex optimization approach (Todorov, 2011; Todorov et al., 2012; Todorov, 2014).
LCP was the standard approach for a long time.
Convex optimization improves upon LCP in terms of ease of computation, while maintaining similar accuracy.
We plan to pursue the convex optimization approach and use differentiable convex optimization layers (Agrawal et al., 2019) to learn differentiable contact models which can extend LNNs.
There has been some recent work in this direction by Zhong et al., 2021b.
This approach however assumes that the geometry of the robot and its surroundings are known and uses them to compute the active contacts and their jacobians.
It remains to be seen, whether we can do away with this assumption.
Upon learning a physics-informed dynamics model that captures rigid body dynamics, friction and contact dynamics, we plan to use it to train a model-based RL algorithm.
Appendix A Environment Details
The task to be accomplished in each environment is as follows,
• Reacher : Control a fully-actuated two-link manipulator in the horizontal plane to reach a ﬁxed target location.
• Pendulum : Swing up and balance a simple pendulum.
• Cartpole : Swing up and balance an unactuated pole by applying forces to a cart at its base.
• Cart-2-pole :
One extra pole is added to Cartpole.
Again, only the cart is actuated.
• Acrobot : Control a two-link manipulator to swing up and balance.
Only the second pole is actuated.
• Cart-3-pole : One extra pole is added to Cart-2-pole.
Only the cart and the third pole are actuated.
• Acro-3-bot : One extra pole is added to Acrobot.
Only the ﬁrst and the third poles are actuated.
[Table]
All actions are in the range [ − 1 , 1] .
The reward is based on proximity to goal position, magnitude of angular velocity and whether the cart is centered (if applicable).
All rewards are in the range [0 , 1] .
There are no terminal states.
Each episode consists of 1000 time steps.
Hence, the total reward over an episode is in the range [0 , 1000] .
B Implementation Details
Table 3 lists the main hyperparameters used in our model-based RL experiments.
We use the following network architecture in our model-based RL experiments,
• Critic : dim ( S ) → 256 → 256 →• Actor : dim ( S ) → 256 → 256 → 2 × dim ( A ) • DNN : dim ( S ) + dim ( A ) → 64 → 64 → dim ( S ) • LNN – L : dim ( S ) → 64 → 64 → dim ( S ) × (dim ( S ) + 2) – V : dim ( S ) → 64 → 64 →• Reward model : dim ( S ) → 64 → 64 → [Table]
For SAC, we use the same hyperparameters and network architecture as the original paper (Haarnoja et al., 2018b).
Although during behaviour learning we use an imagination horizon of only 16 time steps, in Figuresand 7, we plot the average state error and energy error as a function of time for imaginary trajectories of horizon 100 time steps, to see how far into the future our models can predict accurately.
We ﬁnd that, across environments, MBRL-LNN is able to accurately predict quite far into the future, while MBRL-DNN is unable to.