Residual Model-Based Reinforcement Learning for Physical Dynamics
Abstract
Dynamic control problems are a prevalent topic in robotics.
Deep neural networks have been shown to learn accurately many complex dynamics, but these approaches remain data-inefficient or intractable in some tasks.
Rather than learning to reproduce the environment dynamics, traditional control approaches use some physical knowledge to describe the environment’s evolution.
These approaches do not need many samples to be tuned but suffer from approximations and are not adapted to strong modifications of the environment.
In this paper, we introduce a method to learn the parameters of a physical model i.e. the parameter of an Ordinary
Differential Equation (ODE) to approach at best the observed transitions.
This model is completed with a residual data-driven term in charge to reduce the reality gap between simple physical priors and complex environments.
We also show that this approach can be naturally extended to the case of the fine-tuning of an implicit physical model trained on simple simulations.
Introduction & Related Works
Dynamic control problems have been extensively studied in the literature.
Prior work handled the subject with physics-based models [13], but these approaches require a high physical knowledge of the problem, a lot of hand engineering and remain unable to model accurately the dynamics since it’s unrealistic to fully describe the true system dynamics e g. friction and contact parameters are either oversimplified or excluded.
Recently, Model-Free deep Reinforcement Learning (MFRL) techniques [16, 19, 8] have shown excellent performances in learning a wide range of control tasks.
However, these methods remain data inefficient or even intractable in many real-world scenarios.
Model-based Reinforcement Learning (MBRL) is an interesting alternative since it requires fewer data to learn a dynamic function [7, 5, 10] but is still intractable for some tasks where gathering data is costly or risky.
Moreover, an imperfect model can lead to drastic performance degradation when predicting trajectories due to the compounding error effect.
Model Predictive Control (MPC) is often used in a closed loop to avoid this kind of drift.
MPC consists to plan for a quite long horizon H , but we execute only actions on a receding horizon RH ≪ H .
This re-planning has a significant computational cost and represents a challenge in control tasks.
To tackle the challenge of data efficiency, further work had been interested in hybrid methods: [20] used MBRL as initialization to accelerate MFRL, [22] presented a method for learning a residual on top of weak policies.
Most closely to our work, [1, 15, 11, 17, 12, 24] studied the combination of prior physical knowledge with data-driven residuals.
A recent line of work is focused on developing physic-informed neural networks to model at best complex systems [21].
Among those, models that are based on neural ODEs seem particularly promising to grasp complex physical phenomena and are built to incorporate physical knowledge [4, 2].
APHYNITY [9] aims at augmenting incomplete physical models with fully data-driven corrections.
Our work adapts the APHYNITY framework to MRLB in order to learn the residual and correct the physical parameters governing the approximated physical dynamic.
Our approach is mostly physically driven through the integration of a neural ODE describing roughly the environment.
It is thus much more sample efficient than fully data-driven model-based methods and manages to be as efficient as data-driven baselines thanks to the learned residual correction.
Specifically, our augmented model rises to the two main challenges aforementioned:
• SAMPLE EFFICIENCY:
It benefits from the sample efficiency of the physical model which is strongly regularized by a given ODE and is as sample efficient as a full data-driven model.
Moreover, as our model is able to generalize to complex scenarios, we can use large reseeding horizons without catastrophic drift in the prediction.
• PERFORMANCE:
It benefits from the data-driven part to compensate for the gap between the approximate model and reality.
This method has proven good performance in the case where the prior is an approximate physical model, we then generalized it to the case where the prior knowledge is a weak model learned on offline simulation dataset.
Augmenting Physics-Based Model with Data-Driven Residual
We now present our method to learn a data-driven residual on top of a simplified physical model.
The overall training scheme is similar to many other MBRL approaches such as [20] and consists in alternating between sequences of model training and model-based control.
Following recent work in combining deep learning and physical models [9, 21, 2], the model is composed of a physical model, a prior on the environment’s dynamics, and a data-driven residual.
The hybrid model can benefit from the regularization expressed by the physical prior and be both sample-efficient and more explainable than a fully data-driven one.
2.1 Hybrid Dynamic Model
We assume that the environment can be described by a system following an Ordinary Differential
Equation (ODE) of the form:
[Equation]
defined over a finite time interval [0 , T ] , where s t and a t are the state and action vector for a given time t .
The function F is not accessible as it models complex interactions between the agent and the environment.
However, we suppose that we can define a physical prior F p as a rough approximation of the environment’s dynamic.
The true dynamic is then given by the combination of this prior and a residual: F = F r + F p .
The two terms composing F θ are modeled by two different neural networks F θ p p and F θ r r with parameters θ = ( θ p , θ r ) .
F θ p p is responsible for correctly identifying the parameters handling the different terms in the equation describing the physical prior while F θ r r purpose is to fill the reality gap on observed transitions.
Eq.
(1) is solved by integrating the derivative on a trajectory starting in an initial state at time t up to time t + ∆ t .
A differentiable ODE solver [18] gives the approximate solution:
[Equation]
In the end, we want to minimize the error between the prediction
ˆ s t and the true future state s t .
2.2 Training Strategy
Given a set of transitions D
= { ( s t , a t , s t +1 )
} 0 ≤ t ≤ T , we learn the dynamic by minimizing the mean square error (MSE) between the predicted and true future states using stochastic gradient descent.
Moreover, as the decomposition F = F r + F p is not unique, an ℓ 2 constraint is imposed over the residual model leading to the following loss: [Equation] subject to
[Equation]
where α is a coefficient that balances ∥ F r ∥ 2 to be as small as possible.
The model is learned in a similar fashion as [20].
Random trajectories are first sampled to collect an initial dataset D which is then used to train F θ .
Afterward, the model-based controller gathers T new transitions and adds them to the training set D , and the model F θ is retrained.
The algorithm alternates between these two steps of control and data collection on the one hand and training of the model on the second hand until convergence is reached.
In this work, we use a simple Cross-Entropy Method (CEM) [6] to select the best sequence of actions
A ( H ) t =
[Equation]
) over a finite horizon H
[Equation]
Experimental results 3.1 Environments
We evaluated our method on two control benchmark environments from OpenAI Gym [3] classic control tasks:
Pendulum ( s t ∈ S ⊂ R 2 and a t ∈ R 1 ) and Acrobot ( s t ∈ S
⊂ R and A ∈ R 2 ) .
In both tasks, we perform continuous actions to achieve the task of swinging up and balancing the pole on the upright position while the initial state is "hanging down".
Parameter values and environment details are listed in Appendix A. In both tasks, we modify the standard environment to have dynamics governed by an ODE that can be represented as the combination of two terms: F p which the part that can be represented by an analytical model and F r which represents the random phenomena that cannot be captured analytically.
For the differential ODE Solver, we use for all environments the Semi-implicit Euler method [18].
For example, the pendulum dynamic can be described as the combination of a simplistic friction-less model (physical part) and a complex term resulting from friction : s t [Equation] (cid:20) φ
[Equation]
and F
[Equation]
s t ,
[Equation]
(5) where φ represents the pendulum’s angle, ˙ φ its velocity, C g is the gravity norm, C
i is the inertia norm and C Fr is the Friction norm.
3.2 Experiments
In this work, we used a Multi-Layer Perceptron (MLP) for the data-driven model with ReLu activation (except on the last layer).
More parameter details are listed in Appendix B .
We run two experiments:
Experiment 1: Physical model as prior, where we have a simplistic physical model that approximates the dynamic function ( i.e. a frictionless model for an environment with friction)
Experiment 2: Offline learned model as prior, where we have a model already learned from an environment Env A and we want to deploy it in an environment Env B .
This case can be seen as a
Sim-to-Real scenario where Env A is a simulation and Env B is the real world.
We consider two baselines for our experiments.
First, we show the result of running the initial approximate model without learning.
Second, we show the result of learning a model from scratch with a full data-driven MBRL method.
We show that combining a prior approximate model and a data-driven residual improves the sample efficiency and the performance of MBRL.
We compare also our method with a perfect model that encodes the exact dynamic function.
3.3 Results
We present empirical results by comparing our method with the two baselines across the control tasks aforementioned.
In the pendulum experiments, the parameters of the MPC are: H =the planning Horizon and RH =
the Receding Horizon for re-planing, while in the acrobot experiment, H = 30 and RH = 10 . We detail in Appendix C an ablation study on the impact of the receding horizon.
For each task, we show that our method combine the best of both models, it benefits from the physical prior to achieve 3-4 x improvement in sample efficiency compared to the full data-driven model in one hand, and benefits from the data-driven residual to gain considerably on performance compared to the physical model.
[Figure]
Discussion and Future Work
In this paper, we study an efficient method of augmenting a weak approximate model, either a physical model or a model learned on offline simulations, by a data-driven residual to achieve good performances while remaining data efficient.
We believe that our method can accomplish good result in augmenting prior models in many tasks, especially in the case where the prior model encodes a large part of the dynamic but still need a correction residual.
This can be applied in many tasks of robotics that are governed by ODE and need to learn the friction and contact interaction by defining a physical model as prior.
Also, this method can be applied as a fast and efficient transfer learning to novel problem scenarios or adaptation to changes in the environment.
Future works would include the integration of our method into more complex dynamics.
Acknowledgments
We are very grateful to Edouard Leurent for valuable contributions concerning the elaboration of the basis code for this work.
A Environments
In this section, we provide descriptions and details of the environments used in this work.
In all environments, the observations are continuous within [ − S box , S box ] and actions are continuous and restricted to a range [ − a max , a max ] .
An overview of both tasks is illustrated in Fig.
2 specific parameters are given in Table 1.
Pendulum: A single-linked pendulum is fixed on one end, with an actuator located on the joint.
The pendulum is started at the "hanging down" position and the goal is to swing it up and balance it at the upright position.
Let φ ( t ) be the joint angle at time t .
The observation at time t is ( φ ( t ); ˙ φ ( t )) .
Acrobot: It consists of a pendulum with two links.
In this work, we experiment with the fully actuated version of the Acrobot similarly to [25, 23].
Initially, both links point downwards at the "hanging down" position.
The goal is to swing up the Acrobot and balance it in the upright position.
Let φ 1 ( t ) be the joint angles of the first fixed to a hinge at time t .
and φ 2 ( t ) the relative angle between the two links at time t .
The observation at time t is ( φ 1 ( t ); φ 2 ( t ); ˙
φ 1 ( t ); ˙ φ 2 ( t )) .
[Table]
[ Angle , Velocity ] [ Angle 1 , [Table] , Velocity 1 , Velocity 2 ] [Table]
box [ π, ∞ ] [ π, π, ∞ , ∞ ]
[Table]
[0 , 0] [ π, 0 , 0 , 0]
[Table]
[ Torque ] [ Torque 1 , Torque 2 ]
[Table]
[Table]
[Table]
[Figure]
[Table]
∆
[Table]
As described in Sec. 3.1, the dynamic functions are the combination of two terms: a physical part and a complex friction part.
In our work, we defined for each task two environments: the first is a simulation environment Env sim that encodes only the physical part without frictions, while the second environment Env real encodes the full dynamic function with frictions.
In experiment 1, the prior approximate model is a physical model that represents exactly the physical part of the dynamic function.
we train our model and the baselines on Env real .
In experiment 2, we train first a data-driven model on Env sim .
The learned model will be our prior approximate model, then we train all models on Env real
A.1 Dynamic Functions
In this subsection, we present details of the dynamic function for each task.
For each task, the simulation environment will encode only the physical part (in blue) while the real environment will encode all the dynamic functions (blue + red).
Pendulum: Let s t = ( φ ; ˙ φ ) be the observation and a t the action at time t .
the dynamic of the system is described as:
[Equation]
Where C g is the gravity norm, C i is the inertia norm and C Fr is the Friction norm.
Acrobot: Let s t =
( φ 1 ; φ 2 ; ˙ φ 1 ; ˙ φ 2 ) be the observation and a t =
( a 0 ; a 1 ) the action at time t .
the dynamic of the system is similar to [25] described as:
[Equation]
Where: α 1 = a 1 − C fr 1 · ¨ φ 1 such as C fr 1 is the friction norm in the first joint α 2 = a 2 − C fr 2 · ¨ φ 2 such as C fr 2 is the friction norm in the second joint m 1 and m 2 the mass of the first and second links l 1 and l 2 the length of the first and second {+links lc 
+} and lc 2 the position of the center of mass of the first and second links
I
and I 2 the moment of inertia of the first and second links
[Equation]
+ Σ
A.2 Reward Functions
The reward function encodes the desired task.
In this work, our main task consists of swinging up and balancing the pole in an upright position.
Hence, the height of the pole is an appropriate reward function.
Pendulum: Let s t = ( φ ; ˙ φ ) be the observation and a t the action at time t .
the reward at time t is cos( φ )
Acrobot: Let s t =
( φ 1 ; φ 2 ; ˙ φ 1 ; ˙ φ 2 ) be the observation and a t =
( a 0 ; a 1 ) the action at time t .
the reward at time t is r t = 2 − 1 . 5 · cos( φ 1 ) − cos( φ 1 + φ 2 )
B Training Details
In all experiments, we used Adam Optimizer [14] to train our model for 4000 epochs with a learning rate of 0.01.
The data-driven model and residual part were parameterized as an MLP with (3 hidden layers for Pendulum and 4 for Acrobot) with 16 neurons at each layer, and ReLu activation in all layers except the last one.
Offline Model: The Offline Model is an MLP model (similar to the MLP described above) that was trained in the simulation environment until convergence.
C Planing Details
In all experiments, we used CEM [6] as a planner with population size = 300 and elite = 20.
For the
MPC, we used a planning horizon H = 80 and receding horizon RH = 50) for the pendulum and ( H = 30; RH = 10 for the acrobot We studied the impact of changing the receding horizon on the
Acrobot Fig.
[Figure]
Our augmented model shows the best performance when augmenting the receding horizon, compared to fully data-driven model and the approximate physical model