D YNAMIC M IRROR D ESCENT BASED M ODEL P REDICTIVE C ONTROL FOR A CCELERATING R OBOT L EARNING
A BSTRACT
Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efﬁciency of Mb-RL.
Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL.
In particular, two loops are proposed, where the Dynamic Mirror Descent based
Model Predictive Control (DMD-MPC) is used as the inner loop Mb-RL to obtain an optimal sequence of actions.
These actions are in turn used to signiﬁcantly accelerate the outer loop Mf-RL.
We show that our formulation is generic for a broad class of MPC based policies and objectives, and includes some of the well-known
Mb-Mf approaches.
We ﬁnally introduce a new algorithm: Mirror-Descent Model
Predictive RL (M-DeMoRL), which uses Cross-Entropy Method (CEM) with elite fractions for the inner loop.
Our experiments show faster convergence of the proposed hierarchical approach on benchmark MuJoCo tasks.
We also demonstrate hardware training for trajectory tracking in a 2R leg, and hardware transfer for robust walking in a quadruped.
We show that the inner-loop Mb-RL signiﬁcantly decreases the number of training iterations required in the real system, thereby validating the proposed approach.
I NTRODUCTION
Model-Free Reinforcement Learning (Mf-RL) algorithms are widely applied to solve tasks like dexterous manipulation (Rajeswaran et al., 2018) and agile locomotion
(Peng et al., 2020; Lee et al., 2020) as they eliminate the need to model the complex dynamics of the system.
However, these techniques are data hungry and require millions of interactions with the environment.
Furthermore, these characteristics highly limit successful training on hardware as undergoing such high number of transitions in hardware environments is infeasible.
Thus, in order to overcome this hurdle, various works have settled for a two loop model-based approach, typically referred to as Model-based
Reinforcement Learning (Mb-RL).
Such strategies take the beneﬁt of the explored dynamics of the system by learning the dynamics model, and then determining an optimal policy in this model.
Hence this “inner-loop” optimization allows for a better choice of actions before interacting with the original environment.
The inclusion of model-learning in RL has signiﬁcantly improved sampling efﬁciency (Levine &
Koltun, 2013; Nagabandi et al., 2018) leading numerous works in this direction.
Such a learned model has proven to be very beneﬁcial in developing robust control strategies (PDDM (Nagabandi et al., 2020) and PETS (Chua et al., 2018))
based on predictive simulations.
The process of planning with the learnt model is mainly motivated by the Model Predictive Control (MPC), which is a well known strategy used in classical real-time control.
Given the model and the cost formulation, a typi- cal MPC structure can be formulated in the form of a ﬁnite horizon trajectory optimization problem.
With such a view and exploiting the approximated dynamics, methods like Cross-Entropy Method (CEM) (Pourchot & Sigaud, 2019) and Model Predictive Path Integral (MPPI)
(Williams et al., 2017) have been used to achieve high reward gains.
Model Based Policy Optimisation (MBPO)
(Janner et al., 2019) introduced the outer loop policy to collect transition to train approximate model and sample over it to train the policy.
Gradually, POLO (Lowrey et al., 2019) introduced the use of value functions for terminal rewards in such model based settings, optimizing for which can motivate the policy to converge faster.
All these were particularly incorporated by Model Predictive Actor Critic (MoPAC) (Morgan et al., 2021) along with optimization in the inner loop using MPPI to accelerate
Mb-Mf learning.
Such works demonstrate model-based (Mb) additions to typical model-free (Mf) algorithms accelerating the latter ones with signiﬁcant sampling efﬁciency.
With a view toward strengthening existing Mb-Mf approaches for learning, we propose a generic framework that integrates a model-based optimization scheme with model-free off-policy learning.
Motivated by the success of online learning algorithms (Wagener et al., 2019) in RC buggy models, we combine them with off-policy
Mf learning, thereby leading to a two-loop Mb-Mf approach.
In particular, we implement dynamic mirror descent (DMD) algorithms on a model-estimate of the system, and then the outer loop Mf-RL is used on the real system.
The main advantage with this setting is that the inner loop is computationally light; the number of iterations can be large without effecting the overall performance.
Since this is a hierarchical approach, the inner loop policy helps improve the outer loop policy, by effectively utilizing the control choices made on the approximate dynamics.
This approach, in fact, provides a more generic framework for some of the Mb-Mf approaches (e.g., (Morgan et al., 2021), (Nagabandi et al., 2020)).
In addition to the proposed framework, we introduce a new algorithm Mirror-Descent Model Predictive RL (M-DeMoRL), which uses Soft actor-critic (SAC) (Haarnoja et al., 2018b) in the outer loop as off-policy RL, and CEM with elite fractions in the inner loop as DMD-MPC (Wagener et al., 2019).
We show that the DMD-MPC accelerates the learning of the outer-loop by simply enriching the off-policy experience data with better choices of state-control transitions.
Finally, we demonstrate direct hardware training for 2R leg tracking task (Fig. 4) and hardware transfer of policies for quadruped walking with signiﬁcantly lesser environment interactions.
The paper is structured as follows: Section 2 will provide the preliminaries for OL for MPC as followed in the paper.
Section 3 will describe the hierarchical framework for the proposed strategy, followed by the description of the DMD-MPC.
Section 4 formulates the algorithm and discusses our simulation results.
Section 5 presents hardware training on a two-link leg manipulator followed by hardware transfer onto Stochlite quadruped.
Finally, we conclude in Section 6.
P ROBLEM F ORMULATION
We consider an inﬁnite horizon Markov Decision Process (MDP) given by {X , U , r, P, γ, ρ 0 } where
X ⊂ R n refers to set of states of the robot and U ⊂ R m refers to the set of control or actions.
r : X × U
→ R is the reward function, P :
X × U × X → [0 , 1] refers to the function that gives transition probabilities between two states for a given action, and γ ∈ (0 , 1) is the discount factor of the MDP.
The distribution over initial states is given by ρ 0 : X → [0 , 1] and the policy is represented by π θ : X → U parameterized by θ ∈ Θ , a potentially feasible high-dimensional space.
If a stochastic policy is used, then π θ : X × U → [0 , 1] .
For ease of notations, we will use a deterministic policy to formulate the problem.
Wherever a stochastic policy is used, we will show the extensions explicitly.
The system model dynamics can be expressed in the form of an equation: x t +1 ∼ f ( x t , u t ) , where f is the stochastic transition map.
We can obtain an estimate of this map/model as f φ , which is parameterized by φ .
The goal is to maximise the expected return :
[Equation]
Model Predictive Control (MPC).
Given the complexity of solving inﬁnite horizon problems equation 1 via reinforcement learning (RL), there has been a lot of push toward deployment of
Model Predictive Control (MPC) based methods for a ﬁnite H -step horizon (Levine & Koltun, 2013).
Denote the sequence of H states and controls as x t = ( x t, 0 , x t, 1 , . . .
, x t,H ) , and u t = ( u t, 0 , u t, 1 , . . .
, u t,H − 1 ) , with x t, 0 = x t .
The cost for H steps is given by
[Equation]
where, c ( x t,h , u t,h ) = − r ( x t,h , u t,h ) is the cost incurred (for the control problem) and c H ( x t,H ) is the terminal cost 1 .
Each of the x t,h , u t,h are related by
[Equation]
with f φ being the estimate of f .
We will use the short notation x t ∼ f φ to represent equation 4.
The solution for equation 1 with the ﬁnite horizon cost, and with the model estimate f φ can be obtained via MPC (Levine & Koltun, 2013; Nagabandi et al., 2018; Kolter & Manek, 2019).
At every step t , optimal sequence of actions/controls are obtained.
The ﬁrst action is then applied on the real system to obtain the next state.
There are several ways to solve the MPC setup, and online learning (OL) is one such approach, which is described next.
Online Learning for MPC.
Online Learning (OL) is a generic sequential decision making technique that makes a decision at time t to optimise for the regret over time.
Since MPC also involves taking optimal decisions sequentially, (Wagener et al., 2019) proposed to use online learning via dynamic mirror descent (DMD) algorithms.
DMD is reminiscent of the proximal update with a
Bregman divergence that acts as a regularization to keep the current control distribution close to the previous one.
For a rollout time of H , we sample the tuple u t from a control distribution ( π η ) parameterized by η ∈ P , where P is the parameter set.
To be more precise, η t is also a sequence of parameters: η t = ( η t, 0 , η t, 1 , . . .
, η t,H − 1 ) which yield the control tuple u t .
Therefore, given the control distribution paramater η t − 1 at round t − 1 , we obtain η t at round t from the following update rule: [Equation]
[Equation]
where J is the MPC objective/cost expressed in terms of x t and π ˜ η t , Φ t is the shift model, α t >is the step size for the DMD, and D ψ is the Bregman divergence for a strictly convex function ψ .
Note that the shift parameter Φ t is critical for convergence of this iterative procedure.
Typically, this is ensured by making it dependent on the state x t .
In particular, for the proposed two-loop scheme, we make Φ t dependent on the outer loop policy π θ ( x t ) .
Also note that resulting parameter η t is still state-dependent, as the MPC objective J is dependent on x t .
With the two policies, π θ and π η t at time t , we aim to develop a synergy in order to leverage the learning capabilities of both of them.
In particular, the ultimate goal is to learn them in “parallel”, i.e., in the form of two loops.
The outer loop optimizes π θ and the inner loop optimizes π η t for the
MPC Objective.
M ETHODOLOGY
In this section, we discuss a generic approach for combining model-free (Mf) and model-based (Mb) reinforcement learning (RL) algorithms through DMD-MPC.
In classical Mf-RL, data from the interactions with the original environment are used to obtain the optimal policy parameterized by θ .
While the interactions of the policy are stored in memory buffer, D ENV , for ofﬂine batch updates, they are used to optimize the parameters φ for the approximated dynamics of the model, f φ .
Such an optimized policy can then be used in the DMD-MPC strategy to update the control distribution, π η .
The controls sampled from this distribution are rolled out with the model, f φ , to collect new transitions and store these in a separate buffer D MPC .
Finally, we update θ using both the data i.e., from the buffer D ENV ∪ D MPC via one of the off-policy approaches (e.g. DDPG (Lillicrap et al., 2016), SAC (Haarnoja et al., 2018b)).
In this work, we majorly demonstrate this using Soft
Actor-Critic (SAC) (Haarnoja et al., 2018b).
This gives a generalised hierarchical framework with two loops: DMD-MPC forming an inner loop and model-free RL in the outer loop.
A graphical representation of the described framework is given in Figure 2.
There are two salient features in the two-loop approach:
At round t , we obtain the shifting operator Φ t by using the outer loop parameter θ .
This is in stark contrast to the classical DMD-MPC method shown in (Wagener et al., 2019), wherein the shifting operator is only dependent on the control parameter of the previous round η t − 1 . 2.
Inspired by (Lowrey et al., 2019; Morgan et al., 2021), the terminal cost c H ( x t,H ) =
− V ζ ( x t,H ) is the value of the terminal state for the ﬁnite horizon problem as estimated by the value function ( V ζ , parameterized by ζ ) associated with the outer loop policy, π θ .
This will efﬁciently utilise the model learned via the RL interactions and will in turn optimize π θ with the updated setup.
Since there is limited literature on theoretical guarantees of DRL algorithms, it is difﬁcult to show convergences and regret bounds for the proposed two-loop approach.
However, there are guarantees on regret bounds for DMD algorithms in the context of online learning (Hall & Willett, 2013).
We reuse their following deﬁnitions:
[Equation]
By a slight abuse of notations, we have omitted x t in the arguments for J .
We have the following:
Lemma
Denote the optimal cost obtained for equation 5 for the real model by J r .
Then the difference in the returns between the real and the approximated model ( J r and J respectively) is
[Equation]
using Lemma B.3 in (Janner et al., 2019).
Here, (cid:15) f is the uncertainty in dynamics model approximation.
Theorem 1 Let the ˜ η t be sequence obtained from outer loop policy π θ , and let η (cid:63)t be the policy obtained from π θ (cid:63) , where θ (cid:63) is optimal; then for the class of convex MPC objectives J , the maximum regret incurred for the T decision timesteps, each with H -step planning can be formulated as
[Equation]
where η T is the vector of parameters decided at every decision step, W Φ t is given by
[Equation]
and other Willett, 2013).
Proofs of Lemma 1 and Theorem 1 are provided in Appendix.
As we obtain the shift model at every iteration using the outer loop RL policy and the learned dynamics, the maximum regret decreases as the policy θ converges to θ (cid:63) , the sequence ˜ η t +1 approaches η
(cid:63)t +1 .
In other words, the regret is minimum when the inﬁnite horizon optimal outer loop policy is efﬁcient enough in identifying a ﬁnite H-step horizon optimal inner loop policy.
DMD-MPC with Exponential family.
We consider a parametric set of probability distributions for our control distributions in the exponential family, given by natural parameters η , sufﬁcient statistics δ and expectation parameters µ (Wagener et al., 2019).
Further, we set Bregmann divergence in equation 6 to the KL divergence.
The natural parameter of control distribution, ˜ η t , is obtained with the proposed shift model Φ t from the outer loop RL policy π θ by setting the expectation parameter of ˜ η t : ˜ µ t =
π θ ( x t ) .
Note that we have overloaded the notation π θ to map the sequence x t to ˜ µ t , which is the sequence of ˜ µ t,h
= π θ ( x t,h ) 2 .
Then, we have the following gradient of the cost:
[Equation]
In the presented setup, we choose Gaussian distribution for control and δ ( u t ) :=
u t .
We ﬁnally have the following update rule for the expectation parameter (Wagener et al., 2019):
[Equation]
Based on the data collected in the outer loop, the inner loop is executed via DMD-MPC as follows:
• Step 1.
Considering H -step horizon, for h = 0 , 1 , 2 , . .
, H − 1 , obtain
[Equation]
where Σ represents the covariance for control distribution.
1 Initialize SAC and Model f φ , Environment Parameters 2 Initialize memory buffer: D ENV 3 for max iterations do
D ENV ← D ENV ∪ { x, u, r, x (cid:48) } , u ∼ π θ ( x ) for each model learning epoch do
Train model f φ on D ENV with loss : J φ = (cid:107){ ( x (cid:48) − x ) , r } − f φ ( x, u ) (cid:107) end
Initialize D MPC
Calculate M and H from schedule for DMD-MPC iterations do
Sample x t, 0 uniformly from D ENV
Simulate M trajectories of H steps horizon: equation 10, equation 11 and equation
Perform CEM to get optimal action sequence: µ t equation 14 and equation
Collect complete trajectory: x t , r t ∼ f φ ( x t, 0 , µ t )
Add all transitions to D MPC : D MPC ← D MPC ∪ { x t,h , u t,h , ˆ r t,h , x t,h +1 } end for each gradient update step do
Update SAC parameters using data from D ENV ∪ D MPC end 20 end
• Step 2. Collect ˜ η t = (˜ η t, 0 , ˜ η t, 1 , . . . ,
˜ η t,H − 1 ) , and apply DMD-MPC equation 6 to obtain η t .
For the presented work, we use CEM with the method of elite fractions that allows us to select only the best transitions.
This is given by the following:
[Equation]
(13) where we choose C t,max as the top elite fraction from the estimates of rollouts.
It is worth noting that both CEM and MPPI belong to the same family of objective function utilities (Wagener et al., 2019).
I MPLEMENTATION AND R ESULTS
In this section, we implement the two-loop hierarchical framework as explained in the previous section and structure the speciﬁc details about the algorithm associated with the proposed work.
This will be compared with the existing approaches MoPAC
(Morgan et al., 2021) and MBPO (Janner et al., 2019) on the benchmark MuJoCo control environments.
4. A LGORITHM :
M-DeMoRL algorithm derives from other Mb-Mf methods in terms of learning dynamics and follows a similar ensemble dynamics model approach.
We have shown it in Algorithm 1.
There are three parts in this algorithm:
Model learning, Soft Actor-Critic and DMD-MPC.
We describe them below.
Model learning.
The functions to approximate the dynamics and reward function of the system are K -probabilistic deep neural networks (Kolter & Manek, 2019) cumulatively represented as { f φ 1 , f φ 2 , . . .
, f φ K } Using the inputs as the current state and actions, the ensemble model ﬁts all the probabilistic models to output change in states and the reward obtained during the transition.
Such a conﬁguration is believed to account for the epistemic uncertainty of complex dynamics and overcomes the problem of over-ﬁtting generally encountered by using single models (Chua et al., 2018).
SAC.
Our implementation of the proposed algorithm uses Soft Actor-Critic (SAC) (Haarnoja et al., 2018b) as the model-free RL counterpart.
Based on principle of entropy maximization, the choice of SAC ensures sufﬁcient exploration motivated by the soft-policy updates, resulting in a good approximation of the underlying dynamics.
DMD-MPC.
Here, we solve for E π ˜ ηt ,f φ [ C ( x t , u t ) u t ] using a Monte-Carlo estimation approach.
For a horizon length of H , we collect M trajectories using the current policy π θ t and the more accurate dynamic models from the ensemble having lesser validation losses.
For all trajectories, the complete cost is calculated using a deterministic reward estimate and the value function through (2).
After getting the complete state-action-reward H -step trajectories we execute the following based on the CEM (Pourchot & Sigaud, 2019) strategy:
• Step 1.
Choose the p % elite trajectories according to the total H -step cost incurred.
We set p = 10 % for our experiments, and denote the chosen respective action trajectories and costs as U elites and C elites respectively.
Note that we have also tested for other values of p , and the ablations are shown later in this section.
• Step 2.
Using U elites and C elites we calculate E π ˜ ηt ,f φ [ C ( x t , u t ) u t ] as the reward weighted mean of the actions i.e.
[Equation]
and update the current policy actions, ˜ µ t
=
π θ i ( x t ) according to equation 9.
4. E XPERIMENTS AND C OMPARISON
Several experiments were conducted on the MuJoCo (Todorov et al., 2012) continuous control tasks with the OpenAI-Gym benchmark and the performance was compared with recent related works
MoPAC (Morgan et al., 2021) and MBPO (Janner et al., 2019).
First, we discuss the hyperparameters used for all our experiments and then the performance achieved in the conducted experiments.
As the baseline of our framework is built upon MBPO implementation, we derive the “same hyperparameters” for our experiments and all the baseline algorithms.
We compare the results of three different seeds and the reward performance plots are shown in Figure 2(right).
For the inner DMD-MPC loop we choose a varying horizon length from 5 − 15 and performtrajectory rollouts.
With our elite fraction as 10% , the updated model-based transitions are added to the MPC-buffer.
This process is iterated with a varying batch-size with maximum of 10 ,thus completing the DMD-MPC block in Algorithm 1.
These variable batch size and horizon length allows us to exploit the models more when we have achieved signiﬁcant learning considering uncertainties and distribution shifts.
Also, as evident from Theorem 1, increasing H directly increases the maximum regret incurred.
Thus, we start from low horizon length as the regret incurred is more in the initial phases and gradually increase the horizon length to exploit the capabilities of MPC.
Further, the number of interactions with the true environment for outer loop policy were kept constant to 1000 for each epoch, same as MoPAC and MBPO.
We clearly note an accelerated progress for all the environments, with approximately 30% faster rate in the reward performance curve as compared to best of prior works.
Our rewards in Ant-v2 were comparable with MoPAC but still signiﬁcantly better than MBPO.
We would like to emphasize that our ﬁnal rewards are eventually the same as achieved by MoPAC and MBPO, however the progress rate is faster for all our experiments with lesser true environment interactions.
4. A BLATION STUDY ON ELITE PERCENTAGE
Given the sequence of controls µ t , we collect the resulting trajectory and add them to our buffer.
Therefore, the quality of µ t is a signiﬁcant factor affecting the quality of data used for the outer loop
RL-policy.
The selection strategy being CEM, a quality metric is dependent on the choice of elite fractions p .
We perform an ablation study for 6 values of p = 1 , 5 , 10 , 20 , 50 and 100% on HalfCheetah-v
OpenAI gym environment.
The analysis was performed based on the reward performance curves as shown in Fig.
3 (left).
Additionally, we realize the number of the epochs required to reach a certain level of performance as a good metric to measure acceleration achieved.
Such an analysis is provided in Fig.
3 (right).
We make the following observations:
• Having a lesser value of p might ensure that learned dynamics is exploited the most, but decreases the exploration performed in the approximated environment.
• Similarly, having higher value of p on the other hand will do more exploration using a “not-so-perfect” policy and dynamics.
Thus, the elite fraction balances between exploration and exploitation.
H ARDWARE TRAINING AND TRANSFER
In this section, we extend M-DeMoRL to more complex dynamical systems like 2R quadruped leg and a complete quadruped walking task as well.
In the ﬁrst case, we do direct algorithm transfer to execute learning with hardware, whereas for quadruped we show generation of hardware transferable policies.
Leg trajectory tracking: The main idea here is to track a ﬁxed end-foot trajectory with a 2R quadruped robot leg (as shown in Figure 4 left).
The leg is torque controlled and equipped with hall effect sensors and BLDC motors.
In order to perform direct hardware training, we accomplish torque control at 500 Hz, with the motor controller running at 40KHz.
We deﬁne the task as tracking an elliptical trajectory (as shown in Figure 4.
The reward is chosen as follows:
[Equation]
(15) where p, p d ∈ R 2 are the actual and desired end-effector positions of the leg.
We observed enhanced learning performance as compared to baseline SAC within 20 epochs (20k interactions with the real system).
Fig.
1 shows the tile of trajectory tracking from start to end.
The evaluation on convergence of M-DeMoRL was observed till 80 epochs and we concluded at reward gains of times the observed performance with SAC baseline training.
This clearly illustrates the accelerating nature of M-DeMoRL and justiﬁes feasibility of training directly on the hardware setup.
Quadrupedal walking: We extend our approach to StochLite (shown in Figure 4 right) for walking, which is a low-cost robot consisting of 12 servos and an inertial measurement unit (IMU) to detect the body pose.
StochLite is also capable of measuring joint angles (hip and knee) and torques via encoders and motor current sensors respectively.
The robot dynamics model consists of 6 ﬂoating degrees-of-freedom and 12 actuated degrees-of-freedom.
The simulator for StochLite is PyBullet (Coumans & Bai, 2016–2021).
The states consist of the torso pose, velocity (both translational and rotational components) and the slope orientation (roll and pitch).
Slope orientations are obtained via the IMU orientation and the joint angle encoders.
We execute tracking end-foot trajectories via a low-level control law, while the higher level walking control policy focuses on yielding optimal trajectory parameters similar to (Hwangbo et al., 2019;
Paigwar et al., 2020).
Those parameters form the actions/controls, to be obtained from the RL policy.
We use a well-deﬁned trajectory generator, which is in turn shaped via a neural network based policy.
This framework, when combined with our sample efﬁcient M-DeMoRL algorithm, allows to train policies in much shorter time ( < 60 episodes of true environment interactions with 1000 interactions per episode).
Figure 4 (right) shows the reactive behavior of policy in terms of the modulations applied by the neural network controller over the pre-deﬁned elliptical trajectory.
These modulations are able to shift the trajectories along X, Y and Z directions.
The learned response validates that the neural policy to model the correspondence of pitch and roll with the X and Y shifts.
As the pitch orientation increases, the quadruped tries to increase the X-shifts so that the leg moves forward and stabilizes the torso.
Similarly, the increase in roll is accompanied by again an increase in the Y-shifts in the rolling direction.
We also validate the policy with successful transfer to hardware.
Previously, (Haarnoja et al., 2018a) trained policies in Minitaur quadruped in 160 epochs using SAC.
Due to increasing complexity, we will consider direct hardware training for quadrupeds in future.
C ONCLUSION
We have investigated the role of leveraging the model-based optimisation with online learning to accelerate model-free RL algorithms.
With the emphasis to develop a real-time controller, this work primarily deﬁnes a generalised framework that could be used with the existing MPC algorithms and off the shelf Mf-RL algorithms to train efﬁciently.
Simulation results show that our formulation yields sample efﬁcient algorithms, as the underlying online learning tracks for the best policy beneﬁting the convergence of the Mf-RL policy.
We also show that the training is faster than prior
Mf-Mb methods.
Finally, we show hardware-in-loop training with a 2R leg and we successfully generate hardware transferable policies for quadruped walking.
While preliminary hardware results are shown, we look forward to better gaits and more efﬁcient direct training on hardware as future works.
The video for our experiment can be found at: stochlab.github.io/redirects/MDeMoRLPolicies.html.
A A PPENDIX : P ROOFS OF L EMMA 1 AND T HEOREM
Let us consider M and M r as the approximated and real MDP with dynamics model f φ and f respectively.
Let the total variation distance between them be bounded by (cid:15)
f (see (Janner et al., 2019)).
This dynamics model predicts both the next state distribution and rewards.
The corresponding MPC objective is represented as J and J r respectively.
Here, J denotes that the costs are calculated from the approximated reward function setting whereas J r is obtained from rollouts in the true MDP.
Now, we will derive the bounds on the performance improvement in a similar way as demonstrated in (Janner et al., 2019) and (Morgan et al., 2021), however with consideration and assumptions related to the convexity of the losses.
Proof 1 (Proof of Lemma 1)
For any stochastic dynamics model f and reward function r , considering the cost of a trajectory in an MDP with policy π η and value function V ζ is given by,
[Equation]
) where, γ is the discount factor, c ( x t,h , u t,h ) =
− r ( x t,h , u t,h ) and c H is the terminal cost calculated as − V ζ ( x t,H ) .
Let c max be the bound on this cost.
Now, to realize the maximum improvement in the approximated MDP while using the policy parameters ( ˜ η t ), obtained from the shift model, we use a formulation motivated by the bound formulated in Lemma B.3 in (Janner et al., 2019).
We consider p φ as the discounted state-action visitation corresponding to f φ (similarly p for f ) and superscript h to resemble the notations of (Janner et al., 2019).
[Equation]
˜ ηt , x t ∼ f φ
[Equation]
where, | ( p h ( x, u ) − p hφ ( x, u )) | ≤ h(cid:15) f is inherited from Lemma B.2 in (Janner et al., 2019), the uncertainty in dynamics approximation.
Proof 2 (Proof of Theorem 1) From Lemma-1, we know that,
[Equation]
(17) and subtracting J r ( η (cid:63)t ) from both sides of Eq (4) results in
[Equation]
18) where LHS corresponds to the instantaneous regret incurred by rollouts on approximate MDP (with
J ) using shifted parameters ( ˜ η t ) and on true MDP (with J r ) using the DMD-optimized parameters ( η t ).
Now, to get the cumulative regret for T decision steps, both sides of Eq (5) should be summed over
T and can be shown as,
[Equation]
Based on (Hall & Willett, 2013), the DMD update rule directly results in
[Equation]
Substituting Eq (8) in Eq (7), we ﬁnally get the bound on the maximum regret as
[Equation]
which completes the proof
[Equation]