I NFORMATION T HEORETIC
M ODEL P REDICTIVE Q-L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Model-free reinforcement learning (RL) algorithms work well in sequential decision-making problems when experience can be collected cheaply and modelbased RL is effective when system dynamics can be modeled accurately.
However, both of these assumptions can be violated in real world problems such as robotics, where querying the system can be prohibitively expensive and real-world dynamics can be difﬁcult to model accurately.
Although sim-to-real approaches such as domain randomization attempt to mitigate the effects of biased simulation, they can still suffer from optimization challenges such as local minima and handdesigned distributions for randomization, making it difﬁcult to learn an accurate global value function or policy that directly transfers to the real world.
In contrast to RL, model predictive control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effectively contend with real-world dynamics.
MPC performance is usually limited by factors such as model bias and the limited horizon of optimization.
In this work, we present a novel theoretical connection between information theoretic MPC and entropy regularized RL and develop a Q-learning algorithm that can leverage biased models.
We validate the proposed algorithm on sim-to-sim control tasks to demonstrate the improvements over optimal control and reinforcement learning from scratch.
Our approach paves the way for deploying reinforcement learning algorithms on real-robots in a systematic manner.
I NTRODUCTION
Deep reinforcement learning algorithms have recently generated great interest due to their successful application to a range of difﬁcult problems including Computer Go (Silver et al., 2016) and highdimensional control tasks such as humanoid locomotion (Lillicrap et al., 2015; Schulman et al., 2015).
While these methods are extremely general and can learn policies and value functions for complex tasks directly from data, they can also be sample inefﬁcient, and partially-optimized solutions can be arbitrarily poor.
These challenges severely restrict RL’s applicability to real systems such as robots due to data collection challenges and safety concerns.
One straightforward way to mitigate these issues is to learn a policy or value function entirely in a high-ﬁdelity simulator (Shah et al., 2017; Todorov et al., 2012) and then deploy the optimized policy on the real system.
However, this approach can fail due to model bias, external disturbances, the subtle differences between the real robot’s hardware and poorly modeled phenomena such as friction and contact dynamics.
Sim-to-real transfer approaches based on domain randomization (Sadeghi &
Levine, 2016; Tobin et al., 2017) and model ensembles (Kurutach et al., 2018; Shyam et al., 2019) aim to make the policy robust by training it to be invariant to varying dynamics.
However, learning a globally consistent value function or policy is hard due to optimization issues such as local optima and covariate shift between the exploration policy used for learning the model and the actual control policy executed on the task (Ross & Bagnell, 2012).
Model predictive control (MPC) is a widely used method for generating feedback controllers that repeatedly re-optimizes a ﬁnite horizon sequence of controls using an approximate dynamics model that predicts the effect of these controls on the system.
The ﬁrst control in the optimized sequence is executed on the real system and the optimization is performed again from the resulting next state.
However, the performance of MPC can suffer due to approximate or simpliﬁed models and lim- ited lookahead.
Therefore the parameters of MPC, including the model and horizon H should be carefully tuned to obtain good performance.
While using a longer horizon is generally preferred, real-time requirements may limit the amount of lookahead and a biased model can result in compounding model errors.
In this work, we present an approach to RL that leverages the complementary properties of modelfree reinforcement learning and model-based optimal control.
Our proposed method views MPC as a way to simultaneously approximate and optimize a local Q function via simulation, and Q learning as a way to improve MPC using real-world data.
We focus on the paradigm of entropy regularized reinforcement learning where the aim is to learn a stochastic policy that minimizes the cost-to-go as well as KL divergence with respect to a prior policy.
This approach enables faster convergence by mitigating the over-commitment issue in the early stages of Q-learning and better exploration (Fox et al., 2015).
We discuss how this formulation of reinforcement learning has deep connections to information theoretic stochastic optimal control where the objective is to ﬁnd control inputs that minimize the cost while staying close to the passive dynamics of the system (Theodorou & Todorov, 2012).
This helps in both injecting domain knowledge into the controller as well as mitigating issues caused by over optimizing the biased estimate of the current cost due to model error and the limited horizon of optimization.
We explore this connection in depth and derive an inﬁnite horizon information theoretic model predictive control algorithm based on Williams et al.
We test our approach called Model Predictive Q Learning (MPQ) on simulated continuous control tasks and compare it against information theoretic MPC and soft Q-Learning (Haarnoja et al., 2017), where we demonstrate faster learning with fewer system interactions and better performance as compared to MPC and soft Q-Learning even in the presence of sparse rewards.
The learned Q function allows us to truncate the MPC planning horizon which provides additional computational beneﬁts.
Finally, we also compare MPQ versus domain randomization(DR) on sim-to-sim tasks.
We conclude that DR approaches can be sensitive to the hand-designed distributions for randomizing parameters which causes the learned Q function to be biased and suboptimal on the true system’s parameters, whereas learning from data generated on true system is able to overcome biases and adapt to the real dynamics.
R ELATED W ORK
Model predictive control has a rich history in robotics, ranging from control of mobile robots such as quadrotors (Desaraju & Michael, 2016) and aggressive autonomous vehicles (Wagener et al., 2019; Williams et al., 2017) to generating complex behaviors for high-dimensional systems such as contact-rich manipulation (Fu et al., 2016; Kumar et al., 2014) and humanoid locomotion (Erez et al., 2013).
The success of MPC can largely be attributed to online policy optimization which helps mitigate model bias.
The information theoretic view of MPC aims to ﬁnd a policy at every timestep that minimizes the cost over a ﬁnite horizon as well as the KL-divergence with respect to a prior policy usually speciﬁed by the system’s passive dynamics (Theodorou & Todorov, 2012;
Williams et al., 2017).
This helps maintain exploratory behavior and avoid over-commitment to the current estimate of the cost function, which is biased due to modeling errors and a ﬁnite horizon.
Sampling-based MPC algorithms (Wagener et al., 2019;
Williams et al., 2017) are also highly parallelizable enabling GPU implementations that aid with real-time control.
However, efﬁcient MPC implementations still require careful system identiﬁcation and extensive amounts of manual tuning.
Deep RL methods are extremely general and can optimize neural network policies from raw sensory inputs with little knowledge of the system dynamics.
Both value-based and policy-based approaches (Schulman et al., 2015) have demonstrated excellent performance on complex control problems.
These approaches, however, fall short on several accounts when applying them to a real robotic system.
First, they have high sample complexity, potentially requiring millions of interactions with the environment.
This can be very expensive on a real robot, not least because the initial performance of the policy can be arbitrarily bad.
Using random exploration methods such a (cid:15) -greedy can further aggravate this problem.
Second, a value function or policy learned entirely in simulation inherits the biases of the simulator.
Even if a perfect simulation is available, learning a globally consistent value function or policy is an extremely hard task as noted in (Silver et al., 2016; Zhong et al., 2013).
This can be attributed to local optima when using neural network representations or the inherent biases in the Q learning update rules (Fox et al., 2015; Van Hasselt et al., 2016).
In fact, it can be difﬁcult to explain why Q-learning algorithms work or fail (Schulman et al., 2017).
Domain randomization aims to make policies learned in simulation more robust by randomizing simulation parameters during training with the aim of making the policies invariant to potential parameter error (Peng et al., 2018; Sadeghi & Levine, 2016;
Tobin et al., 2017).
However, these policies are not adaptive to unmodelled effects, i.e they take into account only aleoteric and not epistemic uncertainty.
Also, such approaches are highly sensitive to hand-designed distributions used for randomizing simulation parameters and can be highly suboptimal on the real-systems parameters, for example, if a very large range of simulation parameters is used.
Model-based approaches aim to use real data to improve the model of the system and then perform reinforcement learning or optimal control using the new model or ensemble of models (Kurutach et al., 2018; Ross & Bagnell, 2012; Shyam et al., 2019).
Although learning accurate models is a promising avenue, we argue that learning a globally consistent model is an extremely hard problem and instead we should learn a policy that can rapidly adapt to experienced real-world dynamics.
The use of entropy regularization has been explored in RL and Inverse RL for its better sample efﬁciency and exploration properties (Fox et al., 2015; Haarnoja et al., 2017; 2018; Schulman et al., 2017; Ziebart et al., 2008).
This framework allows incorporating prior knowledge into the problem and learning multi-modal policies that can generalize across different tasks.
Fox et al.
(2015) analyze the theoretical properties of the update rule derived using mutual information minimization and show that this framework can overcome the over-estimation issue inherent in the vanilla Q-learning update.
In the past, Todorov (2009) have shown that using KL-divergence can convert the optimal control problem into one that is linearly solvable.
Inﬁnite horizon MPC aims to learn a terminal cost function that can add global information to the ﬁnite horizon optimization.
Rosolia & Borrelli (2017) learn a terminal cost as a control Lyapunov function and a safety set for the terminal state.
These quantites are calculated using all previously visited states and they assume the presence of a controller that can deterministically drive the any state to the goal.
Tamar et al.
(2017) learns a cost shaping to make a short horizon MPC mimic the actions produced by long horizon MPC ofﬂine.
However, since their approach is to mimic a longer horizon MPC, the performance of the learner is fundamentally limited by the the performance of the longer horizon MPC.
On the contrary, learning an optimal value function as the terminal cost can potentially lead to close to optimal performance.
Using local optimization is an effective way of improving an imperfect value function as noted in
RL literature by Anthony et al.
; Lowrey et al.
; Silver et al.
(2016; 2017); Sun et al.
However, these approaches assume that a perfect model of the system is available.
In order to make the policy work on the real system, we argue that it is essential to learn a value function form real data and utilize local optimization to stabilize learning.
P RELIMINARIES 3. R EINFORCEMENT L EARNING WITH E NTROPY R EGULARIZATION
A Markov Decision Process (MDP) is deﬁned by tuple ( S , A , c, P , γ ) where S is the state space,
A is the action space, c is a one step cost function, P is the space of transition functions and γ is a discount factor.
Let P ∈ P be a particular transition function.
A closed-loop policy π ( a | s ) is a distribution over actions given state.
Given a policy π and a prior policy π , the KL divergence between them at a state is given by KL ( π ( a | s ) ||
π ( a | s ))
=
E π [log ( π ( a | s ) /π
( a | s ))] .
Entropyregularized RL (Fox et al., 2015) aims to optimize the following objective
[Equation]
where π t and π t are shorthand for π ( a t | s t ) and
π ( a t | s t ) respectively, λ is the temperature parameter that penalizes deviation of π from π .
For a policy π , we can deﬁne the soft value and action-value functions
[Equation] [Equation]
Given a horizon of H timesteps, we can use above deﬁnitions to write the value functions as
[Equation]
It is straightforward to verify that V π ( s ) =
E a ∼ π [log( π ( a | s ) /π
( a | s ))
+ Q ( s, a )] .
The objective in Eq.
(1) can equivalently be written as
[Equation]
This optimization can be performed either by policy gradient methods that aim to ﬁnd the optimal policy π ∈ Π via stochastic gradient descent (Schulman et al., 2017; Williams, 1992) or value based methods that try to iteratively approximate the value function of the optimal policy.
In either case, the output of solving the above optimization is a global closed-loop control policy π ∗ ( a | s ) .
3. I NFORMATION T HEORETIC MPC
Solving the above optimization can be prohibitively expensive and hard to accomplish online.
In contrast to RL, MPC performs online optimization of a simple policy class with a truncated horizon.
This process effectively creates a closed-loop controller.
In order to do so, MPC algorithms such as
MPPI (Williams et al., 2017) use an approximate dynamics model ˆ P , which can be deterministic.
This is the case when using a simulator such as MuJoCo (Todorov et al., 2012) as the dynamics model.
At timestep t , starting from the current state s t , an open loop sequence of actions A = ( a t , a t +1 , . . .
a t + H ) is sampled from the control distribution denoted by π ( A ) .
The objective is to ﬁnd an optimal sequence of actions to solve
[Equation]
where c f ( s t + H , a t + H ) is a terminal cost function and π ( A ) is the passive dynamics of the system, i.e the distribution over actions produced when the control input is zero.
The ﬁrst action in the sequence is then executed on the system and the optimization is performed again from the resulting next state.
The re-optimization and entropy regularization helps in mitigating model-bias and inaccuracies with optimization by avoiding overcommitment to the current estimate of the cost.
A shortcoming of the MPC procedure is the ﬁnite horizon.
This is especially pronounced in tasks with sparse rewards where a short horizon can make the agent myopic to future rewards.
To mitigate this, an approach known as inﬁnite horizon MPC sets the terminal cost c
f as a value function that adds global information to the problem.
In the next section, we build our approach by focussing on the MPPI algorithm and its relationship with entropy regularized reinforcement learning.
Speciﬁcally, we use the deﬁnitions of the soft value functions from Eq.
(2) to derive an optimal Boltzmann distribution for H -step actions that optimally solves the inﬁnite horizon control problem.
This helps us derive the MPPI update rule from Williams et al.
(2017) for the inﬁnite horizon case, which then leads to our Model Predictive Q Learning (MPQ) algorithm, which utilizes a predictive model for Q updates and stochastic optimal control as the policy.
In the case where H = 1 , the algorithm is equivalent to soft Q learning (Haarnoja et al., 2018) or G-learning (Fox et al., 2015).
A PPROACH 4. O PTIMAL H- STEP B OLTZMANN D ISTRIBUTION
Let π ( A ) and π ( A ) be the joint control distribution and prior over H -horizon open-loop actions.
The distributions are assumed to be independent over timesteps, i.e π ( a 1 . . .
a H )
= (cid:81)
Ht =1 π t , where π t is shorthand for π ( a t ) .
Since ˆ P is deterministic, the following equations hold
[Equation]
For clarity
[Equation]
we consider γ = 1 .
Substituting from Eq.
(3) for Q π ( s, a )
[Equation]
Consider the following distribution over H-horizon
[Equation]
where η is a normalization constant given by
[Equation]
We show that this is the optimal control distribution as ∇ V π ( s ) = 0 .
Substituting Eq.
(8) in (7)
[Equation]
Since η is a constant, we have V π ( s )
= − λ log η .
Hence for π in Eq.
(8), the soft value function is a constant with gradient zero given by
[Equation]
which is often referred to in optimal control literature as the “free energy” of the system (Theodorou & Todorov, 2012; Williams et al., 2017).
For H=1, Eq.
(10) takes the form of the soft value function from Fox et al.
(2015) and Haarnoja et al.
4. I NFINITE H ORIZON MPPI U PDATE R ULE
Similar to Williams et al.
(2017), we derive the MPPI update rule for online policy optimization.
Since sampling actions from the optimal control distribution in Eq.
(8) is intractable, we consider control policies π ( A ) ∈ Π which are easy to sample from.
We then optimize for a vector of H control inputs U , such that the resulting action distribution minimizes the KL divergence with the optimal policy
[Equation]
The objective can be expanded out as
[Equation]
Since the ﬁrst term does not depend on the control input, we can remove it from the optimization
[Equation]
Consider Π to be independent multivariate Gaussians over sequence of the H controls with constant covariance Σ at each timestep.
We can write the control distribution and prior as follows
[Equation]
where u t and a t are the control inputs and actions at timestep t and Z is the normalization constant.
Here the prior corresponds to the passive dynamics of the system (Theodorou & Todorov, 2012;
Williams et al., 2017), although other choices of prior are possible.
Substituting in Eq.
(13) we get
[Equation]
The objective can be simpliﬁed to the following by integrating out the probability in the ﬁrst term
[Equation]
Since this is a concave function with respect to every u t , we can ﬁnd the maximum by setting its gradient with respect to u t to zero to solve for optimal u ∗ t
[Equation]
[Equation]
where the second equality comes from importance sampling to convert the optimal controls into an expectation over the control distribution instead of the optimal distribution which is impossible to sample from.
The importance weight w ( A ) can be written as follows (substituting π ∗ from Eq.
[Equation]
Making change of variables u t + (cid:15) t = a t for noise sequence E
= ( (cid:15) 1 . . .
(cid:15) H ) sampled from independant Gaussians with zero mean and covariance Σ we get
[Equation] [Equation]
Note that η is the optimal H-step free energy derived in Eq.
(10) and can be estimated from N
Monte-Carlo samples as
[Equation]
We can form the following iterative update rule where at every iteration i the sampled control sequence is updated according to
[Equation]
where α is a step-size parameter as proposed by Wagener et al.
This gives us the inﬁnite horizon MPPI update rule.
For H = 1 , this corresponds soft Q-learning where stochastic optimization is performed to solve for the optimal action online.
Now we develop soft Q-learning algorithm that utilizes inﬁnite horizon MPPI to generate actions as well as Q-targets.
4. T HE I NFORMATION T HEORETIC M ODEL P REDICTIVE Q-L EARNING A LGORITHM
Since we do not have access to Q π ∗ , we can not estimate the importance weight in Eq.
(19) exactly.
Hence, we consider Q functions parameterized by θ denoted by Q θ ( s, a ) .
Similar to deep Q-learning algorithms, we maintain an replay buffer (Mnih et al., 2015), and update parameters by stochastic gradient descent on the loss L = 1 K (cid:80) Ki
=
( y i − Q
θ ( s i , a i )) 2 for a batch of K experience tuples ( s, a, c, s (cid:48) ) sampled from the buffer where targets y i are given by
[Equation]
(cid:33)(cid:33) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:48) (cid:35) (22)
Since the value function updates are performed ofﬂine we can utilize large amount of computation (Tamar et al., 2017) to calculate π ∗ ( a 1 . . .
a H ) .
In our case it is obtained by performing the inﬁnite horizon MPPI update in Eq.
(21) for multiple iterations starting from state s (cid:48) .
This allows for directed exploration at a state which leads to better approximation of the free energy (which is akin to approaches such as Covariance Matrix Adaption, except MPPI does not adapt the covariance).
This especially helps in early stages of learning by providing better quality targets than a random Q function.
Intuitively, this update rule leverages the biased dynamics model ˆ P for H steps and a soft Q function at the end learned from interactions with the real system.
At every timestep t during online rollouts, a H -horizon sequence of actions is optimized using a single iteration of inﬁnite horizon MPPI update rule in Eq.
and the ﬁrst action is executed on the system.
Online optimization with predictive models can lookahead to produce better actions than acting greedily with respect to the biased Q function and makes ad-hoc exploration strategies such as (cid:15) -greedy unnecessary.
Using predictive models for generating value targets and online policy optimization helps accelerate convergence as we demonstrate in our experiments in the next section.
Algorithm 1 shows the complete MPQ algorithm.
A closely related approach in literature is (Lowrey et al., 2018) which also uses online MPC and ofﬂine value function learning, however they assume access to the true dynamics of the system and do not explore the connection between MPPI and entropy regularized RL and hence do not use free energy targets, even though they use MPPI in their implementation.
Algorithm
Input : Approximate model ˆ P , initial Q function parameters θ 1 , experience buffer D
Parameter: Number of episodes N , length of episode T , planning horizon H , number of update episodes N update , minibatch-size K , number of minibatches M 1 for i = 1 . . .
N do for t = 1 . . .
T do ( a t , . . .
, a t + H ) ← Inﬁnite horizon MPPI (Eq.
Execute a t on the real system to obtain c ( s t , a t ) and next state s t +
D ← ( s t , a t , c, s t +1 ) if i % N update {+= = 
+}
then
Sample M minibatches of size K from D
Generate targets using Eq.
and update parameters to θ i +  return θ N or best θ on validation.
E XPERIMENTS
We perform experiments to test the efﬁcacy of MPQ in overcoming the shortcomings of stochastic optimal control and model free RL in terms of convergence rate, computational requirements and model bias.
We also compare MPQ against domain randomization for learning policies that perform well on systems for which accurate models are not known.
5. E XPERIMENTAL S ETUP
We test our approach on sim-to-sim continuous control tasks based on the Mujoco simulator (Todorov et al., 2012) to study the properties of the algorithm in a controlled manner.
The agent is not provided with the true dynamics parameters, but a uniform distribution over them with a biased mean and added noise.
This serves as a reasonable approximation of model bias due to inaccurate measurements of physical quantities.
Details of the tasks considered are as follows
P ENDULUM S WINGUP : the agent tries to swingup and stabilize a pendulum by applying torque on the hinge. The agent is provided with a
distribution over the mass and length of the pendulum.
The state of the system is given by (Θ , ˙Θ) , where Θ is the angular displacement.
The cost function penalizes the deviation from the upright position and angular velocity.
The initial state of the system is randomized after every episode which is 10 seconds long.
B ALL
I N C UP
S
PARSE : a sparse version of the ball in cup task inspired from
Tassa et al.
Given a cup and spherical ball attached by a tendon, the goal is to swing and catch the ball.
The agent can actuate motors on the two slide joints on the cup and is provided with a biased distribution over the mass of the ball, its moment of inertia and stiffness of the tendon.
A cost of 1 is incurred at every timestep and 0 if the ball is in the cup.
The position of the ball is randomized after every episode which is 4 seconds long.
An episode is successful if agent catches the ball in the cup .
F ETCH P USH B LOCK : proposed by Plappert et al.
the agent position controls the end
- effector of a simulated Fetch robot to push a block to a goal location on the table.
The cost is the distance between the center of mass of the block and the goal.
We provide the agent a biased distribution over the mass, moment of inertia inertia, friction coefﬁcients and size of the object.
An episode is considered successful if the agent gets the block within 5cm of the goal in 4 seconds.
The positions of both block and goal is randomized after every episode
F RANKA D RAWER O PEN : the agent velocity controls a 7DOF Franka Panda arm to open a drawer on a cabinet.
A simple cost function based on Euclidean distance and relative orientation of the end effector with respect to the
handle and the displacement of the slide joint on the drawer is used.
The agent is provided a biased distribution over damping and frictionloss of robot and drawer joints.
Every episode is 4 seconds long after which the agent’s starting conﬁguration is randomized.
Success corresponds to opening the drawer within 1cm of target displacement. The parameters we selected to randomize are reasonable in real world scenarios since estimating quantities like moment inertia and friction coefﬁcients is especially error prone.
All our experiments are performed on a desktop with 12 Intel Core i7-3930K @ 3.20GHz CPUs and 32 GB RAM with only few hours of CPU training.
Q-functions are parameterized with feed-forward neural networks that take as input an observation vector and action.
Refer to A.1 for further details.
[Table]
Θ 2 + 0 .
1 ˙ Θm = {+
+} kg m =
[0 . 9 , 1 . 5] l = 1 m l
=
[Table] m = 0 .
058 kg m =
I xyz = 1 .
47 × 10 −
I xyz =
22 , 22] × 10 −1 else
T = 0 .
T = [0 . 00375 , 1 . 5]
F ETCH P USH B LOCK d block,goal m = 2 kg bias = 0 .
I xyz = 8 .
33 e −σ = 0 .µ
=
[Table]
5. B ASELINES
We compare MPQ with a ﬁxed horizon H against MPPI without a terminal value function and
S OFT QL EARNING .
For S OFT QL EARNING we additionally use a target network to stabilize learning whereas MPQ does not use target networks. [Figure]
5. A NALYSIS OF O VERALL P ERFORMANCE 5.3. C OMPARISON OF MPQ WITH MPPI AND soft Q L EARNING
We test the hypotheses that (1) using a soft Q function as the terminal cost can signiﬁcantly improve
MPPI performance even with a shorter horizon and (2) ﬁnite horizon optimization can lead to significantly faster convergence as compared to soft Q Learning especially in sparse reward tasks.
Fig.shows the training curves for MPQ versus soft Q learning.
Online optimization is able to improve upon the inaccuracies of the Q function and lead to faster convergence.
In sparse reward task such as B ALL I N C UP S PARSE , S OFT
QL EARNING is unable to learn a consistent policy whereas MPQ improves very rapidly.
Additionally, an MPQ agent with a short horizon consistently outperforms
MPPI with long horizon in all tasks.
This can be attributed to (1) larger model bias in longer horizon optimization (2) hardness of optimizing longer sequences and (3) global information encapsulated in the Q function.
Using a shorter horizon has added computational beneﬁts as well.
Since the Q function is learned using data generated from the true system parameters, it is not affected by model bias.
In F ETCH P USH B LOCK , the agent is able to perform at par with MPPI using 2.5 times the horizon in only 63 episodes of training which corresponds to roughly 4 minutes of experience on simulation with true parameters.
In contrast, S OFT QL EARNING barely ever moves the block and
MPPI with a short horizon succeeds only when arm is close to initial position of block.
Similarly, in F RANKA D RAWER O PEN , the agent with H = 10 achieves a success rate of more than 5 times as compared to MPPI with H=10 and outperforms MPPI with H = 22 demonstrating the ability of
MPQ to scale to high dimensional systems.
5.3. B ENEFIT OF MPQ OVER DOMAIN RANDOMIZATION
Domain randomization(DR) techniques aim to make the policy learned in simulation robust to modelling errors by randomizing the simulation parameters using manually chosen distributions.
However, such policies can be far from optimal on the true system parameters as the learned Q function is inherently biased.
However, A Q function learned using rollouts from the real system can overcome model bias.
We validate this hypothesis on the B ALL
I N C UP
S PARSE task by taking a DR approach inspired by Peng et al.
Simulated rollouts are generated by sampling different parameters at every timestep from a broad range of dynamics parameters in Table 1, whereas real system rollouts use the true parameters.
The average success rate reported in Table 2 demonstrates that a Q function learned solely using DR is unable to generalize to the true system parameters and MPQ has more than twice the success rate when learned on real system [Table]
[Table]
D ISCUSSION
In this work we have presented a theoretical connection between information theoretic MPC and soft
Q learning approaches that naturally provides an algorithm to combine stochastic optimal control and model-free reinforcement learning.
The theoretical insight not only ties together the different ﬁelds but opens avenues to designing pragmatic RL algorithms that leverage the beneﬁts of both.
However, some important questions are yet to be answered.
The optimal horizon for MPC is inextricably tied with the model error and optimization artifacts.
Investigating this dependence in a principled manner is important for real-world applications.
Another interesting avenue of research is characterizing the performance of a parameterized Q function and using it to adapt the horizon of
MPC rollouts for smarter exploration.
A A PPENDIX A. F URTHER EXPERIMENTAL DETAILS
The learned Q function takes as input the current action and an observation vector.
The observation vectors for different tasks are as follows:
P ENDULUM S WINGUP : (cid:104) cos (Θ) , sin (Θ) , ˙Θ (cid:105) (3 dimensional) 2.
B ALL
I N C UP
S
PARSE : [ x ball , x target , ˙ x
b all, ˙ x t arget, x target − x ball , cos (Θ) , sin (Θ) ] ( dimensional) where Θ is angle of line joining ball and target.
F ETCH P USH B LOCK : [ x gripper , x obj , x obj − x grip , gripper opening , rot obj , ˙ x obj , ω obj gripper opening vel , ˙ x gripper , d ( gripper , obj ) , x goal − x obj , d ( goal, obj ) , x goal ] (33 dimensional)
For all our experiments we parameterize Q functions with feedforward neural networks with two layers containing 100 units each and tanh activation.
We use Adam (Kingma & Ba, 2014) optimization with a learning rate of 0.001.
For generating value function targets in Eq.
(22), we useiterations of MPPI optimization.
The MPPI parameters used are listed in Table 3.
All parameters were chosen using a coarse grid search and it is possible to further optimize them.
[Table]
A. I NFINITE H ORIZON MPPI D ERIVATING WITH D ISCOUNT F ACTOR A.2. O PTIME H- STEP B ELLMAN E QUATION
Let π γ = (cid:81)
Ht =1 π γt − 1 t and π γ = (cid:81)
Ht =1 π γt − 1 t .
Following from equation Eq.
V π ( s ) = E [Equation] [Equation]
H
[Equation]
Consider the following policy
[Equation]
where η is a normalization constant given by
[Equation]