Solving Complex Manipulation Tasks with Model-Assisted Model-Free Reinforcement Learning
Abstract:
In this paper, we propose a novel deep reinforcement learning approach for improving the sample e  ciency of a model-free actor-critic method by using a learned model to encourage exploration.
The basic idea consists in generating imaginary transitions with noisy actions, which can be used to update the critic.
To counteract the model bias, we introduce a high initialization for the critic and two ﬁlters for the imaginary transitions.
Finally, we evaluate our approach with the TD3 algorithm on di ↵ erent robotic tasks and demonstrate that it achieves a better performance with higher sample e  ciency than several other model-based and model-free methods.
Introduction
Deep reinforcement learning (DRL) has shown its potential in solving di  cult robotic tasks especially when facing complex dynamics and contact-rich environment.
For instance, dexterous manipulation tasks like rotating a cube or ball to a desired position and orientation can be solved with state-of-the-art model-free methods [1], model-based methods [2, 3].
However, the inherent issues related to the stability and sample e  ciency of DRL algorithms still make them challenging to apply on real robots.
To tackle those issues, various approaches have been investigated.
In this paper, we speciﬁcally investigate how a learned model can help accelerate a model-free method.
Previous research in this direction (see
Section 2) used a learned model for data augmentation [4], to improve critic estimation [5], for gradient computation [6] or for guiding exploration in the true environment [7].
In contrast to previous model-based methods, we propose to ”explore” inside the learned model, which is arguably safer than in the real environment.
In the learned model, noisy actions are executed in states visited in the true environment to obtain imaginary transitions, which can be used to compute imaginary target Q-values for updating the Q-function of the current policy.
With a perfect transition model, this approach would help accelerate learning the Q-function.
However, since both the learned model and the estimated Q-function may be incorrect, directly using those imaginary transitions may lead to potential issues.
To counteract them, we introduce several techniques: high Q-value initialization and ﬁltering of imaginary transitions to favor optimistic targets that are uncertain.
Intuitively, with a higher initialization, we approximate the “optimism in face of uncertainty” principle when evaluating random actions.
In the spirit of this principle, we only keep those that lead to higher evaluations than real transitions.
Moreover, to avoid unnecessary updates, those imaginary transitions are selected with higher probability if there is a larger uncertainty in the corresponding Q-values.
Note that since the noisy actions are performed in the learned model, our model is not a true exploration strategy.
However, if such imaginary targets are indeed used to update the Q-function, these can lead to higher evaluations of the corresponding actions, which would later favor selecting them in the true environment.
In this paper, we implement those techniques in the TD3 algorithm (see
Section 3) [8].
However, they may be beneﬁcial in other DRL algorithms as well.
Contributions: We propose a novel approach to exploit a learned model in DRL (see Section 4).
We validate the proposed method and show that it outperforms relevant state-ofthe-art algorithms in MuJoCo [9] robot environments and especially in some complex manipulation tasks (see Section 5) 1 .
Moreover, we analyze and discuss the di ↵ erent proposed techniques.
Related Work
Researchers have explored various methods for combining a learned model with model-free reinforcement learning algorithms.
They can mainly be divided into four categories: (1) model-based data augmentation, (2) model-based value estimation, (3) analytic gradient calculation, and (4) model-guided exploration.
Dyna [4] is a typical architecture for learning a dynamics model with the true experience and using the dynamics model to generate imaginary data for training a value function and a policy.
Gu et al.
[10] introduce a local linear model for representing the dynamics and show the improvement of using the imaginary rollouts with this model.
ME-TRPO [11] is a method leveraging an ensemble dynamics model in a model-free reinforcement learning algorithm TRPO [12].
Another work, MA-BDDPG [13], aims to alleviate the e ↵ ects of model bias by considering uncertainty when using imaginary transitions stored in a replay bu ↵ er.
The uncertainty of imaginary transitions is measured as the variance of an ensemble of critics.
In contrast to this method, we use the disagreement between the two target critics in TD3 [8] as an uncertainty measure and generate imaginary transitions online instead of maintaining another imaginary replay bu ↵ er.
Focusing on improving the estimation of the Q function, Feinberg et al.
[5] propose a method called MVE which uses the observed states and a learned dynamics model to simulate for a ﬁxed horizon with a current policy.
With this imaginary segment, a better target Q-value (value expansion, which is a variant of n-step return) is applied in the training of the critic.
Instead of using the dynamics model to forward for several steps, our method simulates only one step (to reduce the impact of the model error) but not directly with the current policy.
The performance of MVE is sensitive to a di  cult-to-set hyperparameter, the simulation horizon, which is limited by the quality of the learned model.
To overcome this problem,
Buckman et al.
[14] use a weighted sum of value expansions from di ↵ erent horizons and di ↵ erent models.
They learn ensemble models to approximate the transition function, reward function, and Q-function.
Weights are then assigned to the value estimations of di ↵ erent prediction horizons according to the variance from those models.
To ensure a monotonic improvement under the model bias, Janner et al.
[15] provided theoretical analysis on how to decide the simulation horizon.
Instead of using the learned model as an environment simulator, researchers have also exploited its di ↵ erentiability.
Deisenroth and Rasmussen [16] propose a method called PILCO to learn a probabilistic dynamics model and use it in policy search by calculating analytical gradient of the object with respect to the policy parameters.
Clavera et al.
[6] further extend the idea and train a bootstrap ensemble probabilistic dynamics model.
As for guiding the exploration, a natural idea is to try to explore more in regions where the learned model is uncertain.
For instance, Pathak et al.
[17] formulate the disagreement across an ensemble model as an intrinsic reward.
Another example is the work of Shyam et al.
[7] in which they measure the novelty of state-action pairs with a learned model and use this novelty as the objective of an exploration Markov Decision Process (MDP) to ﬁnd an exploratory policy.
In contrast to these methods, we inﬂuence the exploration in the true environment by trying noisy actions in the learned model and setting a high initialization for the critics.
Background
A Markov Decision Process (MDP) is composed of a set of state S , a set of action A , a transition function T : S ⇥ A !
P ( S ) (with P ( S ) denoting the set of probability distributions over
S ), a reward function r : S ⇥ A !
R , and a distribution over initial state µ 2 P ( S ).
Given a deterministic policy ⇡ :
S ! A , the value function V ⇡ ( s ) =
E ⇡ [ P 1 t =
t r t | s 0 = s ] is deﬁned as the expected discounted cumulative reward an agent will receive starting from a feasible state s and following the policy.
The discounted factor is deﬁned as  2 [0 , 1].
To solve this MDP is to ﬁnd an optimal policy ⇡ ⇤ that maximizes the expected value function ⇡ ⇤ ( s ) =
arg max ⇡ E µ [ V ⇡ ( s ) | s ⇠ µ ].
To ﬁnd this optimal policy, we often need an action-value function Q ⇡ ( s, a ) =
E ⇡
[ r ( s, a ) +  V ⇡ ( s 0 )], which corresponds to the expected discounted sum of rewards obtained by executing action a in state s and acting according to policy ⇡ thereafter.
Since we focus on robotic tasks, we only consider deterministic MDPs in this work, although our approach could certainly be applied in stochastic settings as well.
Deep Deterministic Policy Gradient (DDPG) DDPG [18] is a DRL algorithm with an actor-critic structure for solving an MDP with continuous state and action spaces.
The policy ⇡ (actor) and its Q-function Q ⇡ ( s, a ) (critic) are approximated by neural networks parameterized by ✓ and  respectively.
In DDPG, the actor interacts with the environment generating transitions that are stored in a replay bu ↵ er.
At each training step, a mini-batch { ( s i , a i , r i , s i +1 )
| i = 1 , · · · , N } is sampled from the replay bu ↵ er and used for updating the actor ’s parameters ✓ according to the deterministic policy gradient:
[Equation]
while the critic’s parameters  are updated to minimize the following loss function:
[Equation]
where y i is the target Q-value for the sampled transition ( s i , a i , r i , s i +1 ):
[Equation]
To improve the learning stability, the target Q-value is calculated with a target Q function
Q ( · , · |  0 ) and a target actor ⇡ ( · | ✓ 0 ).
The target networks (  0 , ✓ 0 ) are initialized to the same parameters as the original networks (  , ✓ ), but are then updated slowly towards the original ones with (1  ⌧ )  0 + ⌧ and ✓(1  ⌧ ) ✓ 0 + ⌧✓ , where ⌧ 2 (0 , 1) is a hyperparameter.
Twin Delayed DDPG (TD3) TD3 [8] improves DDPG with three tricks: clipped double-Q learning, target policy smoothing, and delayed policy update.
The ﬁrst trick aims to prevent overestimation by learning two critics Q ( · , · |  1 ), Q ( · , · |  2 ) (with their corresponding target critics Q ( · , · |  0 1 ), Q ( · , · |  0 2 )).
When calculating the target Q-value, the target critic with smaller value is used for training.
The second trick aims to smooth the objective function by injecting a truncated Gaussian noise ✏ to the output of the target policy.
Note that the perturbed actions are clipped to ensure they remain feasible.
For legibility, we do not write this step in our equations.
With these two tricks, the target
Q-value becomes:
[Equation]
Lastly, the delayed policy update enforces that the actor be updated at a lower frequency than the critic.
With those tricks, it has been empirically observed that the performance and stability of learning the Q-function are substantially improved.
Methodology
Our proposed algorithm (see Algorithm 1) extends TD3 to use a learned model to enhance the update of its critics.
Note that while the dynamics model is not known, we assume 15: interact the 16: save
17: if ends end
18: sample {
19: sample { (
, a s k
20: ⌘ 21: ﬁlter ﬁlters 22: update critic ) 23: if
26: end if 27: end for that the reward function is known.
This assumption is natural in robotics since the reward function is deﬁned by the system designer to guide the learning of the robot.
The whole training process can be divided into two phases.
In Phase 1 (time step 0 to h ), transitions are collected by interacting with the environment using a random policy.
No learning occurs up to time step h 0 after which the dynamics model starts to be trained.
Data from Phase 1 is also used to set a high initialization for the critics.
During Phase(time step h +1 to H ), transitions are now generated by the actor and both actor and critic start to be trained.
The dynamics model is further updated online and used for creating imaginary transitions with noisy actions.
The goal in our approach is to train the critics with those imaginary transitions, which can either improve the Q-estimation or guide the exploration such that potentially promising actions could be tried in the true environment.
To that aim, we apply two ﬁlters to the imaginary transitions, since using any imaginary transitions is generally detrimental due to model bias and Q-value estimation error.
The ﬁrst ﬁlter keeps the imaginary transitions with higher target Q-values.
The second ﬁlter selects imaginary transitions whose Q-values are uncertain with higher probability.
With all these components, we propose the modelassisted model-free (MAMF) algorithm
(see Algorithm 1).
To ensure the reproducibility of our results, the source code of MAMF will be released after publication.
Next we explain how we learn the dynamics model, how we implement the high initialization, how we perform exploration in the learned model, and how we ﬁlter the imaginary transitions.
Learning the Dynamics Model.
Our algorithm starts by training a dynamics model
M : S ⇥ A !
S with samples generated by a random policy (e.g., which uniformly randomly selects actions).
Before starting updating
M , h 0 transitions are ﬁrst collected.
Then, M is repeatedly updated via stochastic mini-batch gradient descent until h transitions have been generated by the random policy, which ends Phase 1.
After that, the model is still updated with the same procedure, but the transitions are now generated by the current policy.
Online DRL training is performed until H transitions have been generated.
The model is (pre)trained by minimizing a mean-squared error between predicted next states and true next states computed over a mini-batch { ( s i , a i , r i , s i +1 )
| i = 1 , · · · , N } :
[Equation]
High Initialization of Critics.
The critics are initialized to optimistic values by changing the bias term in the last layer of the critic networks.
This bias is set as follows:
[Equation]
where r ⇤ is the max reward observed during Phase 1, l is the max episode length, and c 2 [0 , 1] is a hyperparameter controlling how optimistic the high initialization is.
Exploration with Learned Model.
In the actor-critic architecture of TD3, the critic is trained using transitions { ( s i , a i , r i , s i +1 ) | i = 1 , · · · , N } sampled from the replay bu ↵ er.
For the same set of states { s i | i = 1 , · · · , N } , a noise from a truncated normal distribution ⌫ ⇠ clip
[Equation]
with a large clipping bound ⌘ is applied on the policy action ⇡ ( s i | ✓ ) to generate imaginary transitions.
The bound ⌘ is initialized as the max action a max the agent can take in each dimension.
After each iteration, the bound for truncating the
Gaussian noise exponentially decays with a ﬁxed decaying rate ⇢ .
Formally, the imaginary transitions ( s i , ˆ a i , ˆ r
i , ˆ s i +1 ) generated by the learned model f and known reward function r are written:
[Equation]
Filtering.
To only use the imaginary transitions whose actions are potentially better than the original ones, these transitions are ﬁltered by their target Q-values.
Only the imaginary transitions whose target Q-values are higher than the original ones are considered:
[Equation]
where ˆ y i and y i are calculated by Equation (4)
The remaining imaginary transitions are ﬁltered with respect to the uncertainty in the Qvalue estimates.
This uncertainty can be measured as the disagreement of the two target critics in TD3, which can be expressed as the absolute value of the di ↵ erence of the target
Q-values given by the two target critic networks:
[Equation]
This di ↵ erence is then normalized by the max di ↵ erence observed so far in the sampled transitions to obtain a ratio P =  max  2 [0 , 1].
With probability P , an imaginary transition is selected for training the critics, otherwise it is dropped.
Therefore, imaginary transitions with larger uncertainty have a higher chance to be kept.
The number of remaining imaginary transitions after the two ﬁlters is denoted by n .
Finally, the loss for the critics is composed of two parts, the original loss and an additional loss obtained from the ﬁltered imaginary transitions:
[Equation]
where the target value y i is calculated with Equation (4) and the target values ˆ y i for imaginary transitions follow almost the same calculation but with the action, next state, and reward substituted by their imaginary counterparts.
[Figure]
Experimental Results
Our experiments are designed to answer the following questions:
• Do our propositions improve the performance with respect to relevant baselines?
• Does each component of our method (i.e., decaying noise, high initialization, ﬁltering) contribute to its performance?
• In which aspects does our method help?
Before answering those questions, we present next the experimental environments and discuss the experimental set-up we used.
5. Environments and Experimental Set-Up
We evaluated our approach on several robot environments in MuJoCo [9] and three complex manipulation tasks proposed in Dexterous Gym [19].
The visualizations of these manipulation tasks are shown in Figure 1.
The task is throwing and catching an object at desired position and orientation with two dexterous manipulation hands in EggCatchUnderarm and
EggCatchOverarm.
The task of PenSpin is rotating a pen without dropping it.
The details of these environments are described in Appendix A. We chose these three representative environments from Dexterous Gym [19].
The other environments like BlockCatchUnderarm are simply variants with di ↵ erent objects.
Across all the environments, all the hyperparameters such as learning rates, batch size and network architectures for the actor and critic are kept the same except for the hyperparameters related to setting the bias.
The hyperparameters and a brief discussion about setting them are shown in Appendix B.
All the experiments are performed over ﬁve di ↵ erent random seeds and the performance is measured by the sum of rewards averaged over 10 episodes.
5. Results
Do our propositions improve the performance w.r.t relevant baselines?
To answer this question, we evaluate our method on the environments introduced above.
We compare with the following baselines: (1) TD3, (2) MVE-TD3: MVE with TD3, and (3)
MA-TD3: MA-BDDPG with TD3.
In Appendix C, we present some additional experiments where we compare and integrate our approach with the state-of-the-art model-free
RL method called SAC [20], which encourages exploration by incorporating an entropy term.
Both MVE [5] and MA-BDDPG [13] are methods that use a dynamics model to generate imaginary transitions for training the critics.
Since their source codes are not publicly available, we implemented those methods in TD3 for consistency.
For MVE, we set the prediction horizon to 3 which is applicable in all the environments.
For MA-BDDPG, we actually implemented a version similar to it, which corresponds to TD3 with only imaginary transitions ﬁltered by uncertainty.
It is equivalent to our method without high initialization, decaying noise and the ﬁlter favoring higher target Q values.
Similar to the original method proposed by Charlesworth and Montana [19], demonstrations are also used in all the experiments in Dexterous Gym.
The training results are shown in Figure 2.
We can see the outperformance of our method across di ↵ erent environments.
Especially in complex environments like EggCatchUnderarm, our method achieves a much higher ﬁnal return and has low variability across di ↵ erent runs.
[Figure]
We also notice that combining MVE with TD3 can not guarantee an improvement in some environments.
This might be due to the target Q smoothing in TD3 (clipped noise added to the action when calculating the target Q-values).
Similar results of MVE can also be observed in the work by Buckman et al.
Their results also show that MVE does not guarantee an improvement when applied on DDPG [18].
Does each component of our method contribute to its performance?
To prove the signiﬁcance of each component in our method, we performed an ablation study.
By comparing the results shown in Figure 3(a)-(b), we can ﬁnd that high initialization is necessary to counteract the e ↵ ect of the model bias and encourage the exploration at the beginning.
Using a decaying noise and choosing imaginary transitions with higher target Q-values achieve the ideas of both seeking optimistic state-action during training and of exploring in the model.
The ﬁlter with uncertainty guides the learning of the Q-function to emphasize on the uncertain part and thus helps the estimation of Q-values.
All the components in synergy improve the training of the critic.
In which aspects does our method help?
We conjecture that our method helps in three ways: better exploration, better estimation of Q-values, and exploitation of independence among action components.
For the ﬁrst point, we believe that the higher ﬁnal performance of our method compared to the baselines provides some evidence of the improved exploration.
For the second point, we compare our method with a variant of TD3 where critics are updated more frequently.
This variant could achieve better Q-estimations and thus better performances in some environments.
Due to the page limit, the results and details for the experiments are shown in Appendix C. We can see that simply updating the critic more often is still not su  cient to achieve the performance gains of our method.
[Figure]
For the last point, we believe that our method also exploits potential action independence (i.e., in some states, parts of the actions do not have an impact on the next environment state).
This kind of independence is common in robotic tasks e.g., in a walking robot, the actions of any limb that does not touch the ground may have less e ↵ ects on the next states, or more concretely, in EggCatchUnderarm, once a hand has thrown a ball, the actions of this hand is not important anymore.
To test this conjecture, we augment HalfCheetah by doubling its action space A to an enlarged action space A ⇥ A .
The transitions in this augmented environment depend only on the ﬁrst or second part of the enlarged actions.
This dependence switches from one part to the other every H time steps.
Following this pattern, part of the actions has no direct e ↵ ect.
The results of training in this artiﬁcial environment with di ↵ erent frequencies of switching are shown in Figure 3(c).
Although the augmented environment has a larger action space, our method can still improve the performance.
Limitations
The limitations of our work mainly include these three aspects: (1) Using a too large or too small bias in the critic will hurt the performance of our method.
Although our heuristic for setting the bias is quite robust (except for HalfCheetah), it is not clear how to analytically determine it.
A more theoretical analysis may be required for applying it in a more general situation.
(2) While much work has investigated exploration strategies in model-free algorithms, in our method, a simple truncated Gaussian noise is applied to generate imaginary transitions.
A more sophisticated way of choosing exploratory actions might further improve our approach.
(3)Although our proposed method signiﬁcantly improves the sample e  ciency and the performance, especially on the harder tasks of Dexterous Gym, we believe that more e ↵ orts may still be needed to fully run the algorithm in a real robot.
Potential avenues to make the approach even more practical could be to use it in combination with sim2real methods [21, 22], exploit any extra a priori known information such as symmetries [23], or exploit the learned model in some other ways [24] for instance.
Conclusion
We propose a method called MAMF, which leverages a dynamics model to help train the critic in a model-free algorithm.
One key novelty is to generate imaginary data with exploratory actions.
Our experiments demonstrate the sample e  ciency and performance improvements of this method even in some complex manipulation tasks.
We believe that our method is a step forward towards making DRL practical on real robots and shows a promising way of solving those complex tasks.
Although our proposition is implemented with TD3, it is actually independent from this DRL algorithm.
Thus, one interesting future work would be to adapt and combine our proposition with other model-free algorithms.
Acknowledgments
This work is supported in part by the program of National Natural Science Foundation of
China (No. 62176154) and the program of the Shanghai NSF (No. 19ZR1426700)