GRAC: Self-Guided and Self-Regularized Actor-Critic
Abstract: Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks.
One dominant component of recent deep reinforcement learning algorithms is the target network which mitigates the divergence when learning the Q function.
However, target networks can slow down the learning process due to delayed function updates.
Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network.
Additionally, we propose a self-guided policy improvement method by combining policy-gradient with zeroorder optimization to search for actions associated with higher Q-values in a broad neighborhood.
This makes learning more robust to local noise in the Q function approximation and guides the updates of our actor network.
Taken together, these components deﬁne GRAC , a novel self-guided and self-regularized actor critic algorithm.
We evaluate GRAC on the OpenAI gym tasks, outperforming popular methods such as TD3 [1] and SAC [2] on four tasks and achieving competitive results on two environments.
We also apply GRAC to enable a non-anthropomorphic robotic hand to successfully accomplish an in-hand manipulation task in the real world.
Introduction
Reinforcement learning (RL) studies decision-making with the goal of maximizing total discounted reward when interacting with an environment.
Leveraging high-capacity function approximators such as neural networks, Deep reinforcement learning (DRL) algorithms have been successfully applied to a range of challenging domains, from video games [3] to robotic control [4].
Actor-critic algorithms are among the most popular approaches in DRL, e g. DDPG [5], TRPO [4],
TD3 [1] and SAC [2].
These methods are based on policy iteration, which alternates between policy evaluation and policy improvement [6].
Actor-critic methods jointly optimize the value function (critic) and the policy (actor) as it is often impractical to run either of these to convergence [2].
In DRL, both the actor and critic use deep neural networks as the function approximator.
However,
DRL is known to assign unrealistically high values to state-action pairs represented by the Q-function.
This is detrimental to the quality of the greedy control policy derived from Q [7].
Mnih et al.
[8] proposed to use a target network to mitigate divergence.
A target network is a copy of the current Q function that is held ﬁxed to serve as a stable target within the TD error update.
The parameters of the target network are either infrequently copied [8] or obtained by Polyak averaging [5].
A limitation of using a target network is that it can slow down learning due to delayed function updates.
We propose an approach that reduces the need for a target network in DRL while still ensuring stable learning and good performance in high-dimensional domains.
We add a self-regularization term to encourage small changes to the target value while minimizing the Temporal Difference (TD)-error [6].
Evolution Strategies (ES) are a family of black-box optimization algorithms which are typically very stable, but scale poorly in high-dimensional search spaces e.g. neural networks [9].
Policy gradient-based DRL methods, unlike evolutionary search methods, can continue to sample previous experiences to improve value estimation, particularly in the off-policy setting.
At the same time, these approaches can also be unstable and highly sensitive to hyper-parameter tuning [9].
We propose a novel policy improvement method which combines both approaches to get the best of both worlds.
Speciﬁcally, after the actor network ﬁrst outputs an initial action, we apply the Cross Entropy
Method (CEM) [10] to search the neighborhood of the initial action to ﬁnd a second action associated with a higher Q value.
Then we leverage the second action in the policy improvement stage to speed up the learning process.
Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network that may slow down learning progress.
In addition, we propose a self-guided policy improvement method which combines policy-gradients and zero-order optimization.
This helps to speed up learning and is robust to local noise in the Q function approximation.
Taken together, these components deﬁne GRAC , a novel self-guided and self-regularized actor critic algorithm.
We evaluate GRAC on six continuous control domains from OpenAI gym [11], where
GRAC outperforms popular methods such as TD3 [1] and SAC [2] on four tasks and achieves competitive results on two environments.
We run our experiments across a large number of seeds with fair evaluation metrics [12], perform extensive ablation studies, and open source both our code and learning curves.
We also run GRAC to enable a non-anthropomorphic, real robotic hand to successfully rotate a cube to the target pose.
Related Work
The proposed algorithm incorporates key ingredients within the actor-critic method: a self-regularized
TD update and self-guided policy improvements based on evolution strategies.
In this section, we review prior work related to these ideas.
Divergence in Deep Q-Learning In Deep Q-Learning, we use a nonlinear function approximator such as a neural network to approximate the Q-function that represents the value of each state-action pair.
Learning the Q-function in this way is known to suffer from divergence issues [13] such as assigning unrealistically high values to state-action pairs [7].
For the case when the control policy is greedily derived from Q [7], unrealistically high Q values are detrimental.
To mitigate the divergence issue, Mnih et al.
[8] introduce a target network which is a copy of the estimated Q-function and is held ﬁxed to serve as a stable target for some number of steps.
However, because of the delayed function updates, target networks can slow down learning [14].
Durugkar and Stone [15] propose
Constrained Q-Learning, which uses a constraint to prevent the average target value from changing after an update.
Achiam et al.
[16] give a simple analysis based on a linear approximation of the
Q function and develop a stable Deep Q-Learning algorithm for continuous control without target networks.
However, their proposed method requires separately calculating backward passes for each state-action pair in the batch, and solving a system of equations at each timestep.
Bhatt et al.
[17] introduce a normalization called cross-normalization which is regarded as an extension of batch normalization that re-centers data for on- and off-policy transitions.
Peng et al.
[18] uses Monte
Carlo instead of TD error to update the Q network and removes the need for a target network.
Our proposed GRAC algorithm adds a self-regularization term to the TD-Learning objective to keep the change of the state-action value small.
Evolution Strategies in Deep Reinforcement Learning Evolution Strategies are typically stable but suffer from scaling to high-dimensional search spaces.
Policy gradient-based deep RL methods, such as DDPG [5] can continue to reuse previous experience to improve value estimations, particularly in the off-policy setting, but can be unstable and highly sensitive to hyper-parameter tuning [9].
Researchers have proposed to combine these approaches to get the best of both worlds.
Pourchot and Sigaud [9] proposed CEM-RL to combine CEM with either DDPG [5] or TD3 [1].
However,
CEM-RL applies CEM within the actor parameter space which is extremely high-dimensional, making the search not efﬁcient.
Kalashnikov et al.
[19] introduce QT-Opt , which leverages CEM to search the landscape of the Q function, and enables Q-Learning in continuous action spaces without using an actor.
Based on QT-Opt , Simmons-Edler et al.
[20] leverage CEM to search the landscape the
Q-function but propose to initialize CEM with a Gaussian prior covering the action space, independent of observations.
However, CEM does not scale well to high-dimensional action spaces [21], such as in the Humanoid task.
We ﬁrst let the actor network output an initial Gaussian action distribution conditioned on current state.
Then we use CEM to search for an action with a higher Q value than the
Q value of the Gaussian mean.
Starting from the predicted distribution, we show that GRAC speeds up the learning process compared to popular actor-critic methods.
Preliminaries
In this section, we deﬁne the notation used in subsequent sections.
Consider a Markov Decision
Process (MDP), deﬁned by the tuple ( S , A , P , r, ρ 0 , γ ) , where S is a ﬁnite set of states, A is a ﬁnite set of actions, P : S ×
A × S → R is the transition probability distribution, r : S ×
A → R is the reward function, ρ 0 : S → R is the distribution of the initial state s 0 , and γ ∈ [0 , 1] is the discount factor.
At each discrete time step t , with a given state s t ∈ S , the agent selects an action a t ∈ A , receiving a reward r and the new state s t +1 of the environment.
Let π denote the policy which maps a state to a probability distribution over the actions, π : S →
P ( A ) .
The return from a state is deﬁned as the sum of discounted reward R t =
(cid:80) i =
t γ i − t r
( s i , a i ) .
In reinforcement learning, the objective is to ﬁnd the optimal policy π ∗ , with parameters φ , which maximizes the expected return J ( φ )
=
(cid:80) t E ( s t ,a t ) ∼ ρ π ( s t ,a t ) [ γ t r ( s t , a t )] where ρ π ( s t ) and ρ π ( s t , a t ) denote the state and state-action marginals of the trajectory distribution induced by the policy π ( a t
| s t ) .
We use the following standard deﬁnitions of the state-action value function Q π .
It describes the expected discounted reward after taking an action a t in state s t and thereafter following policy π :
[Equation]
In this work we use CEM to ﬁnd optimal actions with maximum Q values.
CEM is a randomized zeroorder optimization algorithm.
To ﬁnd the action a that maximizes Q ( s, a ) , CEM is initialized with a paramaterized distribution over a , P ( a ; ψ ) .
Then it iterates between the following two steps [22]:
First generate a 1 , . .
. , a N ∼ P ( s ; ψ ) .
Retrieve their Q values Q ( s, a i ) and sort the actions to have decreasing Q values.
Then keep the ﬁrst K actions, and solve for updated parameters ψ (cid:48) :
[Equation]
In the following, let CEM ( Q ( s, · ) , π ( ·| s ))
denote the action found by CEM to maximize Q ( s, · ) , when CEM is initialized with the distribution predicted by the policy.
Technical Approach 4.1 Self-Regularized TD Learning
Reinforcement learning is prone to instability and divergence when a nonlinear function approximator such as a neural network is used to represent the Q function [13].
Mnih et al.
[8] identiﬁed several reasons for this.
One is the correlation between the current action-values and the target value.
Updates to Q ( s t , a t ) often also increase Q ( s t +1 , a ∗ t +1 ) where a ∗ t +1 is the optimal next action.
Hence, these updates also increase the target value y t which may lead to oscillations or the divergence of the policy.
More formally, given transitions ( s t , a t , r t , s t +1 ) sampled from the replay buffer distribution B , the
Q network can be trained by minimising the loss functions
L ( θ i ) at iteration i :
[Equation]
where for now let us assume y i
=
r t + γ max a Q
( s t +1 , a ; θ i ) to be the target for iteration i computed based on the current Q network parameters θ i .
a ∗ t +1 = arg max a Q ( s t +1 , a ) .
If we update the parameter θ i +1 to reduce the loss L ( θ i ) , it changes both Q ( s t , a t ; θ i +1 ) and Q ( s t +1 , a ∗ t +1 ; θ i +1 ) .
Assuming an increase in both values, then the new target value y
i +
=
r t + γQ ( s t +1 , a ∗ t +1 ;
θ i +1 ) for
[-We also show how maintaining a separatetarget network [8] with frozen parameters θ − to compute y i +
-]
Initialize critic network Q θ 1 , Q θ 2 and actor network π φ with random parameters θ 1 , θ 2 and φ
Initialize replay buffer B , Set α <1: for i = 1 , . . .
do 2:
Select action a ∼ π φ i ( s ) , observe reward r and new state s (cid:48) 3:
Store transition tuple ( s, a, r, s (cid:48) ) in B 4:
Sample mini-batch of N transitions ( s t , a t , r t , s t +1 ) from B .
[Table]
a
[Table]
i ( s t +1 )
[Table] [Table]
a
[Table]
← CEM ( Q (
[Table] [Table]
[Table] [Table]
arg
[Table] [Table]
: y
[Table] [Table]
←
[Table] [Table]
[Table] [Table]
[Table]
) 10: for k = 1 to K do 11:
L k = (cid:107) y
[Table]
Q (
[Table]
, a
[Table]
; θ
[Table]
) (cid:107) 22 + (cid:107) y − Q (
[Table]
, a
[Table]
) (cid:107) 22 + (cid:107) y (cid:48)
[Table]
− Q (
[Table]
, a
[Table]
) (cid:107) 22 + (cid:107) y (cid:48)
[Table]
−
Q ( s
[Table]
+1 , a †
[Table]
θ 2 ) (cid:107)
[Table] [Table]
k , θ 2 ← θ 2 − λ ∇ θ 2 L k 13: if L k < α L 1 then 14:
Break 15: end if 16: end for 17: ˆ a t ∼ π φ i ( s t ) 18:
J π ( φ ) =
E ( s t , ˆ a t ) [ Q ( s t , ˆ a t ; θ 1 )] 19: ¯ a t ← CEM( Q ( s t , · ; θ 1 ) , π φ i ( ·| s t ) ) 20: φ ← φ − λ ∇ φ J π ( φ ) − λ E ( s t , ˆ a t )
[ Q ( s t , ¯ a t ; θ 1 ) − Q ( s t , ˆ a t ; θ 1 )]
+ ∇ φ log π (¯ a t | s t ; φ ) 21: end for the next iteration will also increase leading to an explosion of the Q function. We demonstrated this behavior in an ablation experiment with results in Fig.
We also show how maintaining a separate target network [8] with frozen parameters θ − {+to compute y i +
+}
=
r t + γQ ( s t +1 , a ∗ t +1 ; θ − ) delays the update of the target and therefore leads to more stable learning of the Q function. However, delaying the function updates also comes with the price of slowing down the learning process. We propose a self-Regularized TD-learning approach to minimize the TD-error while also keeping the change of Q ( s t +1 , a ∗ t +1 ) small.
This regularization mitigates the divergence issue [13], and no longer requires a target network that would otherwise slow down the learning process.
Let y (cid:48) i = Q ( s t +1 , a ∗ t +1 ; θ i ) , and y i =
r t + γy (cid:48)
i .
We deﬁne the learning objective as
[Equation]
(2) where the ﬁrst term is the original TD-Learning objective and the second term is the regularization term penalizing large updates to Q ( s t +1 , a ∗ t +1 ) .
Note that when the current Q network updates its parameters θ , both Q ( s t , a t ) and Q ( s t +1 , a ∗ t +1 ) change.
Hence, the target value y i will also change which is different from the approach of keeping a frozen target network for a few iterations.
We will demonstrate in our experiments that this self-regularized TD-Learning approach removes the delays in the update of the target value thereby achieves faster and stable learning.
4.2 Self-Guided Policy Improvement with Evolution Strategies
The policy, known as the actor, can be updated through a combination of two parts.
The ﬁrst part, which we call Q-loss policy update, improves the policy through local gradients of the current Q function, while the second part, which we call CEM policy update, ﬁnds a high-value action via
CEM in a broader neighborhood of the Q function landscape and updates the action distribution to be around this high-value action.
We formally describe the two parts below.
Given states s t sampled from the replay buffer, the Q-loss policy update maximizes the objective
[Equation]
where ˆ a t is sampled from the current policy π ( ·| s t ) .
The gradient is taken through the reparameterization trick.
We reparameterize the policy using a neural network transformation as described by
Haarnoja et al.
[Equation]
where (cid:15) t is an input noise vector, sampled from a ﬁxed distribution, such as a standard multivariate
Normal distribution.
Then the gradient of J π ( φ ) is:
[Equation]
For the CEM policy update, given a minibatch of states s t , we ﬁrst ﬁnd a high-value action ¯ a t for each state by running CEM on the current Q function, ¯ a t =
CEM ( Q ( s t , · ) , π ( ·| s t )) .
Then the policy is updated to increase the probability of this high-value action.
The guided update on the parameter φ of π at iteration i is
[Equation]
We used Q ( s t , ˆ a t ) as a baseline term, since its expectation over actions ˆ a t will give us the normal baseline V ( s t ) :
[Equation]
In our implementation, we only perform an update if the improvement on the Q function, Q ( s t , ¯ a t ) −
Q ( s t , ˆ a t ) , is non-negative, to guard against the occasional cases where CEM fails to ﬁnd a better action.
Combining both parts, the ﬁnal update rule on the parameter φ i of policy π i is
[Equation]
where λ is the step size.
Let Q π be the state-action value function induced by the current policy.
We can prove that if the Q function has converged to Q π then both the Q-loss policy update and the CEM policy update will be guaranteed to improve the current policy.
We formalize this result in Theorem 1 and Theorem 2, and prove them in Appendix.
Theorem 1.
Q -loss Policy Improvement Starting from the current policy
π , we maximize the objective J π
=
E
( s,a ) ∼ ρ π ( s,a ) Q π ( s, a ) .
The maximization converges to a critical point denoted as π new .
Then the induced Q function, Q π new , satisﬁes ∀ ( s, a ) , Q π new ( s, a ) ≥ Q π ( s, a ) .
Theorem 2.
CEM Policy Improvement Assuming the CEM process is able to ﬁnd the optimal action of the state-action value function, a ∗ ( s ) =
arg max a Q π ( s, a ) , where Q π is the Q function induced by the current policy π .
By iteratively applying the update E ( s,a ) ∼ ρ π ( s,a ) [ Q ( s, a ∗ ) −
Q ( s, a )] + ∇ log π ( a ∗ | s ) , the policy converges to π new .
Then Q π new satisﬁes ∀ ( s, a ) , Q π new ( s, a ) ≥
Q π ( s, a ) .
4.3 Max-min Double Q-Learning
Q-learning [23] is known to suffer from overestimation [24].
Hasselt [25] proposed Double-Q learning which uses two Q functions with independent sets of weights to mitigate the overestimation problem.
Fujimoto et al.
[1] proposed Clipped Double Q-learning with two Q functions denoted as Q ( s, a ; θ 1 ) and Q ( s, a ; θ 2 ) , or Q 1 and Q 2 in short.
Given a transition ( s t , a t , r t , s t +1 ) , Clipped
Double Q-learning uses the minimum between the two estimates of the Q functions when calculating the target value in TD-error [6]:
[Equation]
where ˆ a t +1 is the predicted next action.
Fujimoto et al.
[1] mentioned that such an update rule may induce an underestimation bias.
In addition, ˆ a t +1 =
π φ ( s t +1 ) is the prediction of the actor network.
The actor network’s parameter φ is optimized according to the gradients of Q 1 .
In other words, ˆ a t +1 tends be selected according to the Q 1 network which consistently increases the discrepancy between the two Q-functions.
In practice, we observe that the discrepancy between the two estimates of the Q-function, | Q 1 − Q 2 | , can increase dramatically leading to an unstable learning process.
We found that a technique we call Max-min Double Q-Learning reduces the discrepancy between the Q-functions.
We ﬁrst select ˆ a t +1 according to the actor network π φ ( s t +1 ) .
Then we run CEM to search the landscape of Q 2 within a broad neighborhood of ˆ a t +1 to return a second action ˜ a t +1 .
Note that CEM selects an action ˜ a t +1 that maximises Q 2 while the actor network selects an action ˆ a t +1 that maximises Q 1 .
We gather four different Q-values: Q ( s t +1 , ˆ a t +1 ; θ 1 ) , Q ( s t +1 , ˆ a t +1 ; θ 2 ) ,
Q ( s t +1 , ˜ a t +1 ; θ 1 ) , and Q ( s t +1 , ˜ a t +1 ; θ 2 ) .
We then run a max-min operation to compute the target value that cancels the biases induced by ˆ a t +1 and ˜ a t +1 .
[Equation]
The inner min-operation min j =1 , 2 Q ( s t +1 , ˆ a t +1 ; θ j ) is adopted from Eq.
8 and mitigates overestimation [24].
The outer max operation helps to reduce the difference between Q 1 and Q 2 .
In addition, the max operation provides a better approximation of the Bellman optimality operator [6].
We visualize
Q 1 and Q 2 during the learning process in the supplementary material.
We formalize the convergence of the proposed Max-min Double Q-Learning approach in the ﬁnite MDP setting and prove this theorem in the Appendix.
Experiments
[Figure]
5.1 Comparative Evaluation
We present GRAC , a self-guided and self-regularized actor-critic algorithm as summarized in Algorithm 1.
To evaluate GRAC , we measure its performance on the suite of MuJoCo continuous control tasks [26], interfaced through OpenAI Gym [11].
We compare our method with DDPG [5], TD3 [1],
CEM-RL [9], TRPO [4], SAC 1 [2], and SAC 2 [27].
We use the source code released by the original authors and adopt the same hyperparameters reported in the original papers and the number of training steps according to SAC 1 [2].
For CEM-RL [9] and SAC 2 [27], we use the results provided by their corresponding authors.
SAC 2 does not contain results on Swimmer-v2.
CEM-RL does not contain results on Humanoid-v2 and only runs one millions step on every tested task.
Hyperparameters for all experiments are in the Appendix.
Results are shown in Figure 1.
GRAC outperforms or is comparable to all other algorithms in both ﬁnal performance and learning speed across all tasks.
On complex tasks with high state and action dimensions such as Humanoid-v2, GRAC outperforms all other algorithms by a large margin.
Both CEM-RL and GRAC leverage CEM.
CEM-RL applies
CEM within the actor parameter space which is extremely high-dimensional while GRAC utilizes
CEM within the action space of the Q function.
GRAC outperform CEM-RL on tasks such as Ant-v2,
HalfCheetah-v2, Swimmer-v2 by a large margin.
5.2 Ablation Study
[Figure]
In this section, we present ablation studies to understand the contribution of each proposed component: SelfRegularized TD-Learning (Section 4.1) and Self-Guided
Policy Improvment (Section 4.2).
We present our results in Fig. 4 in which we compare the performance of GRAC with alternatives, each removing one component from
GRAC.
Additional learning curves can be found in the Appendix.
We also run experiments to examine how sensitive
GRAC is to some hyperparameters such as α and K listed in Alg. 1, and the results can be found in the Appendix.
Self-Regularized TD Learning To verify the effectiveness of the proposed self-regularized TD-learning method, we apply our method to DDPG (DDPG w/o target network w/ target regularization).
We compare against two baselines: the original DDPG and DDPG without target networks for both actor and critic ( DDPG w/o target network ).
We choose DDPG, because it does not have additional components such as
Double Q-Learning, which may complicate the analysis of this comparison.
In Fig.
2, we visualize the average returns, and average Q 1 values over training batches ( y (cid:48) 1 in Alg.1).
The Q 1 values of DDPG w/o target network changes dramatically which leads to poor average returns.
DDPG maintains stable Q values but makes slow progress.
Our proposed DDPG w/o target network w/ target regularization maintains stable Q values.
In addition, we compare the average returns of
DDPG w/o target network, w/ target regularization and DDPG within one million steps over four random seeds.
DDPG w/o target network, w/ target regularization outperforms DDPG by large margins in ﬁve out of six Mujoco tasks (Fig.
2 shows results on Hopper-v2.
The remaining results can be found in the Appendix).
We also apply self-regularized TD-learning to DQN called DQN w/o target network w/ target regularization on the Atari Breakout environment and it outperforms DQN by 25%.
The learning curve over 50 million steps is shown in Fig.3.
These results on two different
Q-learning methods demonstrate the effectiveness of our method and its potentials to be applied to a wide range of DRL methods.
Policy Improvement with Evolution Strategies
The GRAC actor network uses a combination of two actor loss functions, denoted as QLoss and CEMLoss .
QLoss refers to the unbiased gradient estimators which extend the DDPG -style policy gradients [5] to stochastic policies.
CEMLoss represents the policy improvement guided by the action found with the zero-order optmization method CEM.
We run another two ablation experiments on all six control tasks and compare it with our original policy training method denoted as GRAC .
As seen in Fig.
4, in general GRAC achieves a better performance compared to either using CEMLoss or QLoss .
The signiﬁcance of the improvements varies over the six control tasks.
For example, CEMLoss plays a dominant role in Swimmer while QLoss has a major effect in HalfCheetah.
This suggests that the CEMLoss and
QLoss are complementary
[Figure]
, HalfCheetah-v2). This suggests that individual components of GRAC complement each other.
5.3 In Hand Manipulation
We evaluate our approach on the problem of in-hand manipulation which remains unsolved due the high dimensionality of the problem and the complexity of multi-contact control [28].
Having this capability would allow robots to perform sophisticated tasks requiring repositioning and reorienting of grasped objects.
We apply
GRAC to a non-anthropomorphic robotic hand with 9 degrees of freedom [29].
We train the robotic hand in simulation to rotate a cube by 50 degrees around the gravity direction.
The input states are the current, previous, initial, and target object position and orientation, and all nine gripper joint positions at the current time step.
The actions are nine joint positions for the gripper at the next time step.
GRAC learns a policy successfully accomplishing the task within 500k iterations.
We test the learned policy on the real hand.
Videos are included in the supplementary material.
Conclusion
Leveraging neural networks as function approximators, DRL has been successfully demonstrated on a range of decision-making and control tasks.
However, the nonlinear function approximators also introduce issues such as divergence and overestimation.
As our main contribution, we proposed a self-regularized TD-learning method to address divergence without requiring a target network that may slow down learning progress.
The proposed method is agnostic to the speciﬁc Q-learning method and can be added to any of them.
In addition, we propose self-guided policy improvement by combining policy-gradient with zero-order optimization such as the Cross Entropy Method.
This helps to search for actions associated with higher Q-values in a broad neighborhood and is robust to local noise in the Q function approximation.
Taken together, these components deﬁne GRAC , a novel self-guided and self-regularized actor critic algorithm.
We evaluate GRAC on the OpenAI gym tasks, outperforming popular methods such as TD3 [1] and SAC [2] on four tasks and achieving competitive results on two environments.
We also run GRAC to enable a non-anthropomorphic robotic hand to successfully accomplish an in-hand manipulation task in the real world. Acknowledgments
This work is supported by Toyota Research Institute (TRI).
It is also supported by Siemens and the
HAI-AWS Cloud Credits.
We thank reviewers who gave useful comments.