GRAC: Self-Guided and Self-Regularized Actor-Critic
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Deep reinforcement learning (DRL) algorithms have successfully beendemonstrated on a range of challenging decision making and control tasks.
Onedominant component of recent deep reinforcement learning algorithms is the targetnetwork which mitigates the divergence when learning the Q function.
However,target networks can slow down the learning process due to delayed function updates.
Our main contribution in this work is a self-regularized TD-learning method toaddress divergence without requiring a target network.
Additionally, we propose aself-guided policy improvement method by combining policy-gradient with zeroorder optimization to search for actions associated with higher Q-values in a broadneighborhood.
This makes learning more robust to local noise in the Q functionapproximation and guides the updates of our actor network.
Taken together, thesecomponents deﬁne GRAC , a novel self-guided and self-regularized actor criticalgorithm.
We evaluate GRAC on the OpenAI gym tasks, outperforming popularmethods such as TD3 [1] and SAC [2] on four tasks and achieving competitive results on two environments.
We also apply GRAC to enable a non-anthropomorphicrobotic hand to successfully accomplish an in-hand manipulation task in the realworld.
Introduction
Reinforcement learning (RL) studies decision-making with the goal of maximizing total discountedreward when interacting with an environment.
Leveraging high-capacity function approximators suchas neural networks, Deep reinforcement learning (DRL) algorithms have been successfully applied toa range of challenging domains, from video games [3] to robotic control [4]
Actor-critic algorithms are among the most popular approaches in DRL, e g. DDPG [5], TRPO [4],
TD3 [1] and SAC [2].
These methods are based on policy iteration, which alternates between policyevaluation and policy improvement [6].
Actor-critic methods jointly optimize the value function(critic) and the policy (actor) as it is often impractical to run either of these to convergence [2].
In DRL, both the actor and critic use deep neural networks as the function approximator.
However,
DRL is known to assign unrealistically high values to state-action pairs represented by the Q-function.
This is detrimental to the quality of the greedy control policy derived from Q [7].
Mnih et al.
[8]proposed to use a target network to mitigate divergence.
A target network is a copy of the current Qfunction that is held ﬁxed to serve as a stable target within the TD error update.
The parameters of thetarget network are either infrequently copied [8] or obtained by Polyak averaging [5].
A limitation ofusing a target network is that it can slow down learning due to delayed function updates.
We proposean approach that reduces the need for a target network in DRL while still ensuring stable learningand good performance in high-dimensional domains.
We add a self-regularization term to encouragesmall changes to the target value while minimizing the Temporal Difference (TD)-error [6].
Evolution Strategies (ES) are a family of black-box optimization algorithms which are typicallyvery stable, but scale poorly in high-dimensional search spaces e.g. neural networks [9].
Policygradient-based DRL methods, unlike evolutionary search methods, can continue to sample previous experiences to improve value estimation, particularly in the off-policy setting.
At the same time,these approaches can also be unstable and highly sensitive to hyper-parameter tuning [9].
Wepropose a novel policy improvement method which combines both approaches to get the best of bothworlds.
Speciﬁcally, after the actor network ﬁrst outputs an initial action, we apply the Cross Entropy Method (CEM) [10] to search the neighborhood of the initial action to ﬁnd a second action associatedwith a higher Q value.
Then we leverage the second action in the policy improvement stage to speedup the learning process.
Our main contribution in this work is a self-regularized TD-learning method to address divergencewithout requiring a target network that may slow down learning progress.
In addition, we propose aself-guided policy improvement method which combines policy-gradients and zero-order optimization.
This helps to speed up learning and is robust to local noise in the Q function approximation.
Taken together, these components deﬁne GRAC , a novel self-guided and self-regularized actor criticalgorithm.
We evaluate GRAC on six continuous control domains from OpenAI gym [11], where
GRAC outperforms popular methods such as TD3 [1] and SAC [2] on four tasks and achieves competitive results on two environments.
We run our experiments across a large number of seeds withfair evaluation metrics [12], perform extensive ablation studies, and open source both our codeand learning curves.
We also run GRAC to enable a non-anthropomorphic, real robotic hand tosuccessfully rotate a cube to the target pose.
Related Work
The proposed algorithm incorporates key ingredients within the actor-critic method: a self-regularized
TD update and self-guided policy improvements based on evolution strategies.
In this section, wereview prior work related to these ideas.
Divergence in Deep Q-Learning In Deep Q-Learning, we use a nonlinear function approximatorsuch as a neural network to approximate the Q-function that represents the value of each state-actionpair.
Learning the Q-function in this way is known to suffer from divergence issues [13] such asassigning unrealistically high values to state-action pairs [7].
For the case when the control policy isgreedily derived from Q [7], unrealistically high Q values are detrimental.
To mitigate the divergenceissue, Mnih et al.
[8] introduce a target network which is a copy of the estimated Q-function andis held ﬁxed to serve as a stable target for some number of steps.
However, because of the delayedfunction updates, target networks can slow down learning [14].
Durugkar and Stone [15] propose Constrained Q-Learning, which uses a constraint to prevent the average target value from changingafter an update.
Achiam et al. [16] give a simple analysis based on a linear approximation of the
Q function and develop a stable Deep Q-Learning algorithm for continuous control without targetnetworks.
However, their proposed method requires separately calculating backward passes for eachstate-action pair in the batch, and solving a system of equations at each timestep.
Bhatt et al.
[17]introduce a normalization called cross-normalization which is regarded as an extension of batchnormalization that re-centers data for on- and off-policy transitions.
Peng et al.
[18] uses Monte Carlo instead of TD error to update the Q network and removes the need for a target network.
Ourproposed GRAC algorithm adds a self-regularization term to the TD-Learning objective to keep thechange of the state-action value small.
Evolution Strategies in Deep Reinforcement Learning Evolution Strategies are typically stablebut suffer from scaling to high-dimensional search spaces.
Policy gradient-based deep RL methods,such as DDPG [5] can continue to reuse previous experience to improve value estimations, particularlyin the off-policy setting, but can be unstable and highly sensitive to hyper-parameter tuning [9].
Researchers have proposed to combine these approaches to get the best of both worlds.
Pourchotand Sigaud [9] proposed CEM-RL to combine CEM with either DDPG [5] or TD3 [1].
However, CEM-RL applies CEM within the actor parameter space which is extremely high-dimensional, makingthe search not efﬁcient.
Kalashnikov et al.
[19] introduce QT-Opt , which leverages CEM to searchthe landscape of the Q function, and enables Q-Learning in continuous action spaces without usingan actor.
Based on QT-Opt , Simmons-Edler et al.
[20] leverage CEM to search the landscape the Q-function but propose to initialize CEM with a Gaussian prior covering the action space, independentof observations.
However, CEM does not scale well to high-dimensional action spaces [21], such asin the Humanoid task.
We ﬁrst let the actor network output an initial Gaussian action distributionconditioned on current state.
Then we use CEM to search for an action with a higher Q value than the
Q value of the Gaussian mean.
Starting from the predicted distribution, we show that GRAC speedsup the learning process compared to popular actor-critic methods.
Preliminaries
In this section, we deﬁne the notation used in subsequent sections.
Consider a Markov Decision
Process (MDP), deﬁned by the tuple ( S , A , P , r, ρ 0 , γ ) , where S is a ﬁnite set of states, A is a ﬁniteset of actions, P : S ×
A × S → R is the transition probability distribution, r : S ×
A → R is thereward function, ρ 0 : S → R is the distribution of the initial state s 0 , and γ ∈ [0 , 1] is the discountfactor.
At each discrete time step t , with a given state s t ∈ S , the agent selects an action a t ∈ A ,receiving a reward r and the new state s t +1 of the environment.
Let π denote the policy which maps a state to a probability distribution over the actions, π : S →
P ( A ) .
The return from a state is deﬁned as the sum of discounted reward R t =
(cid:80) i =
t γ i − t r
( s i , a i ) .
In reinforcement learning, the objective is to ﬁnd the optimal policy π ∗ , with parameters φ , which maximizes the expected return J ( φ )
=
(cid:80) t E ( s t ,a t ) ∼ ρ π ( s t ,a t ) [ γ t r ( s t , a t )] where ρ π ( s t ) and ρ π ( s t , a t )denote the state and state-action marginals of the trajectory distribution induced by the policy π ( a t
| s t ) .
We use the following standard deﬁnitions of the state-action value function Q π .
It describes theexpected discounted reward after taking an action a t in state s t and thereafter following policy π :
[Equation]
In this work we use CEM to ﬁnd optimal actions with maximum Q values.
CEM is a randomized zeroorder optimization algorithm.
To ﬁnd the action a that maximizes Q ( s, a ) , CEM is initialized with aparamaterized distribution over a , P ( a ; ψ ) .
Then it iterates between the following two steps [22]:
First generate a 1 , . .
. , a N ∼ P ( s ; ψ ) .
Retrieve their Q values Q ( s, a i ) and sort the actions to havedecreasing Q values.
Then keep the ﬁrst K actions, and solve for updated parameters ψ (cid:48) :
[Equation]
In the following, let CEM ( Q ( s, · ) , π ( ·| s ))
denote the action found by CEM to maximize Q ( s, · ) ,when CEM is initialized with the distribution predicted by the policy.
Technical Approach
4.1 Self-Regularized TD Learning
Reinforcement learning is prone to instability and divergence when a nonlinear function approximatorsuch as a neural network is used to represent the Q function [13].
Mnih et al.
[8] identiﬁed severalreasons for this.
One is the correlation between the current action-values and the target value.
Updatesto Q ( s t , a t ) often also increase Q ( s t +1 , a ∗ t +1 ) where a ∗ t +1 is the optimal next action.
Hence, theseupdates also increase the target value y t which may lead to oscillations or the divergence of the policy.
More formally, given transitions ( s t , a t , r t , s t +1 ) sampled from the replay buffer distribution B , the
Q network can be trained by minimising the loss functions
L ( θ i ) at iteration i :
[Equation]
where for now let us assume y i =
r t + γ max a Q
( s t +1 , a ; θ i ) to be the target for iteration i computedbased on the current Q network parameters θ i .
a ∗ t +1 = arg max a Q ( s t +1 , a ) .
If we update theparameter θ i +1 to reduce the loss L ( θ i ) , it changes both Q ( s t , a t ; θ i +1 ) and Q ( s t +1 , a ∗ t +1 ; θ i +1 ) .
Assuming an increase in both values, then the new target value y
i +
=
r t + γQ ( s t +1 , a ∗ t +1 ;
θ i +1 )
forthe next iteration will also increase leading to an explosion of the Q function.
We demonstrated thisbehavior in an ablation experiment with results in Fig.
We also show how maintaining a separatetarget network [8] with frozen parameters θ − to compute y i +
=
r t + γQ ( s t +1 , a ∗ t +1 ; θ − ) delays theupdate of the target and therefore leads to more stable learning of the Q function.
However, delayingthe function updates also comes with the price of slowing down the learning process.
We propose a self-Regularized TD-learning approach to minimize the TD-error while also keepingthe change of Q ( s t +1 , a ∗ t +1 ) small.
This regularization mitigates the divergence issue [13], and
Initialize critic network Q θ 1 , Q θ 2 and actor network π φ with random parameters θ 1 , θ 2 and φ
Initialize replay buffer B , Set α <1: for i = 1 , . . .
do 2:
Select action a ∼ π φ i ( s ) , observe reward r and new state s (cid:48) 3:
Store transition tuple ( s, a, r, s (cid:48) ) in B 4:
Sample mini-batch of N transitions ( s t , a t , r t , s t +1 ) from B .
[Table]
a
[Table]
i ( s t +1 )
[Table] [Table]
a
[Table]
← CEM ( Q (
[Table] [Table]
[Table] [Table]
arg
[Table] [Table]
: y
[Table] [Table]
←
[Table] [Table]
[Table] [Table]
[Table]
) 10: for k = 1 to K do 11:
L k = (cid:107) y
[Table]
Q (
[Table]
, a
[Table]
; θ
[Table]
) (cid:107) 22 + (cid:107) y − Q (
[Table]
, a
[Table]
) (cid:107) 22 + (cid:107) y (cid:48)
[Table]
− Q (
[Table]
, a
[Table]
) (cid:107) 22 + (cid:107) y (cid:48)
[Table]
−
Q ( s
[Table]
+1 , a †
[Table]
θ 2 ) (cid:107)
[Table] [Table]
k , θ 2 ← θ 2 − λ ∇ θ 2 L k 13: if L k < α L 1 then 14:
Break 15: end if 16: end for 17: ˆ a t ∼ π φ i
[Equation]
s t ) 18:
J
[Equation]
E
[Equation]
t
[Equation]
t
[Equation] [Equation]
19: ¯ a t ← CEM( Q ( s t , · ; θ 1 ) , π φ i ( ·| s t ) ) 20: φ ← φ −
[Equation]
∇
[Equation] [Equation] [Equation]
[Equation]
21: end for no longer requires a target network that would otherwise slow down the learning process.
Lety (cid:48) i = Q ( s t +1 , a ∗ t +1 ; θ i ) , and y i =
r t + γy (cid:48)
i .
We deﬁne the learning objective as
[Equation]
(2) where the ﬁrst term is the original TD-Learning objective and the second term is the regularizationterm penalizing large updates to Q ( s t +1 , a ∗ t +1 ) .
Note that when the current Q network updates itsparameters θ , both Q ( s t , a t ) and Q ( s t +1 , a ∗ t +1 ) change.
Hence, the target value y i will also changewhich is different from the approach of keeping a frozen target network for a few iterations.
We willdemonstrate in our experiments that this self-regularized TD-Learning approach removes the delaysin the update of the target value thereby achieves faster and stable learning.
4.2 Self-Guided Policy Improvement with Evolution Strategies
The policy, known as the actor, can be updated through a combination of two parts.
The ﬁrst part,which we call Q-loss policy update, improves the policy through local gradients of the current Qfunction, while the second part, which we call CEM policy update, ﬁnds a high-value action via
CEM in a broader neighborhood of the Q function landscape and updates the action distribution to bearound this high-value action.
We formally describe the two parts below.
Given states s t sampled from the replay buffer, the Q-loss policy update maximizes the objective
[Equation]
where ˆ a t is sampled from the current policy π ( ·| s t ) .
The gradient is taken through the reparameterization trick.
We reparameterize the policy using a neural network transformation as described by
Haarnoja et al.
[Equation]
where (cid:15) t is an input noise vector, sampled from a ﬁxed distribution, such as a standard multivariate
Normal distribution.
Then the gradient of J π ( φ ) is:
[Equation]
For the CEM policy update, given a minibatch of states s t , we ﬁrst ﬁnd a high-value action ¯ a t foreach state by running CEM on the current Q function, ¯ a t =
CEM ( Q ( s t , · ) , π ( ·| s t )) .
Then the policyis updated to increase the probability of this high-value action.
The guided update on the parameter φof π at iteration i is
[Equation]
We used Q ( s t , ˆ a t ) as a baseline term, since its expectation over actions ˆ a t will give us the normalbaseline V ( s t ) :
[Equation]
In our implementation, we only perform an update if the improvement on the Q function, Q ( s t , ¯ a t ) −
Q ( s t , ˆ a t ) , is non-negative, to guard against the occasional cases where CEM fails to ﬁnd a betteraction.
Combining both parts, the ﬁnal update rule on the parameter φ i of policy π i is
[Equation]
where λ is the step size.
Let Q π be the state-action value function induced by the current policy.
We can prove that if the Qfunction has converged to Q π then both the Q-loss policy update and the CEM policy update will beguaranteed to improve the current policy.
We formalize this result in Theorem 1 and Theorem 2, andprove them in Appendix.
Theorem 1.
Q -loss Policy Improvement Starting from the current policy
π , we maximize the objective J π
=
E
( s,a ) ∼ ρ π ( s,a ) Q π ( s, a ) .
The maximization converges to a critical point denoted as π new .
Then the induced Q function, Q π new , satisﬁes ∀ ( s, a ) , Q π new ( s, a ) ≥ Q π ( s, a ) .
Theorem 2.
CEM Policy Improvement Assuming the CEM process is able to ﬁnd the optimalaction of the state-action value function, a ∗ ( s ) =
arg max a Q π ( s, a ) , where Q π is the Q functioninduced by the current policy π .
By iteratively applying the update E ( s,a ) ∼ ρ π ( s,a ) [ Q ( s, a ∗ ) −
Q ( s, a )] + ∇ log π ( a ∗ | s ) , the policy converges to π new .
Then Q π new satisﬁes ∀ ( s, a ) , Q π new ( s, a ) ≥
Q π ( s, a ) .
4.3 Max-min Double Q-Learning
Q-learning [23] is known to suffer from overestimation [24].
Hasselt [25] proposed Double-Qlearning which uses two Q functions with independent sets of weights to mitigate the overestimationproblem.
Fujimoto et al.
[1] proposed Clipped Double Q-learning with two Q functions denotedas Q ( s, a ; θ 1 ) and Q ( s, a ; θ 2 ) , or Q 1 and Q 2 in short.
Given a transition ( s t , a t , r t , s t +1 ) , Clipped
Double Q-learning uses the minimum between the two estimates of the Q functions when calculatingthe target value in TD-error [6]: [Equation]
[Equation]
where ˆ a t +1 is the predicted next action.
Fujimoto et al.
[1] mentioned that such an update rule may induce an underestimation bias.
Inaddition, ˆ a t +1 =
π φ ( s t +1 ) is the prediction of the actor network.
The actor network’s parameterφ is optimized according to the gradients of Q 1 .
In other words, ˆ a t +1 tends be selected accordingto the Q 1 network which consistently increases the discrepancy between the two Q-functions.
Inpractice, we observe that the discrepancy between the two estimates of the Q-function, | Q 1 − Q 2 | ,can increase dramatically leading to an unstable learning process.
We found that a technique we call Max-min Double Q-Learning reduces the discrepancy betweenthe Q-functions.
We ﬁrst select ˆ a t +1 according to the actor network π φ ( s t +1 ) .
Then we run CEM to search the landscape of Q 2 within a broad neighborhood of ˆ a t +1 to return a second action ˜ a t +1 .
Note that CEM selects an action ˜ a t +1 that maximises Q 2 while the actor network selects an actionˆ a t +1 that maximises Q 1 .
We gather four different Q-values: Q ( s t +1 , ˆ a t +1 ; θ 1 ) , Q ( s t +1 , ˆ a t +1 ; θ 2 ) ,
Q ( s t +1 , ˜ a t +1 ; θ 1 ) , and Q ( s t +1 , ˜ a t +1 ; θ 2 ) .
We then run a max-min operation to compute the targetvalue that cancels the biases induced by ˆ a t +1 and ˜ a t +1 .
[Equation]
The inner min-operation min j =1 , 2 Q ( s t +1 , ˆ a t +1 ; θ j ) is adopted from Eq.
8 and mitigates overestimation [24].
The outer max operation helps to reduce the difference between Q 1 and Q 2 .
In addition, themax operation provides a better approximation of the Bellman optimality operator [6].
We visualize
Q 1 and Q 2 during the learning process in the supplementary material.
We formalize the convergenceof the proposed Max-min Double Q-Learning approach in the ﬁnite MDP setting and prove thistheorem in the Appendix.
Experiments
[Figure]
5.1 Comparative Evaluation
We present GRAC , a self-guided and self-regularized actor-critic algorithm as summarized in Algorithm 1.
To evaluate GRAC , we measure its performance on the suite of MuJoCo continuous controltasks [26], interfaced through OpenAI Gym [11].
We compare our method with DDPG [5], TD3 [1],
CEM-RL [9], TRPO [4], SAC 1 [2] [Figure] and SAC 2 [27].
We use the source code released by the originalauthors and adopt the same hyperparameters reported in the original papers and the number of trainingsteps according to SAC 1 [2].
For CEM-RL [9] and SAC 2 [27], we use the results provided by their corresponding authors.
SAC 2 does not contain results on Swimmer-v2.
CEM-RL does not containresults on Humanoid-v2 and only runs one millions step on every tested task.
Hyperparametersfor all experiments are in the Appendix.
Results are shown in Figure 1.
GRAC outperforms or iscomparable to all other algorithms in both ﬁnal performance and learning speed across all tasks.
On complex tasks with high state and action dimensions such as Humanoid-v2, GRAC outperformsall other algorithms by a large margin.
Both CEM-RL and GRAC leverage CEM.
CEM-RL applies
CEM within the actor parameter space which is extremely high-dimensional while GRAC utilizes
CEM within the action space of the Q function.
GRAC outperform CEM-RL on tasks such as Ant-v2,
HalfCheetah-v2, Swimmer-v2 by a large margin.
5.2 Ablation Study
[Figure]
slowly.
If we remove the target network and add the proposed target regularization, we both maintain stability and achieve faster learning than DDPG .
In this section, we present ablation studies to understand the contribution of each proposed component: SelfRegularized TD-Learning (Section 4.1) and Self-Guided
Policy Improvment (Section 4.2).
We present our resultsin Fig. 4 in which we compare the performance of GRACwith alternatives, each removing one component from
GRAC.
Additional learning curves can be found in the Appendix.
We also run experiments to examine how sensitive
GRAC is to some hyperparameters such as α and K listedin Alg. 1, and the results can be found in the Appendix.
Self-Regularized TD Learning To verify the effectiveness of the proposed self-regularized TD-learning method,we apply our method to DDPG (DDPG w/o target network w/ target regularization).
We compare against twobaselines: the original DDPG and DDPG without target networks for both actor and critic ( DDPGw/o target network ).
We choose DDPG, because it does not have additional components such as
Double Q-Learning, which may complicate the analysis of this comparison.
In Fig.
2, we visualize the average returns, and average Q 1 values over training batches ( y (cid:48) 1 in Alg.1).
The Q 1 values of DDPG w/o target network changes dramatically which leads to poor average returns.
DDPG maintains stable Q values but makes slow progress.
Our proposed DDPG w/o target networkw/ target regularization maintains stable Q values.
In addition, we compare the average returns of
DDPG w/o target network, w/ target regularization and DDPG within one million steps over fourrandom seeds.
DDPG w/o target network, w/ target regularization outperforms DDPG by largemargins in ﬁve out of six Mujoco tasks (Fig.
2 shows results on Hopper-v2.
The remaining resultscan be found in the Appendix).
We also apply self-regularized TD-learning to DQN called DQN w/otarget network w/ target regularization on the Atari Breakout environment and it outperforms DQNby 25%.
The learning curve over 50 million steps is shown in Fig.3.
These results on two different
Q-learning methods demonstrate the effectiveness of our method and its potentials to be applied to awide range of DRL methods.
Policy Improvement with Evolution Strategies
The GRAC actor network uses a combination oftwo actor loss functions, denoted as QLoss and CEMLoss .
QLoss refers to the unbiased gradient estimators which extend the DDPG -style policy gradients [5] to stochastic policies.
CEMLossrepresents the policy improvement guided by the action found with the zero-order optmizationmethod CEM.
We run another two ablation experiments on all six control tasks and compare itwith our original policy training method denoted as GRAC .
As seen in Fig.
4, in general GRACachieves a better performance compared to either using CEMLoss or QLoss .
The signiﬁcance ofthe improvements varies over the six control tasks.
For example, CEMLoss plays a dominant rolein Swimmer while QLoss has a major effect in HalfCheetah.
This suggests that the CEMLoss and
QLoss are complementary.
[Figure]
components of complement
5.3 In Hand Manipulation
We evaluate our approach on the problem of in-hand manipulationwhich remains unsolved due the high dimensionality of the problem and the complexity of multi-contact control [28].
Having thiscapability would allow robots to perform sophisticated tasks requiring repositioning and reorienting of grasped objects.
We apply
GRAC to a non-anthropomorphic robotic hand with 9 degrees offreedom [29].
We train the robotic hand in simulation to rotate acube by 50 degrees around the gravity direction.
The input statesare the current, previous, initial, and target object position andorientation, and all nine gripper joint positions at the current timestep.
The actions are nine joint positions for the gripper at the nexttime step.
GRAC learns a policy successfully accomplishing thetask within 500k iterations.
We test the learned policy on the realhand.
Videos are included in the supplementary material.
Conclusion
Leveraging neural networks as function approximators, DRL has been successfully demonstratedon a range of decision-making and control tasks.
However, the nonlinear function approximatorsalso introduce issues such as divergence and overestimation.
As our main contribution, we proposeda self-regularized TD-learning method to address divergence without requiring a target networkthat may slow down learning progress.
The proposed method is agnostic to the speciﬁc Q-learningmethod and can be added to any of them.
In addition, we propose self-guided policy improvement bycombining policy-gradient with zero-order optimization such as the Cross Entropy Method.
Thishelps to search for actions associated with higher Q-values in a broad neighborhood and is robust tolocal noise in the Q function approximation.
Taken together, these components deﬁne GRAC , a novelself-guided and self-regularized actor critic algorithm.
We evaluate GRAC on the OpenAI gym tasks,outperforming popular methods such as TD3 [1] and SAC [2] on four tasks and achieving competitiveresults on two environments.
We also run GRAC to enable a non-anthropomorphic robotic hand tosuccessfully accomplish an in-hand manipulation task