Stochastic Policy Optimization with Heuristic Information for Robot Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Stochastic policy-based deep reinforcement learning (RL) approacheshave remarkably succeeded to deal with continuous control tasks.
However, applying these methods to manipulation tasks remains a challenge since actuatorsof a robot manipulator require high dimensional continuous action spaces.
Inthis paper, we propose exploration-bounded exploration actor-critic (EBE-AC),a novel deep RL approach to combine stochastic policy optimization with interpretable human knowledge.
The human knowledge is deﬁned as heuristic information based on both physical relationships between a robot and objects andbinary signals of whether the robot has achieved certain states.
The proposed approach, EBE-AC, combines an off-policy actor-critic algorithm with an entropymaximization based on the heuristic information.
On a robotic manipulation task,we demonstrate that EBE-AC outperforms prior state-of-the-art off-policy actorcritic deep RL algorithms in terms of sample efﬁciency.
In addition, we found that
EBE-AC can be easily combined with latent information, where EBE-AC with latent information further improved sample efﬁciency and robustness.
Introduction
A challenge to human-level AI [1] is still ongoing research area and recently deep reinforcementlearning (RL) approaches have remarkably shown outstanding results beyond human in the ﬁelds ofboard and video games [2, 3].
As a greatly unexplored area which adapts deep RL, robot manipulation is one of the challenging applications due to complex nonlinear dynamics and huge continuousaction spaces [4]-[7].
Various and stable robot manipulation skills in real-world environments [8],where the laws of physics are strictly applied, are required to solve complex tasks, such as moving,grasping, pushing and placing.
It is, therefore, still a challenge to solve complex tasks through deep RL approaches.
In front of the advanced dexterity capabilities of human hands, learning the dexterity with functionality to autonomously pick and place objects has always been a long-standing challenge forrobotics [9, 10].
To solve complex manipulation tasks, various approaches have been presentedto utilize human knowledge, which includes learning from human demonstrations and optimizinghyper-parameters empirically [11]-[18].
The ﬁrst improves sample efﬁciency of learning by providing good expert demonstration data to solve a given task [13]-[16].
In general, the human demonstrations can be used not only as a loss function of inverse RL but also as reusable experience dataof off-policy RL.
However, it is expensive to obtain real-world demonstrations for every single task.
As the second approach utilizing human knowledge, empirical hyper-parameter tuning [19], hashelped optimize learning progress since hyper-parameters are highly related to not only neural network architectures but also reward designs in the deep RL.
On the whole, hyper-parameter tuningis widely used to ensure that learning algorithms gain good performance, however, it is vulnerableto initial hyper-parameter settings for each task [18].
This results in the inefﬁciency of the learningprocess because it is necessary to perform hyper-parameter tuning every task.
A heuristic (informed) strategy using problem-speciﬁc knowledge has been generally known asa more efﬁcient solution than an uninformed strategy using only the problem itself [20].
Sincemanipulation skills are under the laws of physics, informed human knowledge, such as moving inthe direction of narrowing the distance, being in alignment with the target object, and grasping thecenter of gravity of the object, can be helpful to ﬁnd solutions efﬁciently in robot manipulation.
In this paper, we propose a novel deep RL approach to combine stochastic policy optimization withhuman knowledge, which is exploration-bounded exploration actor-critic (EBE-AC).
EBE-AC applies interpretable and reusable human knowledge, heuristic information based on both physical relationships between a robot and objects and binary signals of whether the robot has achieved certainstates, to an off-policy actor-critic algorithm.
The main contributions of this paper are summarizedas follows:
• We design a novel information theoretic metric in order for EBE-AC to have dependenciesbetween actions and heuristic information.•
We propose an (cid:15) -exploration method using two types of stochastic policies, exploration andbounded exploration, in order to store diverse experiences in a shared replay buffer. • We propose a temperature optimization for given entropies corresponding to the two poli- cies, in order to balance between the two.
To verify the performance of EBE-AC, we conducted experiments on a robotic manipulation task,
Pick-and-Lift .
The experimental results show that EBE-AC outperforms prior state-of-the-art offpolicy actor-critic deep RL algorithms in terms of sample efﬁciency.
In addition, we found that
EBE-AC can be easily combined with latent information as DIAYN [21] does, where EBE-AC withlatent information further improved sample efﬁciency and robustness.
Related Work
Due to the difﬁculty of obtaining stable and steady improvement of deterministic policy in applyingcomplex and high dimensional tasks, stochastic policy optimization based on information theoryhas been examined in problems with continuous state-action spaces [22]-[24].
Many works showthat applying a stochastic policy’s entropy to an objective function results in good performances.
In particular, regularizing both policy and value functions with entropy shows the convergence andstability improvements during training processes.
Through maximum entropy RL [23], a policy is encouraged to perform diverse actions to widelyexplore a state space while maximizing expected returns.
Owing to the diverse actions being able tomaximize expected returns, the learned policy can have multiple optimal solutions and obtain stableperformance regardless of random initial states.
In addition, the maximum entropy approaches arealso robust to environmental disturbances such as the presence of adversaries and dynamics changeover time because these works focus on the entropy of the action distribution for a given state.
In the unsupervised emergence of diverse skills dictating the states [21], the mutual informationbetween states and latent variables is leveraged to autonomously discover diverse skills.
To ensurethe different skills being distinguishable, the mutual information between actions and skills shouldbe minimized for a given state.
By utilizing the mutual information as a metric of the objectivefunction, a stochastic policy can be also stably and steadily improved under the consideration ofadditional latent variables.
These prior works utilize rich exploration to obtain various experiences at the beginning of learning, and tend to reduce the exploration as the learning progresses in order to acquire good expectedreturns stably.
However, when a policy is stuck in inevitable local optimums which must be passedto complete a given complex task, the policy needs to utilize rich exploration in order to escape thelocal optimums even in the middle of learning.
In other words, it is necessary that a policy simultaneously pursues rich exploration and stable expected returns throughout the learning period.
Thispaper proposes a novel way to combine human knowledge with a stochastic policy by leveragingthe information theoretic metrics.
Unlike reducing the mutual information between actions and latent variables in [21], the proposed method increases the mutual information between actions andheuristic information based on human knowledge.
As a result, the dependency between actions andheuristic information will be guaranteed and experiences obtained by actions are related to heuristic information.
Based on the metric, (cid:15) -exploration method using two types of stochastic policies is proposed in order to maintain rich exploration throughout the learning period.
Based on the proposedmethod, the sample efﬁciency is also improved and the stochastic policy achieves the robustness tocomplex tasks.
Preliminaries3.1 Notation
A Markov decision process is deﬁned by the tuple ( S , A , p, r ) , where S is a state space, A is acontinuous action space, p : S ×
A × S → R + is a transition probability distribution, and r :
S ×
A → R is a reward function.
In episodic tasks, a discounted return at time t is given as
G t = (cid:80) Tk =
t +1 γ k − t − 1 r ( s k , a k ) , where T is an episode horizon, γ is a discount factor, and r ( s k , a k )is a reward for the state-action pair ( s k , a k ) .
A value function for a policy π ( a t | s t ) is represented asv π
( s t ) =
E π
[ G t ] .
3.2 Objective function
We consider combining RL with the information theoretic metrics, which can encourage policyoptimization in both ways to diverse actions and speciﬁc actions in a given state.
Therefore, wepropose an objective function to learn a policy conditioned on heuristic information according tohuman knowledge, where the policy can select actions considering not only the states but also theheuristic information.
First, we deﬁne an information theoretic reward as
[Equation]
where z t is heuristic information, I ( a t ; z t | s t ) is a mutual information between a t and z t for thegiven s t , h ρ ( · ) is an entropy for the distribution ρ , w s and w z are positive temperatures, and ρ sand ρ z are distributions of action a t conditioned on s t and ( s t , z t ) , respectively.
The distributionsρ s and ρ z are induced by a policy π .
The heuristic information z t is deﬁned by external functionz t = f
( s t ) based on s t , where f ( · ) can be physical information between a robot and objects inaddition to binary signals representing speciﬁc states.
Maximizing the reward (cid:101) r
( s t , a t ) in Equation (1) means that a policy has to ﬁnd an optimal solutionto maximize the reward from the environment while guaranteeing high relation between a t andz t , and diverse actions for given information s t and z t .
In episodic tasks, a discounted return for(cid:101) r ( s t , a t ) is derived as
[Equation]
where H ρ ( · ) = (cid:80) Tk =
t +1 γ k − t − 1 h ρ ( · ) is a discounted sum of entropy.
By using the discounted return in Equation (3), the objective function is deﬁned as follows:
[Equation]
where H ρ ( · ) =
E π
[ H ρ ( · )] .
To maximize the objective function in Equation (4) with a simple and fast way to implement, thispaper designs (cid:15) -exploration based policy using two types of stochastic policies as follows: {+
 [Equation] [Figure] Selection+}
[Figure]
ObservationReward [Figure] Temperatures 𝒘 𝒛 𝒘 𝒔 where p m ( x ) = P ( m = x ) is a probability mass function for selecting policy π x , and (cid:15) is a positiveconstant.
By Equation (5), the objective function in Equation (4) is reformulated as follows:
[Equation]
where
[Equation]
[Equation]
Thus, the maximization problem for Equation (6) is represented as
[Equation]
[Equation]
Exploration-Bounded Exploration Actor-Critic
4.1 Problem decomposition
The maximization problem in Equation (8) is decomposed into two sub-problems: a maximizationproblem with exploration entropy and a maximization problem with bounded exploration entropyfor the policies π s and π z , respectively.
4.1.1 Learning with exploration entropy
For the policy π s , the problem in Equation (8) is reduced as
[Equation]
where J π s ( s t ) = v π s ( s t ) +
w s H π s ( a t | s t ) .
By representing J π s ( s t ) = (cid:101) v π s ( s t ) , Equation (9) can bederived as follows:
[Equation]
where (cid:101) q π s ( s t , a t ) =
r ( s t , a t ) + γ E s t +1 ∼ p
[ (cid:101) v π s ( s t +1 )] is the soft Q-function for π s [23].
3 Becausethe entropy term w s h π s ( a t | s t ) incentivizes the policy to choose actions with a high variance, thepolicy learns to be exploratory for given states.
with bounded exploration entropy
For the policy π z , the problem in Equation (8) is reduced as
[Equation]
where J π z ( s t ) =
v π z ( s t )
+ ( w z − w s )
H π z ( a t | s t , z t ) .
Without loss of generality, v π z ( s t ) can berewritten as v π z ( s t ) = v π z ( s t , z t ) because z t = f ( s t ) is a deterministic function of s z . Through thesame procedure in Section 4.1.1 with a replacement s (cid:48) t = ( s t , z t ) , J π z ( s t ) is derived as
[Equation]
[Equation]
where (cid:101) v π s ( s (cid:48) t ) and (cid:101)
q π z ( s (cid:48) t , a t ) are the soft value function and soft Q-function for π z under the givens (cid:48) t = ( s t , z t ) , respectively.
Unlike the entropy term w s h π s ( a t | s t ) in Equation (10) which incentivizes the policy to have richexploration, the entropy term ( w z − w s ) h π z ( a t | s (cid:48) t ) in Equation (12) does not guarantee that thepolicy has rich exploration.
The variance of the policy’s actions depends on whether ( w z − w s ) ispositive or not.
For example, when ( w z − w s ) is negative, the policy gets a penalty for actions with ahigh variance.
Thus, the policy learned from Equation (12) has bounded exploration.
The boundedexploration reduces an action variance such that the policy focuses on exploring around a given stateand does not explore unnecessarily widely.
In a perspective of experience data for learning, detailedexperiences from π z can supplement widely obtained experiences from π s .
4.2 Automating entropy adjustment
Because an entropy of policy is affected by not only its own distribution but also the type of taskand the magnitude of rewards, the entropy of the policy can vary during training.
For this reason,the temperatures w s and w z need to be set according to these considerations.
As shown in [25], atemperature can be adjusted by optimizing dual problem for a constrained optimization problem.
First, the objective function in Equation (4) can be represented by soft Q-function as follows:
[Equation] [Equation]
where (cid:101) q π ( s t , a t ) =
r ( s t , a t ) + γ E s t +1 ∼ p
[ (cid:101) v π ( s t +1 )] and (cid:101) v π ( s t ) =
E π [ (cid:101) G t ] are the soft Q-functionand soft value function for π , respectively.
For the objective function in Equation (13), a constrained optimization problem can be derived as
[Equation]
[Equation]
[Equation]
For the constrained optimization problem, each entropy can be the bound condition to each other,i.e., h ρ s ( a t | s t ) is the upper bound from the perspective of h ρ z ( a t | s t , z t ) and h ρ z ( a t | s t , z t ) is thelower bound from the perspective of h ρ s ( a t
| s t ) .
Because h ρ s ( a t | s t ) does not have an upper bound,the policy is able to pursue diverse actions only using the state.
Using the state and heuristic information, however, the policy is limited to get diverse actions due to the upper bound h ρ s ( a t
| s t ) .
Thus, for the case of the given state and heuristic information, the policy has to ﬁnd balanced actionsbetween the upper bound h ρ s ( a t
| s t ) and the lower bound 0.
A dual problem for the constrained optimization problem is deﬁned as
[Equation]
[Equation]
where w s and w z are the Lagrange multipliers.
The primal problem for π and the dual problem for( w s , w z ) can be solved by using gradient descent method recursively.
In practice, the soft Q-functionis modeled by function approximation based on deep RL.
Under the consideration of a parameterizedsoft Q-function for a ﬁxed policy, (cid:101) q π ( s t , a t ) does not affect to the Lagrange multipliers.
Then, theoptimal Lagrange multipliers for the given policy can be obtained by
[Equation]
[Equation]
where w = [ w s , w z ] and
[Equation]
After obtaining the optimal Lagrange multipliers, the optimization for the policy is performed forthe given Lagrange multipliers.
Therefore, π and w are updated by such a recursive optimization.
Algorithm policies: 15:
Update functions: 19:
20: end 21: end
4.3 Exploration-bounded exploration actor-critic algorithm
To solve the optimization problems in Section 4.1, actor-critic method with function approximationis adopted.
For function approximations, parameterization method is applied to soft Q-function(cid:101)
q θ m ( s t , a t ) and policy π φ m for m ∈ { s, z } in Equations (10) and (12).
As shown in [25], theparameterized soft Q-function (cid:101) q θ m ( s t , a t ) is trained to minimize the soft Bellman residual as
J θ m = E ( s t ,a t ) ∼D (cid:20)2 (cid:8)(cid:101)
q θ m ( s t , a t ) − (cid:0) r
( s t , a t ) + γ E s t +1 ∼ p [ (cid:101) v ¯ θ m ( s t +1 )] (cid:1)(cid:9)(cid:21) , (20) where D is a replay buffer, (cid:101) q θ m ( s t , a t ) is (cid:101) q θ s ( s t , a t ) for m = s and (cid:101) q θ z ( s (cid:48) t , a t ) for m =
z ,(cid:101) v ¯ θ m ( s t +1 ) = (cid:26) (cid:101) q ¯ θ s ( s t +1 , a t +1 ) + w s h π φs ( a t | s t ) , for m =
s (cid:101) q ¯ θ z ( s (cid:48) t +1 , a t +1 ) + ( w z − w s )
h π φz ( a t | s (cid:48) t ) , for m =
z , and ¯ θ m is the parameters of a target soft Q-function obtained as an exponentially moving average ofthe soft Q-function parameters.
After obtaining the parameterized soft Q-functions, the parameterized policies and temperatures are updated sequentially.
These updates are summarized in Algorithm 1.
The soft Q-functions and policies are updated in thesame manner with SAC [25].
As SAC uses two soft Q-functions to mitigate positive bias in a policyimprovement, we also use two soft Q-functions for each policy.
For the simplicity of Algorithm 1,we omit a detailed description of the two soft Q-functions for each policy.
Experiments
In order to validate our method, we implement a robotic manipulation task, Pick-and-Lift , modiﬁedfrom that of RLBench [26], where the task environment consists of a single UR3 arm with 6 DoFincluding the two-ﬁnger gripper, a cube object on a table, and a target position.
We design thereward of Pick-and-Lift as the sum of dense and sparse rewards, in detail the rewards for reachingand moving sub-tasks are given by dense terms of distances among the robot’s gripper, the cubeand the target position, and the rewards for grasping is given by sparse terms of the binary signalaccording to whether or not to succeed in grasping.
[Figure]
To demonstrate our method, we evaluate EBE-AC compared to SAC [25] and DIAYN [21], state-ofthe-art methods in the ﬁeld of stochastic policy based on deep RL.
In addition, we evaluate EBE-ACwith DIAYN to show effects of our method combining with latent information used in DIAYN.
5.1 Policy evaluation
We evaluate EBE-AC( (cid:15) ) and EBE-AC( (cid:15) ) combined with DIAYN, where (cid:15) is the probability deﬁnedin Equation (5).
5 Figure 2 represents training curves for episode rewards and success rates.
In Figure2a, a range of the episode reward in y-axis is roughly divided into three cases as follows:
• Range 1 .
(episode reward < 0): The robot tries to reach the cube.
• Range 2 .
(0 < episode reward < 100): The robot tries to grasp the cube. • Range 3 .
(100 < episode reward): The robot tries to move the cube to the target position.
Compared with SAC and DIAYN, EBE-AC methods reach Range 2 faster than the other methods.
SAC can not reach Range 3 and it means that SAC are stuck in local optimums Range 2 during training.
EBE -AC(0.1) and EBE-AC(0.2) with DIAYN methods reach Range 3 faster than DIAYN.
Especially, EBE-AC(0.2) with DIAYN is noticeably faster than DIAYN.
In Figure 2b which represents the success rate, it is also observed that only the methods reaching
Range 3 have meaningful success rates.
EBE-AC(0.2) with DIAYN outperforms the other methodsand EBE-AC(0.1) has better success rates than DIAYN.
Since EBE-AC is composed of the policywith exploration entropy and the policy with bounded exploration entropy, it can be seen as thecombination of SAC’s policy and the proposed policy with bounded exploration entropy, where thepolicies are linked by the temperature optimization.
Through the composition characteristics of
EBE-AC and experimental results for the fast convergences, therefore, it is shown that our methodis able to help the prior works in terms of the sample efﬁciency.
Figure 3 represents an analysis for temperatures according to the number of episodes, 7 where thetemperatures of SAC and DIAYN are optimized in the same manner with SAC [25].
As shown in Figure 3, temperatures of all the methods decrease rapidly at the beginning of learning. The degradation of temperatures affects to reducing the proportion of entropy in a soft value function Appendix D.5.
Appendix D. 3b [Figure] which is a sum of soft Q-function and entropy.
It means that the learning takes into account expectedreturns approximated by the soft Q-function more importantly. The value of w z is always higher than that of w s and it leads to the value ( w z − w s ) to be positive.
Thus, the entropy term in Equation (11) affects π z to maintain exploratory. In addition, it is observedthat w s in Equation (10) is higher than ( w z − w s ) .
It means that π s is affected from the entropyand pursues more exploration than π z .
The temperatures of EBE-AC(0.1) increase rapidly at about3000 and 8000 in x-axis, which is consistent with the increase of episode reward in Figure 2a.
Thisfeature is also observed for DIAYN at about 5000 and 10000 in x-axis.
Therefore, EBE-AC(0.1)considers weighted entropies more than SAC and DIAYN, and is more exploratory than SAC and
DIAYN.
For this reason, EBE-AC(0.1) has better sample efﬁciency than SAC and DIAYN.
Conclusion
In this paper, we proposed exploration-bounded exploration actor-critic (EBE-AC), a novel deep RLalgorithm that can optimize a stochastic policy through the use of heuristic information.
The heuristic information includes interpretable human knowledge related to physical relationships between arobot and objects, which can be reused in various learning tasks.
In a robotic manipulation task, wedemonstrated that the proposed algorithm outperformed state-of-the-art off-policy actor-critic deep RL algorithms, such as SAC and DIAYN, in terms of sample efﬁciency.
In addition, we extendedthe algorithm by means of latent information as DIAYN such that the extended algorithm can furtherimprove sample efﬁciency and robustness.