Stochastic Policy Optimization with Heuristic Information for Robot Learning
Abstract: Stochastic policy-based deep reinforcement learning (RL) approaches have remarkably succeeded to deal with continuous control tasks.
However, applying these methods to manipulation tasks remains a challenge since actuators of a robot manipulator require high dimensional continuous action spaces.
In this paper, we propose exploration-bounded exploration actor-critic (EBE-AC), a novel deep RL approach to combine stochastic policy optimization with interpretable human knowledge.
The human knowledge is deﬁned as heuristic information based on both physical relationships between a robot and objects and binary signals of whether the robot has achieved certain states.
The proposed approach, EBE-AC, combines an off-policy actor-critic algorithm with an entropy maximization based on the heuristic information.
On a robotic manipulation task, we demonstrate that EBE-AC outperforms prior state-of-the-art off-policy actorcritic deep RL algorithms in terms of sample efﬁciency.
In addition, we found that
EBE-AC can be easily combined with latent information, where EBE-AC with latent information further improved sample efﬁciency and robustness.
Introduction
A challenge to human-level AI [1] is still ongoing research area and recently deep reinforcement learning (RL) approaches have remarkably shown outstanding results beyond human in the ﬁelds of board and video games [2, 3].
As a greatly unexplored area which adapts deep RL, robot manipulation is one of the challenging applications due to complex nonlinear dynamics and huge continuous action spaces [4]-[7].
Various and stable robot manipulation skills in real-world environments [8], where the laws of physics are strictly applied, are required to solve complex tasks, such as moving, grasping, pushing and placing.
It is, therefore, still a challenge to solve complex tasks through deep
RL approaches.
In front of the advanced dexterity capabilities of human hands, learning the dexterity with functionality to autonomously pick and place objects has always been a long-standing challenge for robotics [9, 10].
To solve complex manipulation tasks, various approaches have been presented to utilize human knowledge, which includes learning from human demonstrations and optimizing hyper-parameters empirically [11]-[18].
The ﬁrst improves sample efﬁciency of learning by providing good expert demonstration data to solve a given task [13]-[16].
In general, the human demonstrations can be used not only as a loss function of inverse RL but also as reusable experience data of off-policy RL.
However, it is expensive to obtain real-world demonstrations for every single task.
As the second approach utilizing human knowledge, empirical hyper-parameter tuning [19], has helped optimize learning progress since hyper-parameters are highly related to not only neural network architectures but also reward designs in the deep RL.
On the whole, hyper-parameter tuning is widely used to ensure that learning algorithms gain good performance, however, it is vulnerable to initial hyper-parameter settings for each task [18].
This results in the inefﬁciency of the learning process because it is necessary to perform hyper-parameter tuning every task.
A heuristic (informed) strategy using problem-speciﬁc knowledge has been generally known as a more efﬁcient solution than an uninformed strategy using only the problem itself [20].
Since manipulation skills are under the laws of physics, informed human knowledge, such as moving in the direction of narrowing the distance, being in alignment with the target object, and grasping the center of gravity of the object, can be helpful to ﬁnd solutions efﬁciently in robot manipulation.
In this paper, we propose a novel deep RL approach to combine stochastic policy optimization with human knowledge, which is exploration-bounded exploration actor-critic (EBE-AC).
EBE-AC applies interpretable and reusable human knowledge, heuristic information based on both physical relationships between a robot and objects and binary signals of whether the robot has achieved certain states, to an off-policy actor-critic algorithm.
The main contributions of this paper are summarized as follows:
• We design a novel information theoretic metric in order for EBE-AC to have dependencies between actions and heuristic information.
• We propose an (cid:15) -exploration method using two types of stochastic policies, exploration and bounded exploration, in order to store diverse experiences in a shared replay buffer.
• We propose a temperature optimization for given entropies corresponding to the two policies, in order to balance between the two.
To verify the performance of EBE-AC, we conducted experiments on a robotic manipulation task,
Pick-and-Lift .
The experimental results show that EBE-AC outperforms prior state-of-the-art offpolicy actor-critic deep RL algorithms in terms of sample efﬁciency.
In addition, we found that
EBE-AC can be easily combined with latent information as DIAYN [21] does, where EBE-AC with latent information further improved sample efﬁciency and robustness.
Related Work
Due to the difﬁculty of obtaining stable and steady improvement of deterministic policy in applying complex and high dimensional tasks, stochastic policy optimization based on information theory has been examined in problems with continuous state-action spaces [22]-[24].
Many works show that applying a stochastic policy’s entropy to an objective function results in good performances.
In particular, regularizing both policy and value functions with entropy shows the convergence and stability improvements during training processes.
Through maximum entropy RL [23], a policy is encouraged to perform diverse actions to widely explore a state space while maximizing expected returns.
Owing to the diverse actions being able to maximize expected returns, the learned policy can have multiple optimal solutions and obtain stable performance regardless of random initial states.
In addition, the maximum entropy approaches are also robust to environmental disturbances such as the presence of adversaries and dynamics change over time because these works focus on the entropy of the action distribution for a given state.
In the unsupervised emergence of diverse skills dictating the states [21], the mutual information between states and latent variables is leveraged to autonomously discover diverse skills.
To ensure the different skills being distinguishable, the mutual information between actions and skills should be minimized for a given state.
By utilizing the mutual information as a metric of the objective function, a stochastic policy can be also stably and steadily improved under the consideration of additional latent variables.
These prior works utilize rich exploration to obtain various experiences at the beginning of learning, and tend to reduce the exploration as the learning progresses in order to acquire good expected returns stably.
However, when a policy is stuck in inevitable local optimums which must be passed to complete a given complex task, the policy needs to utilize rich exploration in order to escape the local optimums even in the middle of learning.
In other words, it is necessary that a policy simultaneously pursues rich exploration and stable expected returns throughout the learning period.
This paper proposes a novel way to combine human knowledge with a stochastic policy by leveraging the information theoretic metrics.
Unlike reducing the mutual information between actions and latent variables in [21], the proposed method increases the mutual information between actions and heuristic information based on human knowledge.
As a result, the dependency between actions and heuristic information will be guaranteed and experiences obtained by actions are related to heuristic information.
Based on the metric, (cid:15) -exploration method using two types of stochastic policies is proposed in order to maintain rich exploration throughout the learning period.
Based on the proposed method, the sample efﬁciency is also improved and the stochastic policy achieves the robustness to complex tasks.
Preliminaries 3.1 Notation
A Markov decision process is deﬁned by the tuple ( S , A , p, r ) , where S is a state space, A is a continuous action space, p : S ×
A × S → R + is a transition probability distribution, and r :
S ×
A → R is a reward function.
In episodic tasks, a discounted return at time t is given as
G t = (cid:80) Tk =
t +1 γ k − t − 1 r ( s k , a k ) , where T is an episode horizon, γ is a discount factor, and r ( s k , a k ) is a reward for the state-action pair ( s k , a k ) .
A value function for a policy π ( a t | s t ) is represented as v π ( s t ) =
E π
[ G t ] .
3.2 Objective function
We consider combining RL with the information theoretic metrics, which can encourage policy optimization in both ways to diverse actions and speciﬁc actions in a given state.
Therefore, we propose an objective function to learn a policy conditioned on heuristic information according to human knowledge, where the policy can select actions considering not only the states but also the heuristic information.
First, we deﬁne an information theoretic reward as
[Equation]
where z t is heuristic information, I ( a t ; z t | s t ) is a mutual information between a t and z t for the given s t , h ρ ( · ) is an entropy for the distribution ρ , w s and w z are positive temperatures, and ρ s and ρ z are distributions of action a t conditioned on s t and ( s t , z t ) , respectively.
The distributions ρ s and ρ z are induced by a policy π .
The heuristic information z t is deﬁned by external function z t = f
( s t ) based on s t , where f ( · ) can be physical information between a robot and objects in addition to binary signals representing speciﬁc states.
Maximizing the reward (cid:101) r
( s t , a t ) in Equation (1) means that a policy has to ﬁnd an optimal solution to maximize the reward from the environment while guaranteeing high relation between a t and z t , and diverse actions for given information s t and z t .
In episodic tasks, a discounted return for (cid:101) r ( s t , a t ) is derived as
[Equation]
[Equation]
) where H ρ ( · ) = (cid:80) Tk =
t +1 γ k − t − 1 h ρ ( · ) is a discounted sum of entropy.
By using the discounted return in Equation (3), the objective function is deﬁned as follows:
[Equation]
where H ρ ( · ) =
E π
[ H ρ ( · )] .
To maximize the objective function in Equation (4) with a simple and fast way to implement, this paper designs (cid:15) -exploration based policy using two types of stochastic policies as follows:[-
-]
[Equation]
In comparison with an (cid:15) -greedy method which uses policy-based actions and random actions with (cid:15) prob- ability for exploration effects, the (cid:15) -exploration method uses two types of policies having different exploration characteristics measured in entropies where one learns to maximize a return with rich explorations and the other learns to maximize a return with bounded explorations.
[Figure]
ObservationReward
[Figure]
Temperatures 𝒘 𝒛 𝒘 𝒔 where p m ( x ) =
P ( m = x ) is a probability mass function for selecting policy π x , and (cid:15) is a positive constant.
By Equation (5), the objective function in Equation (4) is reformulated as follows:
[Equation]
(6) where H
[Equation]
=
[Equation]
for [Equation] Thus, the maximization problem for Equation (6) is represented as
[Equation]
Exploration-Bounded Exploration Actor-Critic 4.1 Problem decomposition
The maximization problem in Equation (8) is decomposed into two sub-problems: a maximization problem with exploration entropy and a maximization problem with bounded exploration entropy for the policies π s and π z , respectively.
4.1.1 Learning with exploration entropy
For the policy π s , the problem in Equation (8) is reduced as
[Equation]
where J π s ( s t ) = v π s ( s t ) +
w s H π s ( a t | s t ) .
By representing J π s ( s t ) = (cid:101) v π s ( s t ) , Equation (9) can be derived as follows:
[Equation]
where (cid:101) q π s ( s t , a t ) =
r ( s t , a t ) + γ E s t +1 ∼ p
[ (cid:101) v π s ( s t +1 )] is the soft Q-function for π s [23].
Because the entropy term w s h π s ( a t | s t ) incentivizes the policy to choose actions with a high variance, the policy learns to be exploratory for given states.
4.1.2 Learning with
bounded
exploration entropy
For the policy π z , the problem in Equation (8) is reduced as
[Equation]
where J π z ( s t ) =
v π z ( s t )
+ ( w z − w s )
H π z ( a t | s t , z t ) .
Without loss of generality, v π z ( s t ) can be rewritten as v π z ( s t ) =
v π z ( s t , z t ) because z t = f
( s t ) is a deterministic function of s z .
Through the same procedure in Section 4.1.1 with a replacement s (cid:48) t = ( s t , z t ) , J π z ( s t ) is derived as
[Equation]
where (cid:101) v π s ( s (cid:48) t ) and (cid:101)
q π z ( s (cid:48) t , a t ) are the soft value function and soft Q-function for π z under the given s (cid:48) t
= ( s t , z t ) , respectively.
Unlike the entropy term w s h π s ( a t | s t ) in Equation (10) which incentivizes the policy to have rich exploration, the entropy term ( w z − w s ) h π z ( a t | s (cid:48) t ) in Equation (12) does not guarantee that the policy has rich exploration.
The variance of the policy’s actions depends on whether ( w z − w s ) is positive or not.
For example, when ( w z − w s ) is negative, the policy gets a penalty for actions with a high variance.
Thus, the policy learned from Equation (12) has bounded exploration.
The bounded exploration reduces an action variance such that the policy focuses on exploring around a given state and does not explore unnecessarily widely.
In a perspective of experience data for learning, detailed experiences from π z can supplement widely obtained experiences from π s .
4.2 Automating entropy adjustment
Because an entropy of policy is affected by not only its own distribution but also the type of task and the magnitude of rewards, the entropy of the policy can vary during training.
For this reason, the temperatures w s and w z need to be set according to these considerations.
As shown in [25], a temperature can be adjusted by optimizing dual problem for a constrained optimization problem.
First, the objective function in Equation (4) can be represented by soft Q-function as follows:
[Equation]
where (cid:101) q π ( s t , a t ) =
r ( s t , a t ) + γ E s t +1 ∼ p
[ (cid:101) v π ( s t +1 )] and (cid:101) v π ( s t ) =
E π [ (cid:101) G t ] are the soft Q-function and soft value function for π , respectively.
For the objective function in Equation (13), a constrained optimization problem can be derived as
[Equation]
For the constrained optimization problem, each entropy can be the bound condition to each other, i.e., h ρ s ( a t | s t ) is the upper bound from the perspective of h ρ z ( a t | s t , z t ) and h ρ z ( a t | s t , z t ) is the lower bound from the perspective of h ρ s ( a t
| s t ) .
Because h ρ s ( a t | s t ) does not have an upper bound, the policy is able to pursue diverse actions only using the state.
Using the state and heuristic information, however, the policy is limited to get diverse actions due to the upper bound h ρ s ( a t
| s t ) .
Thus, for the case of the given state and heuristic information, the policy has to ﬁnd balanced actions between the upper bound h ρ s ( a t
| s t ) and the lower bound 0.
A dual problem for the constrained optimization problem is deﬁned as
[Equation]
[Equation]
where w s and w z are the Lagrange multipliers.
The primal problem for π and the dual problem for ( w s , w z ) can be solved by using gradient descent method recursively.
In practice, the soft Q-function is modeled by function approximation based on deep RL.
Under the consideration of a parameterized soft Q-function for a ﬁxed policy, (cid:101) q π ( s t , a t ) does not affect to the Lagrange multipliers.
Then, the optimal Lagrange multipliers for the given policy can be obtained by
[Equation]
(18) where w = [ w s , w z ] and
[Equation]
After obtaining the optimal Lagrange multipliers, the optimization for the policy is performed for the given Lagrange multipliers.
Therefore, π and w are updated by such a recursive optimization.
Algorithm a Store a D D a s
10: end do 12: Update : 13: θ Update policies: 15: φ
← Update temperatures: 17: Update target soft Q functions: 19: ¯ θ m ← τθ + (1 − τ ) θ m m
{ s 20: end for 21: end for
4.3 Exploration-bounded exploration actor-critic algorithm
To solve the optimization problems in Section 4.1, actor-critic method with function approximation is adopted.
For function approximations, parameterization method is applied to soft Q-function (cid:101) q θ m ( s t , a t ) and policy π φ m for m ∈ { s, z } in Equations (10) and (12).
As shown in [25], the parameterized soft Q-function (cid:101) q θ m ( s t , a t ) is trained to minimize the soft Bellman residual as
J θ m = E ( s t ,a t ) ∼D (cid:20)2 (cid:8)(cid:101)
q θ m ( s [Equation] , a t ) − (cid:0) r ( s t , a t ) + γ E s t +1 ∼ p
[ (cid:101) v ¯ θ m ( s t +1 )] (cid:1)(cid:9)(cid:21) , (20) where D is a replay buffer, (cid:101)
q θ m ( s t , a t ) is (cid:101)
q θ s ( s t , a t ) for m = s and (cid:101) q θ z ( s (cid:48) t , a t ) for m =
z , (cid:101) v ¯ θ m ( s t +1 )
= (cid:26) (cid:101) q ¯ θ s ( s t +1 , a t +1 )
+ w s h π φs ( a t | s t +1 ) , for m =
s (cid:101) q ¯ θ z
( s (cid:48)
t +1 , a t +1 )
+ ( w z − w s )
h π φz ( a t | s (cid:48) t +1 ) , for m =
z , and ¯ θ m is the parameters of a target soft Q-function obtained as an exponentially moving average of the soft Q-function parameters.
After obtaining the parameterized soft Q-functions, the parameterized policies and temperatures are updated sequentially.
These updates are summarized in Algorithm 1.
The soft Q-functions and policies are updated in the same manner with SAC [25].
As SAC uses two soft Q-functions to mitigate positive bias in a policy improvement, we also use two soft Q-functions for each policy.
For the simplicity of Algorithm 1, we omit a detailed description of the two soft Q-functions for each policy.
Experiments
In order to validate our method, we implement a robotic manipulation task, Pick-and-Lift , modiﬁed from that of RLBench [26], where the task environment consists of a single UR3 arm with 6 DoF including the two-ﬁnger gripper, a cube object on a table, and a target position.
We design the reward of Pick-and-Lift as the sum of dense and sparse rewards, in detail the rewards for reaching and moving sub-tasks are given by dense terms of distances among the robot’s gripper, the cube and the target position, and the rewards for grasping is given by sparse terms of the binary signal according to whether or not to succeed in grasping
[Figure]
To demonstrate our method, we evaluate EBE-AC compared to SAC [25] and DIAYN [21], state-ofthe-art methods in the ﬁeld of stochastic policy based on deep RL.
In addition, we evaluate EBE-AC with DIAYN to show effects of our method combining with latent information used in DIAYN.
5.1 Policy evaluation
We evaluate EBE-AC( (cid:15) ) and EBE-AC( (cid:15) ) combined with DIAYN, where (cid:15) is the probability deﬁned in Equation (5).
6 Figure 2 represents training curves for episode rewards and success rates.
In Figure 2a, a range of the episode reward in y-axis is roughly divided into three cases as follows:
• Range 1 .
(episode reward < 0):
The robot tries to reach the cube.
• Range 2 .
(0 < episode reward < 100): The robot tries to grasp the cube.
• Range 3 .
(100 < episode reward):
The robot tries to move the cube to the target position.
Compared with SAC and DIAYN, EBE-AC methods reach Range 2 faster than the other methods.
SAC can not reach Range 3 and it means that SAC are stuck in local optimums Range 2 during training.
EBE-AC(0.1) and EBE-AC(0.2) with DIAYN methods reach Range 3 faster than DIAYN.
Especially, EBE-AC(0.2) with DIAYN is noticeably faster than DIAYN.
In Figure 2b which represents the success rate, it is also observed that only the methods reaching
Range 3 have meaningful success rates.
EBE-AC(0.2) with DIAYN outperforms the other methods and EBE-AC(0.1) has better success rates than DIAYN.
Since EBE-AC is composed of the policy with exploration entropy and the policy with bounded exploration entropy, it can be seen as the combination of SAC’s policy and the proposed policy with bounded exploration entropy, where the policies are linked by the temperature optimization.
Through the composition characteristics of
EBE-AC and experimental results for the fast convergences, therefore, it is shown that our method is able to help the prior works in terms of the sample efﬁciency.
5.2 Temperature Comparison
Figure 3 represents an analysis for temperatures according to the number of episodes, 8 where the temperatures of SAC and DIAYN are optimized in the same manner with SAC [25].
As shown in Figure 3, temperatures of all the methods decrease rapidly at the beginning of learning.
The degradation of temperatures affects to reducing the proportion of entropy in a soft value function
[Figure]
which is a sum of soft Q-function and entropy.
It means that the learning takes into account expected returns approximated by the soft Q-function more importantly.
The value of w z is always higher than that of w s and it leads to the value ( w z − w s ) to be positive.
Thus, the entropy term in Equation (11) affects π z to maintain exploratory.
In addition, it is observed that w s in Equation (10) is higher than ( w z − w s ) .
It means that π s is affected from the entropy and pursues more exploration than π z .
The temperatures of EBE-AC(0.1) increase rapidly at about 3000 and 8000 in x-axis, which is consistent with the increase of episode reward in Figure 2a.
This feature is also observed for DIAYN at about 5000 and 10000 in x-axis.
Therefore, EBE-AC(0.1) considers weighted entropies more than SAC and DIAYN, and is more exploratory than SAC and
DIAYN.
For this reason, EBE-AC(0.1) has better sample efﬁciency than SAC and DIAYN.
Conclusion
In this paper, we proposed exploration-bounded exploration actor-critic (EBE-AC), a novel deep RL algorithm that can optimize a stochastic policy through the use of heuristic information.
The heuristic information includes interpretable human knowledge related to physical relationships between a robot and objects, which can be reused in various learning tasks.
In a robotic manipulation task, we demonstrated that the proposed algorithm outperformed state-of-the-art off-policy actor-critic deep
RL algorithms, such as SAC and DIAYN, in terms of sample efﬁciency.
In addition, we extended the algorithm by means of latent information as DIAYN such that the extended algorithm can further improve sample efﬁciency and robustness. Acknowledgments
This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government.
[21ZR1100, A Study of Hyper-Connected Thinking Internet
Technology by autonomous connecting, controlling, and evolving ways]