E XPLORE WITH D YNAMIC M AP : G RAPH S TRUCTURED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In reinforcement learning, a map with states and transitions built based on historical trajectories is often helpful in exploration and exploitation.
Even so, learning and planning on such a map within a sparse environment remains a challenge.
As a step towards this goal, we propose Graph Structured Reinforcement Learning (GSRL), which leverages graph structure in historical trajectories to slowly adjust exploration directions and rapidly update value function estimation with related experiences.
GSRL constructs a dynamic graph on top of state transitions in the replay buffer based on historical trajectories and develops an attention strategy on the map to select an appropriate goal direction, which decomposes the task of reaching a distant goal state into a sequence of easier tasks.
We also leverage graph structure to sample related trajectories for efﬁcient value learning.
Results demonstrate that GSRL can outperform the state-of-the-art algorithms in terms of sample efﬁciency on benchmarks with sparse reward functions.
I NTRODUCTION
How can humans learn to solve tasks with complex structure and delayed and sparse feedback?
Take a complicated navigation task as an example in which the goal is to go from a start state to an end state.
A straightforward approach that one often takes is by decomposing this complicated task into a sequence of easier ones, identifying/forming some intermediate goals that help to get to the ﬁnal goal, and ﬁnally choosing a route with the highest return among a poll of candidate routes that lead to the ﬁnal goal.
Despite recent success and progress in reinforcement learning (RL) approaches (Mnih et al., 2013; Fakoor et al., 2020;
Hansen et al., 2018; Huang et al., 2019; Silver et al., 2016), there is still a huge gap between how humans and RL agents learn.
In many real-world applications, an RL agent only has access to sparse and delayed rewards, which by itself leads to two major challenges.
(C1)
The ﬁrst one is how can an agent effectively learn from sparse and delayed rewards?
One possible solution is to build a new reward function, known as an intrinsic reward, that helps expedite the learning process and ultimately solve a given task.
Although this may seem like an appealing solution, it is often not obvious how to formulate an effective intrinsic reward.
Recent works have introduced goal-oriented RL (Schaul et al., 2015;
Chiang et al., 2019) as a way of constructing intrinsic rewards.
However, most goal-oriented RL algorithms require large amounts of reward shaping (Chiang et al., 2019) or human demonstrations (Nair et al., 2018).
(C2)
The second issue is how to retrieve relevant past experiences that help to learn faster and improve sample efﬁciency.
Recent attention has focused much on episodic RL (Blundell et al., 2016), which builds a non-parametric episodic memory to store past experiences, and thus can rapidly latch onto related ones through search with similarity.
However, most episodic
RL algorithms require additional memory space (Pritzel et al., 2017).
[Figure]
In this work, we propose Graph Structured
Reinforcement Learning (GSRL), which constructs a state-transition graph, leverages structural graph information, and presents solutions for the problems raised above.
As shown in Figure 1, when we encounter a complex task, we draw a map for planning with a long horizon (Eysenbach et al., 2019) and scan through the graph to seek related experiences within a short horizon (Lee et al., 2019).
[Figure]
Inspired by this, we propose a method to address the ﬁrst problem (C1) by learning to generate an appropriate goal g e based on map G e 0 at episode e and form the intrinsic rewards during goal generation.
This goal generation can be optimized in hindsight.
That is, we can update the goal generation model under the supervision generated by planning algorithms based on map G e + at episode e + 1 .
We can solve the second problem (C2) by updating the value estimation of the current state with the trajectories that include neighborhood states.
Note that GSRL draws a map to help agents plan an appropriate direction in the long horizon and learn an optimal strategy in the short horizon, which can be widely incorporated with various learning strategies such as DQN (Mnih et al., 2013), DDPG (Lillicrap et al., 2015), etc.
However, arriving at a well-explored map in terms of state-transitions in the ﬁrst place from scratch remains a challenging problem.
In this regard, GSRL should be able to jointly optimize map construction, goal selection, and value estimation.
As illustrated in Figure 2, we design GSRL mainly following four steps: (a) At the beginning, an agent is located at the initial state and aims at the terminal state with an empty map; (b) The agent then explores the environment with the current policy and records those explored states on the map; (c) Considering the far distance to the terminal state and sparse reward during the exploration, we decompose the whole task into a sequence of easier tasks guided by goals; (d) We leverage structured map information to select related trajectories to update the current policy; The agent can repeat procedures (b)-(d) and ﬁnally reach the terminal state with a well-explored map.
Our primary contribution is a novel algorithm that constructs a dynamic state-transition graph from scratch and leverages structured information to ﬁnd an appropriate goal in goal selection with a long horizon and sample related trajectories in value estimation with a short horizon.
This framework provides several beneﬁts: (1) How to balance exploration and exploitation is a fundamental problem in any reinforcement learning (Andrychowicz et al., 2017; Nachum et al., 2018).
GSRL learns an attention mechanism on the graph to decide whether to explore the states with high uncertainty or exploit the states with well estimated value.
(2) Almost all the existing reinforcement learning approaches (Paul et al., 2019; Eysenbach et al., 2019) utilize randomly sampled trajectories to update the current policy leading to inefﬁciency.
GSRL selects highly related trajectories for updates through the local structure in the graph.
(3) Unlike previous approaches (Kipf et al., 2020; Shang et al., 2019), GSRL constructs a dynamic graph through agent exploration without any additional object representation techniques and thus is not limited to several speciﬁc environments.
Empirically, we ﬁnd that our method constructs a useful map and learns an efﬁcient policy for highly structured tasks.
Comparisons with state-of-the-art RL methods show that GSRL is substantially more successful for boosting the performance of a graph-structured RL algorithm.
P RELIMINARIES
Let M =
( S, A, T, P, R ) be an Markov Decision Process (MDP) where S is the state space, A is the action space whose size is bounded by a constant, T ∈ Z
+ is the episode length, P : S ×
A → ∆( S ) is the transition function which takes a state-action pair and returns a distribution over states, and
R : S ×
A → ∆( R ) is the reward distribution.
The rewards from the environment, called extrinsic, are usually sparse and, therefore, difﬁcult to learn.
To address this, goal-oriented RL introduces goal g ∈ G to build dense intrinsic rewards
R g : S ×
A × G → ∆( R ) .
In order to make the learning procedure stable, the goal is limited to be updated episodically.
A policy π : S × G → ∆( A ) prescribes a distribution over actions for each state and goal.
At each timestep, the agent samples an action a ∼ π ( s, g ) and receives a corresponding reward r g ( s, a ) that indicates whether or not the agent has reached the goal.
The episode terminates only after T timesteps even when the agent reaches the goal.
The agent’s task is to maximize its cumulative discounted future reward.
We use an off-policy algorithm to learn such a policy, as well as its associated goal-oriented Q -function and value function:
[Equation] [Equation]
We use off-policy algorithms named DQN (Mnih et al., 2013) for discrete action space and DDPG (Lillicrap et al., 2015) for continuous action space to learn Q -functions and policies by utilizing off-policy data (i.e., data stored in the replay buffer).
G RAPH S TRUCTURED R EINFORCEMENT L EARNING
As illustrated in Figure 1, GSRL constructs a dynamic graph on top of state transitions.
However, learning and planning algorithms on the state transition graph are non-trivial.
In this section, we provide a theoretical analysis to show that exploration without any constraint would lead to the explosion of the graph.
In order to guide the exploration with directions, we develop a novel goaloriented RL framework which incorporates the structure information of the state-transition graph.
Speciﬁcally, we ﬁrst divide states into several groups according to their uncertainty and locations in the graph. We then adopt an attention mechanism to select an appropriate group and assign the state with the highest value in the graph as a goal to encourage further exploration.
We also propose to update goal generation hindsightly and value estimation with related trajectories, to help RL agents learn efﬁciently.
3. E XPLORE WITH D YNAMIC G RAPH
We directly build a dynamic and directed graph using sample trajectories from the replay buffer.
We denote the state-transition graph at timestep t in episode e by G et = <
V et , E et > with relations
R et , the node space V et represents state space S G et , edge space E et represents action space
A G et , and relations
R et represents transition P G et .
We denote the graph’s boundary by ∂ G et , which consists of observed but not fully explored states.
An illustrated example of these notations above is available at Figure 7 in Appendix A.
Let an agent start from an initial state s 0 , G et grow from a single node (i.e., G
= { s 0 } ) and expand itself at each timestep, leading to a sequence of {{G 1 t } T − 1 t =0 , {G 2 t } T − 1 t =0 , . . .
, {G Et } T − 1 t =0 } where T denotes episode length and E is the number of episodes.
Given that state-transition graph G et is always a connected directed graph, we describe the expansion behavior as consecutive expansion, which means no jumping across neighborhood allowed.
We denote graph increment at timestep t in episode e as ∆ G et , and then we can ensure that G et ∪ ∆ G et
=
G et +1 ⊆ G et ∪ ∂ G et .
Proposition 1.
Given the whole state-transition graph G whole , we assume that the probability of degree of an arbitrary state being less than or equal to d is larger than p (i.e., P ( deg ( s ) ≤ d ) > p, ∀ s ∈ S G ). Considering a sequence of consecutively expanding sub-graphs {{
[Equation]
[Equation] [Equation] [Equation]
[Equation] , {G [Equation]
[Equation] } , starting with G [-
-]01 = { s 0 } , for all t ≥ 0 , e ≥ 1 , we can ensure that P (cid:0) | S G et | ≤ (cid:15) (cid:1) > p (cid:15) , where (cid:15) = d · ( d − 1) T · e + t − 2 d −when d > 2 and (cid:15) = 1 , 3 when d = 1 ,respectively.
Proof. The proof of Proposition 1 can be found in Appendix D.1.
The proposition implies the guarantee of upper-bounding | S G et | becomes looser and weaker as t, e get larger even if the given assumption has a small d and a large p (close to 1 ).
In other words, randomly appending G et through exploration would enhance computation complexity for learning and plan on the dynamic graph.
In order to prevent G et from an explosion, we need to constrain ∆ G et by guiding exploration with goals.
In this paper, we propose to leverage the graph structure to assist agent exploration.
Before the detailed introduction of exploration strategy, we ﬁrstly introduce the deﬁnition of certainty as
Deﬁnition 1.
Given a state s , we deﬁne its certainty as to the number of candidate actions that have been already taken.
In this paper, we build the state-transition graph G on the replay buffer and thus can adopt the out degree ∗ of each state to approximately measure the certainty.
In other words, we can approximate the certainty of state cert ( s ) ≈ deg ( s ) , where deg ( s ) denotes the degree of node s .
Note that certainty of state in Deﬁnition 1 can be served as a local measurement to show the extent of exploration on some states, which is actually proportioned to the global measurement (i.e., the number of visited states) in a deterministic environment (See Appendix E.1 for details).
In addition, we also can deﬁne uncertainty of state as the number of untaken candidate actions.
Exploration Strategy.
A simple but effective way is to do exploration based on goal-oriented
RL, which decomposes the whole task into a sequential of goal-oriented tasks.
In this case, newly explored states in each episode are guided by a speciﬁc goal.
Therefore, we constrain the statetransition graph from the explosion and balance the exploration and exploitation through assigning an appropriate goal to the agent.
In order to further discuss the deﬁnition of an appropriate goal, we here provide the deﬁnition of the optimal goal.
As shown in Figure 4, once we are accessible to the whole state-transition graph G whole , we can obtain the optimal solution by short path planning algorithm such as Dijkstra’s Algorithm.
We deﬁne the optimal goal as the terminal state on the path.
Deﬁnition 2.
For fully explored state-transition graph G full † , we deﬁne the optimal goal g ∗ full as the terminal state (e.g., s 10 in Figure 4(a)).
For any not fully explored state-transition graph G e 0 , we deﬁne the optimal goal g ∗ e hindsightly, where we generate the optimal solution path P
e +1 based on the shortest path planning on the next episode graph G e + and regard the reachable terminal state included both in G e 0 and P e +1 as g ∗ e (e.g., s 6 in Figure 4(c)).
One should be noted that the optimal goal is hindsight generated and keep approaching the terminal state during exploration.
We further analyze and discuss the relations between goals in this paper and previous goal-oriented RL literature in Appendix E.2.
Proposition 2.
Assume that Q -value of each state is well estimated (i.e., Q = Q ∗ ), then optimal goal g ∗ e at the beginning timestep of any episode e is always included in the boundary of the statetransition graph G e 0 (i.e., ∂ G e 0 ).
Proof.
The proof of Proposition 2 can be found in Appendix D.2. [Figure]
The proposition implies that the candidate set for each goal generation can be limited to the boundary.
However, learning to generate an appropriate goal is non-trivial with respect to a dynamic and large-scale state-transition graph G et .
One intuitive solution is to divide the whole candidate goal space (i.e., ∂ G e t ) into several candidate groups C 1 , . . .
, C N , where N is the number of groups.
In order to both cover all the states and control exploration size in these groups, our group segment should follow the principle that ∪ Nn =1 C n = ∂
G et and C m ∩ C n = ∅ for any m (cid:54) = n ; m, n = 1 , 2 , . . .
, N .
Here, we set the last visited state s last in the previous episodes as C 0 , since s last often is both close to the terminal state and has high uncertainty.
As shown in Figure 3, based on the last explored state, we build group C n , n = 1 , . .
, N by extending scales in two different perspectives.
We list two perspectives, as follows:
• Extending from Neighbor Nodes .
One perspective is to leverage local structure on graph, where C n := { s } , s ∈ N n ( s last
) ∩ ∂ G et for n = 1 , 2 , . .
. , N , and C N :=
∂ G et −∪ N − 1 n =1 C n .
N n ( s last ) means n -hop neighbors of s last .
The intuitive motivation behind this is to keep the learning procedure stable by slowly adjusting the goal to explore the surrounding environment.
• Extending from Uncertain Nodes .
The other perspective is to utilize certainty information to guide goal generation, where C n := { s } , s ∈ S d = | A |− n ∩ ∂ G et for n = 1 , 2 , . . .
, N , and C N :
= ∂ G et −∪ Nn =1 C n .
S d = | A |− n denotes set of states whose degree equals | A |− n , and | A | is the size of action space.
The intuitive motivation behind this is to eliminate uncertainty in the graph through exploration.
We provide the complexity analysis for two perspectives as follows.
Let d ∂ G et denote the maximum degree of states in ∂ G et , and | S ∂ G et | denote the number of states in ∂ G et .
The complexity to construct
C 1 , . .
, C N by extending from neighbor nodes is O ( d N − 1 ∂ G et ) and by extending from uncertain nodes is O ( | S ∂ G et | ) . The detailed analysis and method are available in Appendix E.3. Attention Strategy. One should be noted that extending from either neighbor or uncertain nodes can guarantee the number of groups C 1 , . . .
, C N on a dynamic graph G et is ﬁxed.
In order to keep all the groups not empty, we append s last to them at the beginning timestep of each episode.
However, RL algorithms face an exploration-exploitation dilemma, which requires the agent to makes appropriate selection rather than random exploration over these groups.
In order to the trade-off between exploration and exploitation, we apply an attention mechanism to select an appropriate group on the current situation.
For each group, we learn an embedding vector to represent its feature (i.e., certainty for extending from uncertain nodes ).
In this paper, the attention module runs over
N groups at each episode and select the appropriate one, which needs to take the features of other groups into consideration.
Therefore, it’s natural to adopt self-attention mechanism (Vaswani et al., 2017) here.
For simplicity, we denote all the features of groups as [ f 1 , . . . , f N ] , and then deﬁne Q = K = V ( f 1 , . . , f N ) T .
The self-attention can be deﬁned as [Equation] where ATT φ denotes self-attention function parameterized by φ . And the selected group can be obtained according to
[Equation] where σ ( · ) denotes a sigmoid function, and C ATT means the group selected according to the attention score. At the beginning timestep of any episode e , we have access to state-transition graph G eand its boundary ∂ G e 0 . We then divide ∂ G e 0 into groups C 1 , . . .
, C N , and obtain C ATT through the aforementioned attention mechanism.
We select one state with the highest value in
C ATT as the goal, which can be formulated as
[Equation]
In this way, we can generate an appropriate goal g e to guide the agent exploration in the episode e . 3. L EARN WITH G RAPH S TRUCTURED R EINFORCEMENT L EARNING As illustrated in Figure 1, we update goal generation under supervised learning and value estimation under reinforcement learning. Speciﬁcally, we introduce learning strategies as follows:
Goal Learning Strategy.
As mentioned above, we learn the goal generation hindsight. As illustrated in Figure 4, at the beginning timestep of each episode e + 1 , we ﬁrst compute the shortest path distance between initial state ( s 0 in (c)) and highest value state ( s 9 in (c)) to obtain the solution path ( P e +1 = (cid:104) s 0 , s 1 , s 3 , s 6 , s 9 (cid:105) in (d)). We then ﬁnd the optimal goal g ∗ e § at episode e based on the solution path generated at episode e + 1 .
Speciﬁcally, we search for the intersection state between
P e +1 and ∂ G et in the inverse order of the solution path.
Given that s 9 is unexplored and unreachable in the episode e , we select s 6 as the optimal goal g ∗ e .
We then ﬁnd the optimal group C ∗ that contains goal g ∗ e .
With this hindsight supervision on the group selection, we are able to update the attention mechanism in Eq.
(3) via a standard supervised learning algorithm, where the objective function can be formulated as
[Equation]
[Figure]
[Figure]
[Figure] [Figure] [Figure] where (cid:107) φ (cid:107) 2 is regularizer and α is the corresponding hyper-parameter.
This method updating goal generation under the supervision of the group instead of goal can eliminate instability brought from potentially inaccurate value estimation because our group division does not involve the result from value estimation. Value Learning Strategy. With a generated goal at the beginning of each episode, we can build the critical block of our method, i e., a goal-conditioned policy and its associated value function. We consider a goal-reaching agent interacting with an environment.
The agent observes its current state s ∈ S and a goal state g ∈ G .
The dynamics are governed by the distribution P ( s t +1 | s t , a t ) .
At every timestep, the agent samples an action a ∼ π ( a | s, g ) and receives a corresponding reward r g ( s, a ) that indicates whether the agent has reached the goal. The episode terminates after T timesteps, and even the agent reaches the goal. The agent’s task is to maximize its cumulative and undiscounted reward.
We use an off-policy algorithm to learn such a policy, as well as its associated goal-conditioned Q -function.
For example, we obtain a policy by acting greedily, w.r.t. , the
Q -function as [Equation] In order to improve data efﬁciency and obtain good value estimation, we consider choosing an offpolicy RL algorithm with goal relabelling and update parameters with related trajectories. Specifically, we choose an off-policy RL algorithm with goal relabelling, such as Andrychowicz et al.
We deﬁne related data as those trajectories that contain neighborhood nodes of the current state.
Formally, when we update Q -function of state s , we sample D related := { τ } from replay buffer
D , where τ denotes the trajectory that contains at least one state in s ’s neighborhood N 1 ( s ) .
The
Q -network is learned by minimizing the following objective function:
[Equation]
where β is the weight of the regularization term.
Besides, as we show in Proposition 3, our GSRL algorithm can converge to a unique optimal point if the Q -learning strategy is adopted.
Proposition 3.
Denote the Bellman backup operator in Eq.
(6) as B : R | S |×| A |×| G | → R | S |×| A |×| G | and a mapping Q : S ×
A × G → R | S |×| A |×| G | with | S | < ∞ and | A | < ∞ .
Repeated applications of the operator B for our graph-based state-action value estimate ˆ Q G converges to a unique optimal value ˆ Q ∗G ∗ with well-explored graph G ∗ (i.e., fully explored graph G full ) including optimal solution path.
Proof.
The proof is shown in Appendix D.3.
Overall Algorithm.
We provide the overall algorithm in Algorithm 1 in Appendix B, and an illustrated example in Figure 8 in Appendix C. One can see that our state-transition graph can be applied to both the learning and planning phase.
Previous graph-based RL algorithms either focus on learning (Zhu et al., 2019) or planning (Eysenbach et al., 2019) based on structured information of the state-transition graph.
E XPERIMENTS
In this section, we perform an experimental evaluation of our algorithm and compare it with other state-of-the-art methods.
Our experiment environments are based on the standard robotic manipulation environments in the OpenAI Gym (Brockman et al., 2016).
We provide the experimental results to answer the following questions:
Can GSRL obtain better convergence in various environments? 2.
Can GSRL tackle a high-dimensional and continuous environment with obstacles?
Can GSRL perform higher sample efﬁciency?
What is the inﬂuence of group selection for goal generation and related experience for value estimation in GSRL?
Then we check whether the exploration provided by the goals generated by HGG can result in better policy training performance.
As shown in Figure 3, we compare the vanilla HER, HER with EnergyBased Prioritization (HER+EBP), HGG, HGG+EBP.
It is worth noting that since EBP is designed for the Bellman equation updates, it is complementary to our HGG-based exploration approach.
Among the eight environments, HGG substantially outperforms HER on four and has comparable performance on the other four, which are either too simple or too difﬁcult.
When combined with EBP,
HGG+EBP achieves the best performance on six environments that are eligible.
Performance on tasks with obstacle In a more difﬁcult task, crafted metric may be more suitable than ` 2 -distance used in Eq.
As shown in Figure 4, we created an environment based on
FetchPush with a rigid obstacle.
The object and the goal are uniformly generated in the green and the red segments respectively.
The brown block is a static wall which cannot be moved.
In addition to ` 2 , we also construct a distance metric based on the graph distance of a mesh grid on the plane, the blue line is a successful trajectory in such hand-craft distance measure.
A more detailed description is deferred to Appendix B.3.
Intuitively speaking, this crafted distance should be better than ` 2 due to the existence of the obstacle.
Experimental results suggest that such a crafted distance metric provides better guidance for goal generation and training, and signiﬁcantly improves sample efﬁciency over ` 2 distance.
It would be a future direction to investigate ways to obtain or learn a good metric.
5.2 Comparison with Explicit Curriculum Learning
Since our method can be seen as an explicit curriculum learning for exploration, where we generate hindsight goals as intermediate task distribution, we also compare our method with another recently proposed curriculum learning method for
RL.
Florensa et al.
(2018) leverages Least-Squares GAN (Mao et al., 2018b) to mimic the set called Goals of Intermediate
Difﬁcult as exploration goal generator.
Speciﬁcally, in our task settings, we deﬁne a goal set GOID ( ⇡ ) = { g : ↵  f ( ⇡, g )  1  ↵ } , where f ( ⇡, g ) represents the average success rate in a small region closed by goal g .
To sample from GOID , we implement an oracle goal generator based on rejection sampling, which could uniformly sample goals from GOID ( ⇡ ) .
Result in Figure 5 indicates that our Hindsight Goal Generation substantially outperforms HER even with GOID from the oracle generator.
Note that this experiment is run on a environment with ﬁxed initial state due to the limitation of Florensa et al.
The choice of ↵ is also suggested by Florensa et al.
5.3 Ablation Studies on Hyperparameter Selection
In this section, we set up a set of ablation tests on several hyper-parameters used in the Hindsight
Goal Generation algorithm.
Lipschitz L : The selection of Lipschitz constant is task dependent, since it iss related with scale of value function and goal distance.
For the robotics tasks tested in this paper, we ﬁnd that it is easier to set L by ﬁrst divided it with the upper bound of the distance between any two ﬁnal goals in a environment.
We test that is
We is
A
Proof of Theorem 1 In this section we provide the proof of Theorem 1.
Theorem 1.
Assuming that the generalizability condition (Eq.
(4) ) holds for two distributions ( s, g ) ⇠ T and ( s 0 , g 0 )
⇠ T 0 , we have V ⇡ ( T 0 )
V ⇡ ( T )  L · D ( T , T 0 ) .
(6) where D ( · , · ) is the Wasserstein distance based on d ( · , · ) D ( T (1) , T (2) )
= inf µ 2 ( T (1) , T (2) )
⇣ E µ [ d (( s 0(1) , g (1) ) , ( s 0(2) , g (2) ))]
⌘ where ( T (1) , T (2) ) denotes the collection of all joint distribution µ ( s 0(1) , g (1) , s 0(2) , g (2) )
whose marginal probabilities are T (1) , T (2) , respectively.
Proof.
By Eq.
(4), for any quadruple ( s, g, s 0 , g 0 ) , we have
[Equation]
For any µ 2 ( T , T 0 ) , we sample ( s, g, s 0 , g 0 )
⇠ µ and take the expectation on both sides of Eq.
(10), and get
[Equation]
( s, g ) , ( s 0 , g 0 ))] .
Since Eq.
(11) holds for any µ 2 ( T , T 0 ) , we have
[Equation]
inf µ 2 ( T , T 0 )
( E
[Equation]
[ d ((
[Equation]
g ) , ( s 0 , g 0 ))])
=
V ⇡ ( T )  L · D ( T , T 0 ) .
B Experiment Settings B. Modiﬁed Environments
[Figure]
• FetchPush-v1: Let the origin (0 , 0 , 0) denote the projection of gripper’s initial coordinate on the table.
The object is uniformly generated on the segment (  0 .
15 ,  0 .
15 , 0)  (0 . 15 ,  0 .
, and the goal is uniformly generated on the segment (  0 . 15 , 0 . 15 , 0)  (0 . 15 , 0 . 15 , 0) .
• FetchPickAndPlace-v1: Let the origin (0 , 0 , 0) denote the projection of gripper’s initial coordinate on the table.
The object is uniformly generated on the segment (  0 .
15 ,  0 .
15 , 0)  (0 . 15 ,  0 .
, and the goal is uniformly generated on the segment (  0 . 15 , 0 . 15 , 0 . 45)  (0 . 15 , 0 . 15 , 0 . 45) .
• FetchSlide-v1: Let the origin (0 , 0 , 0) denote the projection of gripper’s initial coordinate on the table.
The object is uniformly generated on the segment (  0 .
05 ,  0 .
1 , 0)  (  0 .
05 , 0 . 1 , 0) , and the goal is uniformly generated on the segment (0 .
55 ,  0 .
15 , 0)  (0 . 55 , 0 . 15 , 0) .
[Figure]
Published
[Figure]
conference paper at ICLRmemorized states like episodic memory, and maintain a graph on top of these states based on state transitions at the same time.
Then we develop an efﬁcient reverse-trajectory propagation strategy to allow the values of new experiences to rapidly propagate to all memory items through the graph.
Finally, we use the fast-adjusted non-parametric high values in associative memory as early guidance for a parametric RL agent so that it can rapidly latch on states that previously yield high returns instead of waiting for many slow gradient updates.
[Figure]
To illustrate the superiority of the associative memory in reinforcement learning, consider a robot exploring in a maze to seek out the apple (at place G), as shown in Figure 1.
It collects two trajectory experiences starting from place A and B, respectively.
All the states of tra- jectory A (the top blue dash line) receive no reward because the agent terminates at a state with a non-zero
re-
ward (at place C), while in trajectory B (the bottom blue dash line) the ﬁnal non-zero reward of catching an ap- ple (at place G) back-propagates through the whole path.
Episodic memory keeps a high value at the intersection of two trajectories (the door) when taking actions toward lower-right corner while recording zero values at the other states in trajectory A.
If an episodic memory based robot starts from place A, it will wander around A because there are no positive values indicating the way to goal.
Thus based on the episodic memory, the robot may eventually take a policy like the green line after multiple attempts.
However, if the robot adopts associative memory, the high value in the door collected from trajectory B will be further propagated to the start point A and thus the robot can correctly take the red-line policy.
To some extent, our associative memory is equivalent to automatic augmentation of counterfactual combinatorial trajectories in memory.
Thus, our framework signiﬁcantly improves the sampleefﬁciency of reinforcement learning.
Comparisons with state-of-the-art episodic reinforcement learning methods show that ERLAM is substantially more sample efﬁcient for general settings of reinforcement learning.
In addition, our associative memory can be used as a plug-and-play module and is complementary to other reinforcement learning models, which opens the avenue for further researches on associative memory based reinforcement learning.
2 B ACKGROUND
In the framework of reinforcement learning (Sutton & Barto, 1998), an agent learns a policy to maximize its cumulative rewards by exploring in a Markov Decision Processes (MDP) environment.
An MDP is deﬁned by a tuple ( S , A , P, R ,  ) , where S is a ﬁnite set of states, A is a ﬁnite set of actions available to the agent, P :
S ⇥ A ⇥ S !
R deﬁnes the transition probability distribution,
R is the reward function, and  2 (0 , 1] is the discount factor.
At each time step t , the agent observes state s t 2 S , selects an action a t 2 A according to its policy ⇡ : S !
A , and receives a scalar reward r t .
In the setting of ﬁnite horizon, the accumulated discounted return is calculated as,
R t =
P
{+Tk =
+}
k r t
+ k where T is the episode length and goal of the agent is to maximize the expected return for each state s t .
The state-action value function Q ⇡ ( s, a ) =
E [ R t | s t =
s, a ] is the expected return for executing action a on state s and following policy ⇡ afterwards.
DQN (Mnih et al., 2015) parameterizes this action-value function by deep neural networks Q ✓ ( s, a ) and use Q-learning (Watkins & Dayan, 1992) to learn it to rank which action at is best to take in each state s t at time step t .
The parameters of the value network ✓ are optimized by minimizing the L 2 difference between the networks output
Q ✓ ( s, a ) and the Q-learning target y t =
r t +  max a Q ˆ
In order to answer the ﬁrst question, we demonstrate our method in various robotic manipulation tasks including reach task (see Figure 5(a)) and fetch task (see Figure 5(c)).
To answer the second one, we investigate how our method performs on environment with high-dimensional state and action space (see Figure 5(b)) and obstacle (see Figure 5(d)).
We answer the third one by comparison on the convergence of learning curves in Figure 6.
By sample efﬁciency, we further analysis exploration performance in Appendix F.4.
Speciﬁcally, we conduct experiments with previous approaches:
• HER : Andrychowicz et al.
(2017) generated imaginary goals in a simple heuristic way to tackle the sparse reward issue.
• MAP :
Huang et al.
(2019) explicitly modeled the environment in a hierarchical manner, with a high-level map abstracting the state space and a low-level value network to derive local decisions.
• GoalGAN : Florensa et al.
(2018) leveraged Least-Squares GAN (Mao et al., 2018b) to mimic the set of Goals of Intermediate Difﬁculty as an automatic goal generator.
• CHER :
Fang et al.
(2019) proposed to enforce more curiosity in earlier stages and changes to larger goal-proximity later.
One should be noted that graph structure in GSRL is constructed on top of the replay buffer for goal generation and value estimation, which can be closely incorporated with policy networks such as
DQN (Mnih et al., 2013), DDPG (Lillicrap et al., 2015), etc.
To demonstrate the real performance gain from our GSRL, we set the policy network with DDPG for GSRL and all the baselines.
The detailed description of environments, experiment settings, and implementation details can be found in Appendix F.1.
Maze .
We ﬁrst test our method and other strong baselines in the Maze environment, where the ant agent learns to reach the speciﬁed target position ( (cid:15) -ball depicted in red) located at the other end of the U-turn as shown in Figure 5(a).
This environment is quite simple, where most RL approaches can converge with a success rate 1 .
As Figure 6(a) illustrates, GSRL performs with high sample efﬁciency.
AntMaze .
We show that our GSRL is efﬁcient in a complex environment of robotic agent navigation tasks, as illustrated in Figure 5(b), where the state space is of 8 dimensions and the action space is of 30 dimensions.
Duan et al.
(2016) showed that standard RL methods are unable to solve it.
As shown in Figure 6(b), GSRL achieves the best asymptotic performance and sample efﬁciency.
FetchPush .
As shown in Figure 5(c), in the fetch environment, the agent is trained to fetch an object from the initial position (rectangle in green) to a distant position (rectangle in red).
Although the fetch tasks are more complicated than they reach ones in the maze, GSRL also yields large performance gain, as shown in Figure 6(c).
FetchPush with Obstacle .
As Figure 5(d) illustrates, we create an environment based on FetchPush with a rigid obstacle, where the brown block is a static wall that cannot be moved.
Experimental results in Figure 6(d) suggests that the graph structure of state-transitions can provide additional useful information, which leads to better guidance for goal generation and value estimation.
Impact of Environment Size .
In order to investigate whether GSRL can be well adapted in the environments with different complexity sizes, we extend the Maze environment, as shown in Figure 5(a).
Maze environment with larger size usually means more sparse rewards.
We report the performance comparisons on the three environments with SMALL , MEDIUM , and LARGE sizes in Figure 6(e).
More details of these environments are available in Appendix F.1.
One can see that
GSRL can be well address sparse reward issue in various environments.
Impact of Environment Complexity .
In order to investigate whether GSRL can be well adapted in the environments with different complexity levels, we extend the Maze environment with a more complex or high-dimensional structure.
We create AntMaze with Obstacle environment, where we extend the AntMaze environment with an obstacle.
The performance of GSRL in this environment is illustrated in Figure 6(f).
More details of these environments are available in Appendix F.1.
One can see that GSRL can be well adapted in various environments with different complexity levels.
Impact of Group Selection .
We provide two strategies to divide the boundary of the graph into several groups, namely extending from neighbor nodes and extending from uncertain nodes in Section 3.
We demonstrate the performance of these two strategies in Figure 6(g).
We set GSRL without using the attention strategy on the groups as NOGroup .
We adopt the ﬁrst strategy and set the number of groups as 3 , which corresponds to NEIGH3 .
We then show the performance of the second strategy with the number of groups equal to 2 , 3 and 4 , which correspond to UNCERT2 , UNCERT3 and
UNCERT4 .
Results show that GSRL without any group selection would lead to inefﬁciency since the goal generation strategy at the beginning is almost random.
Both of these two strategies perform well, as illustrated.
We also provide complexity analysis in Appendix E.3.
In the main experiment, we adopt extending from uncertain nodes with 3 groups (i.e., UNCERT3).
Impact of Discretization .
We apply K -bins discretization technique (Kotsiantis & Kanellopoulos, 2006) to discretize the continuous state and action spaces, where there is a wrapper that converts a
Box observation into a single integer.
DISCRETE10 , DISCRETE20 and DISCRETE30 in Fig- ure 6(h) denote the performances of K = 10 , 20 and 30 respectively.
We ﬁnd that for the simple tasks, the choice of K is not critical.
K is set at 20 in the main experiment.
Impact of Related Experience .
We further study the impact of using neighborhood structure of the state-transition graph to select related experience to efﬁciently update value estimation.
We build a variant to update value estimation from transitions drawn uniformly from the replay buffer D , denoted as ALL .
RELATED is used to denote GSRL, where D related is adopted instead of D .
R ELATED W ORK
Structured Models of Environments .
Recent work has investigated leveraging structured environments to make great strides in improving predictive accuracy (Kipf et al., 2018;
Xu et al., 2019;
Chang et al., 2017; Battaglia et al., 2016; Watters et al., 2017; Sanchez-Gonzalez et al., 2018) and accelerating reinforcement learning procedures (Shang et al., 2019; Kipf et al., 2020;
Eysenbach et al., 2019; Wang et al., 2018).
These methods mainly ﬁrst construct some form of graph neural network where node update functions model the dynamics of individual objects, parts, or agents and edge update functions model their interactions and relations through unsupervised object discovery (Xu et al., 2019), contrastive learning (Kipf et al., 2020) or agent exploration (Eysenbach et al., 2019).
Unlike these previous works based on well-organized graphs, our model GSRL constructs and makes use of the graph from scratch.
Specially, GSRL constructs a dynamic map via exploration and builds efﬁcient exploration with structured information from this map, which allows our approach to be widely deployed into more complex environments besides object interaction simulations.
Goal-oriented Reinforcement Learning.
Goal-oriented RL allows an agent to generate intrinsic rewards, which is deﬁned with respect to target subsets of the state space called goals (Florensa et al., 2018;
Paul et al., 2019).
Recently, goal-oriented RL has been investigated widely in various deep RL scenarios such as imitation learning (Pathak et al., 2018; Srinivas et al., 2018), disentangling task knowledge from the environment (Mao et al., 2018a; Ghosh et al., 2019), constituting lower-level controller in hierarchical RL (Shang et al., 2019; Nachum et al., 2018) and organizing cooperation in multi-agent RL (Jin et al., 2019; Yang et al., 2020).
Hence, how to generate appropriate goals is the essential technique in any goal-oriented RL (Andrychowicz et al., 2017; Ren et al., 2019).
Eysenbach et al. (2019) proposed to utilize the shortest-path search on replay buffer to generate a sequence of goals. Instead of planning goals from a well-explored state-transition graph, in this paper, we investigate guiding exploration with structured information from a dynamic graph, which can largely improve the performance in terms of sample efﬁciency in tasks with sparse reward signals. Also, different from (Zhao et al., 2019), our work utilize the structure information in the state-transition graph to select related trajectories for efﬁcient learning.
Hierarchical Reinforcement Learning .
Hierarchical RL learns a set of primitive tasks that together help an agent learn the complex task.
There are mainly two lines of work.
One class of algorithms (Shang et al., 2019; Nachum et al., 2018; Bacon et al., 2017; Vezhnevets et al., 2017) jointly learn a low-level policy together with a high-level policy, where the lower-level policy interacts directly with the environment to achieve each task, while the higher-level policy instructs the lower-level policy via high-level actions or goals to sequence these tasks into the complex task.
The other class of methods (Bagaria & Konidaris, 2019; Fox et al., 2017; Hartikainen et al., 2019; Pitis et al., 2020;
Pong et al., 2019) focuses on discovering sub-tasks or sub-goals that are easy to reach in a short time and can guide the agent to the terminal state.
The common idea GSRL shares with Hierarchical RL is to decompose the complex task into several sub-tasks to achieve.
The key difference is that in this paper, we propose to build the state-transition graph and utilize the structure information for goal generation and value estimation.
CONCLUSION
In this paper, we propose a novel framework called GSRL, which leverages structure information of the state-transition graph for efﬁcient goal generation and value estimation.
We provide theoretical analysis to show the efﬁciency and converge property of our method.
Results on various challenging robotic manipulation environments demonstrate that GSRL can outperform the state-of-the-art RL algorithms.
In the future, this work would shed light on graph-structured RL for efﬁcient learning and planning, where various graph-based algorithms can help RL agents learn and infer in highly structured environments. A N OTATIONS [Figure] neighborhood
[Figure]
graph at timestep t graph at timestep t+ boundary of graph graph increment (b)   2denote t t+ t
In this paper, we construct a state-transition graph on top of the replay buffer.
As Figure 7(a) shows, we build the graph G et based on historical explored trajectories at timestep t in episode e .
For any not fully explored state-transition graph, there exist many poorly-explored states.
We measure the exploration (i.e., certainty in Deﬁntion 1) of these states according to the number of their untaken candidate actions.
As illustrated in Figure 7(b), we deﬁne the boundary of G et as a set of states, at least one of whose candidate actions is not readily taken.
Each untaken action may lead to unvisited states (denoted by ? icon).
We denote the boundary as ∂ G et .
As illustrated in Figure 7(c), after each timestep t + 1 , the agent explored a new state denoted as ∆ G et .
Then, G et and ∆ G et together make up the dynamic graph at timestep t + 1 denoted as G et +1 .
B A LGORITHM
g ) 8:
Take action a t , receive reward r t and next state s t + 9:
Append ( s t , a t , r t , s t +1 , g e ) to D 10:
Relabel rewards r g with g e 11:
Append ( s t , a t , s t +1 ) to G if ( s t , a t , s t +1 ) / ∈ G 12: if t mod update interval =
= 0 then 13:
Sample related experience ( s t , a t , s t +1 , r t , g e ) to D related 14:
Update parameter θ using Eq.
(7) 15: end if 16: end for 17:
Compute optimal goal g ∗ e according to Deﬁnition18:
Update parameter φ using Eq.
(5) 19: end for We provide the overall algorithm in Algorithm 1. The key contribution of our paper is to leverage structured information in the state-transition graph for efﬁcient goal generation and value estimation, which is represented in line 4 and 13, respectively. We then describe the overall procedure of GSRL according to Algorithm 1 as follows:
There is no graph structure for the agent to support when the task starts.
Hence, the agent initializes the replay buffer D and the state-transition graph G in line 1.
At the beginning timestep of each episode e , we divide the boundary of the state-transition graph ∂ G e 0 into N groups and adopt attention mechanism to select an appropriate one for exploration in line 3. Within the selected group, we choose the state with the highest value as the generated goal g e in line 4. The agent tries to reach the goal state through current policy-based Q value in line 7 and record interaction history in the replay buffer in line 9. As goal-oriented RL provides the agent intrinsic reward conditioned on the current goal, the agent is required to relabel reward with r g conditioned on g e in line 10.
Then the agent updates the state-transition graph in line 11.
In order to efﬁciently update policy, the agent sample related trajectories that contain at least one neighbor states of the current state in line 13.
In line 14, the policy is updated with DDPG (Lillicrap et al., 2015).
At the end of each episode e , the state-transition graph is actually built for episode e + 1 denoted as G e + . The agent is able to ﬁnd optimal goal g ∗ e through planning algorithm on G e + in line 17.
The attention mechanism is updated by supervised learning in line 18. C I LLUSTRATION [Figure] We provide an illustrated example for GSRL in Figure 8.
When an agent encounters a complex environment, GSRL ﬁrst discretize the continuous space into a discrete one, as illustrated in (a).
We color the last state (i.e., s last =
s 7 ) visited in the previous episodes with red. Meanwhile, the historical trajectories stored in the replay buffer can be represented with several paths (e.g., (cid:104) s 0 , s 1 , s 3 , s 6 (cid:105) ). With these paths, we can easily construct a state-transition graph at the beginning timestep of episode (i.e., G e 0 ).
As illustrated in (b), we can ﬁnd the boundary of the graph (i.e., ∂ G et ) as shown in Appendix A. In this case, s 5 , s 6 are in the boundary of the current graph (i.e., ∂ G e 0 ). We then follow extending from uncertain nodes to divide the boundary into several groups. The number groups is the hyper-parameter, and we choose 3 here.
For each group, we include s last since s last is often with high value and close to the terminal state. As shown in (c), we then adopt attention mechanism to select one group C ATT and assign the state with the highest value in the select group as the goal g e . With the generated goal, the agent can perform goal-oriented exploration to divide the complex task into several sub-tasks to deal with.
There are two learned parts in GSRL, namely attention network for goal generation, and value network for value estimation. We design a hindsight learning approach to update the attention network at the beginning timestep of each episode (as shown in Figure 4), and use related experiences to update value network at each update interval (as shown in Figure 2). D P ROOFS D. P ROOF OF P ROPOSITION Proposition 1.
Given the whole state-transition graph G whole , we assume that the probability of degree of an arbitrary state being less than or equal to d is larger than p (i.e., P ( deg ( s ) ≤ d ) > p, ∀ s ∈ S G ). Considering a sequence of consecutively expanding sub-graphs {{G 1 t } T − 1 t =0 , {G 2 t } T − 1 t =0 , . . . , {G Et } T − 1 t =0 } , starting with G 01 =
{ s 0 } , for all t ≥ 0 , e ≥ 1 , we can ensure that P (cid:0) | S G et | ≤ (cid:15) (cid:1) > p (cid:15) , where (cid:15) =
d · ( d − 1) T · e + t − 2 d −when d > 2 and (cid:15) = 1 , 3 when d = 1 ,respectively.
Proof.
We consider the extreme case of greedy consecutive expansion at each timestep t in any episode e , where G et +1 =
G et ∪ ∆ G et =
G et ∪ ∂ G et , since if this case satisﬁes the inequality, any case of consecutive expansion can also satisfy it.
By deﬁnition, all the subgraphs
G et are a connected graph.
Here, we use ∆ S t to denote S ∆ G t for short. In each episode, we can ensure that the newly added nodes ∆ S et at timestep t only belong to the neighborhood of the last added nodes ∆ S et − 1 .
Within each episode e , we study the sequence { ∆ G e 0 , ∆ G e 1 , . . .
, ∆ G eT − 1 } , where T is the episode length.
In this case, each node in ∆ S et already has at least one edge within ∆
G et − 1 due to the deﬁnition of connected graphs.
We can have
[Equation]
For e = 1 and t = 0 , we have P ( | ∆ S 11 | ≤ d ) > p and thus
[Equation]
For e ≥ 1 and t ≥ 1 , we analyze the conseutive expansion of the state-transition graph G as
G
[Equation]
Given that | ∆ S G et | ≥ 1 , ∀ t ∈ [0 , T − 1] , we consider the extreme case that | ∆ S G et | = 1 , ∀ t ∈ [0 , T − 1] , which means that every exploration will result in a new explored state and should be respondes to the upper bound of the explosion.
Based on | ∆ S G et | = 1 + | ∆ S G 10 | + | ∆ S G 11 | + · · · + | ∆ S
[Equation]
T −
[Equation]
|
[Equation]
|
[Equation]
S
[Equation]
|
[Equation]
|
[Equation]
S
[Equation]
|
[Equation]
[Equation]
|
[Equation]
1 |
[Equation]
· · · + | ∆ S G e 0 | + · · · + | ∆ S G et | , we have
[Equation]
When d = 1 , there can be only one node, so in this case, (cid:15) = 1 .
When d = 2 , we follow Eq.
(11) and derive that in this case, (cid:15) = 3 .
When d > 2 , it holds that
[Equation]
We can ﬁnd that t = 0 also satisﬁes this inequality.
Notice that here we share some mathematical derivations with Xu et al.
D. P ROOF OF P ROPOSITION
Proposition 2.
Assume that Q -value of each state is well estimated (i.e., Q = Q ∗ ), then optimal goal g ∗ e at the beginning timestep of any episode e is always included in the boundary of the statetransition graph G e 0 (i.e., ∂ G e 0 ).
Proof.
According to Deﬁnition 2, as shown in Figure 9(a), in the fully explored graph G full , the optimal goal g ∗ full is the terminal state in the optimal solution P full , which is also the terminal state in the environment, i.e. , s 10 .
The intuitive explanation behind this is very natural, where the environment in this case is fully explored, and thus the agent is ready to target at the terminal state.
In the other cases, we generate the optimal goal g ∗ e of episode e at the episode e + 1 .
Specially, we ﬁnd the shortest path to the highest value state in G e + as the optimal solution path P
e +1 .
As Figure 9 illustrates, in the episode e + 1 , the highest value is s 9 and the optimal solution path in this case is
P e +1 =
(cid:104) s 0 , s 1 , s 3 , s 6 , s 9 (cid:105) .
We then compare the explored states in G e 0 with the states in P inverse e +1 , where P inverse e + = (cid:104) s 9 , s 6 , s 3 , s 1 , s 0 (cid:105) is the inverse order of P e +1 .
As Figure 9(d) shows, ﬁnally we obtain s 6 as the optimal goal g ∗ e .
As stated above, it’s easy to ﬁnd that there are two cases in the optimal goal generation.
One is the last node of solution path P
e +1 .
The other is one of the rest nodes in P e +1 except the last one.
We then prove that in both of these cases, optimal goal g ∗ e is always included in the boundary of the state-transition graph ∂ G e 0 .
Case I: Node at Last.
If Q -value of each state is well estimated, i.e., Q = Q ∗ , then the optimal solution path P e +1 at episode e
+1 should be close to the optimal solution path P full in the full graph
G full , and the last state of the path P
e +1 should be closest to the terminal state.
Hence, if g ∗ e is not in
[Figure]
ssssssssss 0 sssssssss
[Figure] [Figure]
ssssss
[Figure]
e
[Figure]
optimal solution path
[Figure]
generated goal the boundary, there must be one neighbor node closer to the terminal state.
Otherwise, g ∗ e is the dead end and thus should not be regarded as the [-solution path ( P e +
-]optimal goal.
And if there is one neighbor node closer to the terminal state, then this state should be regarded as the optimal goal.
Therefore, we obtain a contradiction.
Case II: Node Not at Last.
If the optimal goal is not the last state, then there must exist the state unexplored at episode e .
Take Figure 9 as an example, if we take s 6 as the optimal goal g ∗ e in (d), state s 9 must be unexplored in G e 0 in (c) and explored in G e + in (b).
If g ∗ e is not included in ∂ G e 0 , then there should not exist any unexplored state that is included in its neighborhood.
According to the deﬁnition of the boundary of the graph, we have proved the proposition by contradiction.
In summary, we have proved the proposition in both two cases by contradiction.
D. P ROOF OF P ROPOSITION
Proposition 3.
Denote the Bellman backup operator in Eq.
(6) as B : R | S |×| A |×| G | → R | S |×| A |×| G | and a mapping Q : S ×
A × G → R | S |×| A |×| G | with | S | < ∞ and | A | < ∞ .
Repeated applications of the operator B for our graph-based state-action value estimate ˆ Q G converges to a unique optimal value ˆ Q ∗G ∗ with well-explored graph G ∗ (i.e., fully explored graph G full ) including optimal solution path. Proof.
The proof of Proposition 3 is done in two main steps. The ﬁrst step is to show that our statetransition graph G can converge to the well-explored graph G ∗ . Here, we deﬁne G ∗ as the graph that includes the optimal path (i.e.,
P full in Deﬁnition 2). In the second step, we prove that given graph G , our graph-based method can converge to unique optimal value Q ∗G . Step I. Since | S | < ∞ and | A | < ∞ , we can obtain that V G < ∞ and E G < ∞ .
Note that the state-transition graph G is a dynamic graph, and goals g generated on G are updated at the beginning timestep of each episode. Hence, there is a sequence of goals denoted as ( g 1 , g 2 , · · · , g E ) and corresponding sequence of graphs denoted as ( G 10 , G 20 , · · · , G E 0 ) , where E here is the number of episodes. Given that | S | < ∞ and | A | < ∞ , the number of nodes and edges in the full graph G full is also bounded.
Based on the explore strategy introduced in Section 3, we know that goal-oriented RL will ﬁrst search for a path leading to the terminal state. After that, the terminal state will be included in G . Then the agent will seek the shortest path to the terminal state because the agent is given a negative reward at each timestep. Hence, the optimal solution path P full will be involved.
Hence, we can obtain that [Equation] Assume that E is large enough, our state-transition graph G can ﬁnally converge to well-explored graph G ∗ . Step II.
Note that the proof of convergence for our graph-based goal-oriented RL is quite similar to Q -learning (Bellman, 1966; Bertsekas, 1995; Sutton & Barto, 2018). The differences between our approach and Q -learning are that Q value Q ( s, a, g ) is also conditioned on goal g , and that the state-transition probability P G ( s (cid:48) | s, a ) can be reﬂected by graph G . We provide detailed proof as follows: For any state-transition graph G , we can obtain goal g ∈ G conditioned on G from Step I. Based on that, our estimated graph-based action-value function ˆ Q G can be deﬁned as
[Equation] For any action-value function estimates ˆ Q 1 G , ˆ Q 2 G , we study that [Equation]
So the contraction property of Bellman operator holds that [Equation] For the ﬁxed point Q ∗G , we have that [Equation]
[Equation]
Combining Step I and II, we can conclude that our graph-based estimated state-action value ˆ Q G can converge to a unique optimal value Q ∗G ∗ .
E D ISCUSSIONS E. D ISCUSSION ON C ERTAINTY OF S TATE
In this section, we further discuss the relationship between the certainty of state and the number of states.
In the previous exploration RL literature (Ostrovski et al., 2017; Bellemare et al., 2016), the performance of exploration often is measured by the number of the visited states.
Namely, given a ﬁxed number of episodes, more visited states, better performance.
In this paper, we propose to utilize a new measurement, i.e. , certainty of state as illustrated in Deﬁnition 1.
We conclude the relations between certainty and the number of visited states as Proposition 4.
Proposition 4. Given a whole state-transition graph G whole , we can regard the certainty of states as the local measurement and the number of states as the global measurement for exploration, which share a similar trend during agent exploration. Proof. We illustrate and prove the proposition hindsightly.
If we have the full observation for states as shown in Figure 10(a), we can model the agent ﬁnding new states as connecting new states with visited states.
In other words, since the state-transition graph G et must keep being a fully connected graph at any timestep t in any episode e .
Hence, adding new states into the visited state set can always be regarded as ﬁnding new edge between new states and the visited state set.
And each directed edge in the state-transition graph, as shown in Figure 10(b) is determined by action and state-transition function.
If the environment is determined, we can roughly regard the number of edges as the approximate measurement for exploration.
The certainty of states is the local perspective for this measurement.
[Figure]
state terminal state (a) s 0 ssssssssssinitial state terminal state (b) s 0 ssssssssss
E. D ISCUSSION ON O PTIMAL G OAL
In the previous goal-oriented RL literature (Andrychowicz et al., 2017; Ren et al., 2019), what kind of generated goals is helpful for the agent to efﬁciently learn a well-performed policy is one of the key questions to be answered.
The basic idea of goal-oriented RL architecture is to generate goals to decompose the complex task into several goal-oriented tasks.
In this paper, we analyze our generated goals from two perspectives, namely reachability and curriculum.
Reachability .
The ﬁrst property required in the optimal goal is that the generated goal is guaranteed to be reachable for the agent.
To this end, in this paper, the candidate goal set is constrained into the visited states.
In other words, the goal generated in the episode e must be visited before the episode e .
Therefore, we can guarantee that the generated goal is reachable.
Curriculum .
The second property is the curriculum, which means that our optimal goals are required to approach the terminal state during the exploration.
If the Q -value of each state is well estimated, our goal generation under the supervision of forward-looking planning at the next episode will focus on the potential highest value states in the future, which is actually the terminal state when the agent has the full observation of states.
E. D ISCUSSION ON G ROUP D IVISION
Motivation .
The intuitive motivation behind the group division is very natural.
Proposition 1 implies that exploration on the state-transition graph G et at timestep t in episode e without any constraint may lead to explosion of graph and inefﬁciency of exploration.
Therefore, the agent is expected to do exploration within a limited domain.
Considering that G et is always changing and the number of nodes ( i.e. , | S G et | ) keeps increasing, it is non-trivial for the agent to learn to select state as the goal for further exploration.
Hence, we ﬁrst restrict the exploration within the boundary of state-transition graph ∂ G et according to Proposition 2.
We then consider partitioning ∂ G et into several groups.
We set the last visited state s last as the original point because s last is likely to be close to the target state and reachable for current policy.
As introduced in Section 3, we propose to extend groups from s last following two possible perspectives, namely neighbor and uncertain nodes.
Complexity . Let d ∂ G et denote the maximum degree of states in ∂ G et , and | S ∂ G et | denote the number of states in ∂ G et .
Note that ∂ G et is always a directed fully connected graph.
If we want to ﬁnd the nhop neighbors of s last , we need to iteratively go through related nodes’ neighborhood.
In other words, the computation complexity should be O ( d n∂ G et ) .
Hence, the complexity to construct C 1 , . .
, C N by extending from neighbor nodes is O ( d 1 ∂ G et )
+ O ( d 2 ∂ G et )
+ · · · + O ( d N − 1 ∂ G et )
=
O ( d N − 1 ∂ G et ) .
If we want to ﬁnd nodes whose uncertainty equals n , we need to go through the graph once.
In this case, the computation complexity should be O ( | S ∂ G et | ) .
Hence, the complexity to construct C 1 , . .
, C N extending from uncertain nodes is O ( | S ∂ G et | ) .
F E XPERIMENTS F. E NVIRONMENT C ONFIGURATION
Maze .
As shown in Figure 11(a), in the maze environment, a point in a 2 D U -maze aims to reach a goal represented by a red point.
The size of maze is 15 × 15 , the state space and is in this 2 D
U -maze, and the goal is uniformly generated on the segment from (0 , 0) to (15 . 0 , 15 . 0) .
The action space is from ( − 1 . 0 , − 1 . 0) to (1 . 0 , 1 . 0) , which represents the movement in x and y directions.
AntMaze .
As shown in Figure 11(b), in the AntMaze environment, an ant is put in a U -maze, and the size of the maze is 12 × 12 .
The ant is put on a random location on the segment from ( − 2 . 0 , − 2 . 0)
to Under review as a conference paper at ICLR Then we check whether the exploration provided by the goals generated by HGG can result in better policy training performance.
As shown in Figure 3, we compare the vanilla HER, HER with EnergyBased Prioritization (HER+EBP), HGG, HGG+EBP.
It is worth noting that since EBP is designed for the Bellman equation updates, it is complementary to our HGG-based exploration approach.
Among the eight environments, HGG substantially outperforms HER on four and has comparable performance on the other four, which are either too simple or too difﬁcult.
When combined with EBP,
HGG+EBP achieves the best performance on six environments that are eligible.
Performance on tasks with obstacle In a more difﬁcult task, crafted metric may be more suitable than ` 2 -distance used in Eq.
As shown in Figure 4, we created an environment based on
FetchPush with a rigid obstacle.
The object and the goal are uniformly generated in the green and the red segments respectively.
The brown block is a static wall which cannot be moved.
In addition to ` 2 , we also construct a distance metric based on the graph distance of a mesh grid on the plane, the blue line is a successful trajectory in such hand-craft distance measure.
A more detailed description is deferred to Appendix B.3.
Intuitively speaking, this crafted distance should be better than ` 2 due to the existence of the obstacle.
Experimental results suggest that such a crafted distance metric provides better guidance for goal generation and training, and signiﬁcantly improves sample efﬁciency over ` 2 distance.
It would be a future direction to investigate ways to obtain or learn a good metric.
5.2 Comparison with Explicit Curriculum Learning
Since our method can be seen as an explicit curriculum learning for exploration, where we generate hindsight goals as intermediate task distribution, we also compare our method with another recently proposed curriculum learning method for
RL.
Florensa et al.
(2018) leverages Least-Squares GAN (Mao et al., 2018b) to mimic the set called Goals of Intermediate
Difﬁcult as exploration goal generator.
Speciﬁcally, in our task settings, we deﬁne a goal set GOID ( ⇡ ) = { g : ↵  f ( ⇡, g )  1  ↵ } , where f ( ⇡, g ) represents the average success rate in a small region closed by goal g .
To sample from GOID , we implement an oracle goal generator based on rejection sampling, which could uniformly sample goals from GOID ( ⇡ ) .
Result in Figure 5 indicates that our Hindsight Goal Generation substantially outperforms HER even with GOID from the oracle generator.
Note that this experiment is run on a environment with ﬁxed initial state due to the limitation of Florensa et al.
The choice of ↵ is also suggested by Florensa et al.
5.3 Ablation Studies on Hyperparameter Selection
In this section, we set up a set of ablation tests on several hyper-parameters used in the Hindsight Goal Generation algorithm.
Lipschitz L : The selection of Lipschitz constant is task dependent, since it iss related with scale of value function and goal distance.
For the robotics tasks tested in this paper, we ﬁnd that it is easier to set L by ﬁrst divided it with the upper bound of the distance between any two ﬁnal goals in a environment.
We test a few choices of L on several environments and ﬁnd that it is very easy to ﬁnd a range of L that works well and shows robustness for all the environments tested in this section.
We show the learning curves on FetchPush with different L .
It appears that the performance of HGG is reasonable as long as L is not too small.
For all tasks we tested in the comparisons, we set L = 5 . 0 .
Distance weight c : Parameter c deﬁnes the trade-off between the initial state similarity and the goal similarity.
Larger c encourages our algorithm to choose hindsight goals that has closer initial state. A
Proof of Theorem 1 In this section we provide the proof of Theorem 1.
Theorem 1.
Assuming that the generalizability condition (Eq.
(4) ) holds for two distributions ( s, g ) ⇠ T and ( s 0 , g 0 )
⇠ T 0 , we have V ⇡ ( T 0 )
V ⇡ ( T )  L · D ( T , T 0 ) . (6) where D ( · , · ) is the Wasserstein distance based on d ( · , · ) D ( T (1) , T (2) )
= inf µ 2 ( T (1) , T (2) )
⇣ E µ [ d (( s 0(1) , g (1) ) , ( s 0(2) , g (2) ))] ⌘ where ( T (1) , T (2) ) denotes the collection of all joint distribution µ ( s 0(1) , g (1) , s 0(2) , g (2) ) whose marginal probabilities are T (1) , T (2) , respectively.
Proof.
By Eq.
(4), for any quadruple ( s, g, s 0 , g 0 ) , we have V ⇡ ( s 0 , g 0 )
V ⇡ ( s, g )  L · d (( s, g ) , ( s 0 , g 0 )) . For any µ 2 ( T , T 0 ) , we sample ( s, g, s 0 , g 0 )
⇠ µ and take the expectation on both sides of Eq.
(10), and get
V ⇡ ( T 0 )  V ⇡ ( T )  L · E µ [ d (( s, g ) , ( s 0 , g 0 ))] .
(11) Since Eq.
(11) holds for any µ 2 ( T , T 0 ) , we have V ⇡ ( T 0 )  V ⇡ ( T )  L · inf µ 2 ( T , T 0 )
( E µ [ d (( s, g ) , ( s 0 , g 0 ))]) =
V ⇡ ( T )  L · D ( T , T 0 ) . B Experiment Settings B. Modiﬁed Environments
Fetch Environments:
• FetchPush-v1: Let the origin (0 , 0 , 0) denote the projection of gripper’s initial coordinate on the table.
The object is uniformly generated on the segment (  0 .
15 ,  0 .
15 , 0)  (0 . 15 ,  0 .
, and the goal is uniformly generated on the segment (  0 . 15 , 0 . 15 , 0)  (0 . 15 , 0 . 15 , 0) .
• FetchPickAndPlace-v1: Let the origin (0 , 0 , 0) denote the projection of gripper’s initial coordinate on the table.
The object is uniformly generated on the segment (  0 .
15 ,  0 .
15 , 0)  (0 . 15 ,  0 .
, and the goal is uniformly generated on the segment (  0 . 15 , 0 . 15 , 0 . 45)  (0 . 15 , 0 . 15 , 0 . 45) .
• FetchSlide-v1: Let the origin (0 , 0 , 0) denote the projection of gripper’s initial coordinate on the table.
The object is uniformly generated on the segment (  0 .
05 ,  0 .
1 , 0)  (  0 .
05 , 0 . 1 , 0) , and the goal is uniformly generated on the segment (0 .
55 ,  0 .
15 , 0)  (0 . 55 , 0 . 15 , 0) .
[Figure]
Published as a conference paper at ICLR 2020 memorized states like episodic memory, and maintain a graph on top of these states based on state transitions at the same time.
Then we develop an efﬁcient reverse-trajectory propagation strategy to allow the values of new experiences to rapidly propagate to all memory items through the graph.
Finally, we use the fast-adjusted non-parametric high values in associative memory as early guid- ance for a parametric RL agent so that it can rapidly latch on states that previously yield high returns instead of waiting for many slow gradient updates.
A riences.
To illustrate the superiority of the associative memory in reinforcement learning, consider a robot exploring in a maze to seek out the apple (at place G), as shown in Figure 1.
It collects two trajectory experiences starting from place A and B, respectively.
All the states of tra- jectory A (the top blue dash line) receive no reward because the agent terminates at a state with a non-zero
re-
ward (at place C), while in trajectory B (the bottom blue dash line) the ﬁnal non-zero reward of catching an ap- ple (at place G) back-propagates through the whole path.
Episodic memory keeps a high value at the intersection of two trajectories (the door) when taking actions toward lower-right corner while recording zero values at the other states in trajectory A.
If an episodic memory based robot starts from place A, it will wander around A because there are no positive values indicating the way to goal.
Thus based on the episodic memory, the robot may eventually take a policy like the green line after multiple attempts.
However, if the robot adopts associative memory, the high value in the door collected from trajectory B will be further propagated to the start point A and thus the robot can correctly take the red-line policy.
To some extent, our associative memory is equivalent to automatic augmentation of counterfactual combinatorial trajectories in memory.
Thus, our framework signiﬁcantly improves the sampleefﬁciency of reinforcement learning.
Comparisons with state-of-the-art episodic reinforcement learning methods show that ERLAM is substantially more sample efﬁcient for general settings of reinforcement learning.
In addition, our associative memory can be used as a plug-and-play module and is complementary to other reinforcement learning models, which opens the avenue for further researches on associative memory based reinforcement learning.
2 B ACKGROUND
In the framework of reinforcement learning (Sutton & Barto, 1998), an agent learns a policy to maximize its cumulative rewards by exploring in a Markov Decision Processes (MDP) environment.
An MDP is deﬁned by a tuple ( S , A , P, R ,  ) , where S is a ﬁnite set of states, A is a ﬁnite set of actions available to the agent, P :
S ⇥ A ⇥ S !
R deﬁnes the transition probability distribution,
R is the reward function, and  2 (0 , 1] is the discount factor.
At each time step t , the agent observes state s t 2 S , selects an action a t 2 A according to its policy ⇡ : S !
A , and receives a scalar reward r t .
In the setting of ﬁnite horizon, the accumulated discounted return is calculated as,
R t = P Tk =
k r t
+ k where T is the episode length and goal of the agent is to maximize the expected return for each state s t .
The state-action value function Q ⇡ ( s, a ) =
E [ R t | s t =
s, a ] is the expected return for executing action a on state s and following policy ⇡ afterwards.
DQN (Mnih et al., 2015) parameterizes this action-value function by deep neural networks Q ✓ ( s, a ) and use Q-learning (Watkins & Dayan, 1992) to learn it to rank which action at is best to take in each state s t at time step t .
The parameters of the value network ✓ are optimized by minimizing the L 2 difference between the networks output
Q ✓ ( s, a ) and the Q-learning target y t =
r t +  max a Q ˆ ✓ ( s t +1 , a t ) , where ˆ ✓ are parameters of a target network that is a older version of the value network and updated periodically.
DQN uses an off-policy learning strategy, which samples ( s t , a t , r t , s t +1 ) tuple from a replay buffer for training.
(10 . 0 , 10 . 0) , and the goal is uniformly generated on the segment from ( − 2 . 0 , − 2 . 0) to (10 . 0 , 10 . 0) .
The state of ant is 30 -dimension, including its positions and velocities.
FetchPush .
As shown in Figure 11(c), in the fetch environment, the agent is trained to fetch an object from the initial position (rectangle depicted in green) to a distant position (rectangle depicted in red). Let the origin (0 , 0 , 0) denote the projection of the gripper’s initial coordinate on the table. The object is uniformly generated on the segment from ( − 0 . 0 , − 0 . 0 , 0) to (8 , 8 , 0) , and the goal is uniformly generated on the segment from ( − 0 . 0 , − 0 . 0 , 0) to (8 , 8 , 0) .
FetchPush with Obstacle .
As shown in Figure 11(d), in the fetch with obstacle environment, we create an environment based on FetchPush with a rigid obstacle, where the brown block is a static wall that can’t be moved.
The object is uniformly generated on the segment from ( − 0 . 0 , − 0 . 0 , 0) to (8 , 8 , 0) , and the goal is uniformly generated on the segment from ( − 0 . 0 , − 0 . 0 , 0) to (8 , 8 , 0) .
AntMaze with Obstacle .
This environment is an extended version of AntMaze, where a 1 × 1 rigid obstacle is put in U-maze. F. E VALUATION D ETAILS
• All curves presented in this paper are plotted from 10 runs with random task initialization and seeds. • The shaded region indicates 60% population around the median. • All curves are plotted using the same hyper-parameters (except the ablation section).
• Following (Andrychowicz et al., 2017), an episode is considered successful if (cid:107) g − s object (cid:107) 2 ≤ δ g is achieved, where s object is the object position at the end of the episode.
δ g is the threshold. • The max timestep for each episode is set as 200 for training and 500 for tests. • The average success rate using in the curve is estimated by 10 2 samples. F. H YPER -P ARAMETERS
Almost all hyper-parameters using DDPG (Lillicrap et al., 2015) and HER Andrychowicz et al.
(2017) are kept the same as benchmark results, except these:
• Number of MPI workers: 1 ; • Actor and critic networks: 3 layers with 256 units and ReLU activation; • Adam optimizer with 5 × 10 − 4 learning rate; • Polyak-averaging coefﬁcient: 0 . 98 ; • Action l 2 -norm penalty coefﬁcient: 0 . 5 ; • Batch size: 256 ; • Probability of random actions: 0 . 2 ; • Scale of additive Gaussian noise: 0 . 2 ; • Probability of HER experience replay: 0 . 8 ; • Number of batches to replay after collecting one trajectory: 50 .
Hyper-parameters in goal generation: • Adam optimizer with 1 × 10 − 3 learning rate; • K of K -bins discretization: 20 ; • Number of groups to depart the graph: 3 .
F.
By sample efﬁciency, we show the comparisons according to the number of states visited and actions taken. In other words, given the ﬁx number of episodes, more unique states visited and actions taken usually denote the efﬁciency of exploration. We report the log ﬁles of GSRL and HER in Maze environment here at 10 , 50 , 100 episodes, which contain the number of visited nodes and actions taken
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]