TRL: D ISCRIMINATIVE H INTS FOR S CALABLE R EVERSE C URRICULUM L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep reinforcement learning algorithms have proven successful in a vast variety of domains.
However, tasks with sparse rewards remain challenging when the state space is large.
Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the ﬁnal goal is accomplished.
In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum.
This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning.
We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efﬁciently and robustly than prior approaches in tasks with long time horizons and large state space.
In addition, we demonstrate that using an optional checkpoint scheme with very small quantity of key states, our approach can solve difﬁcult robot manipulation challenges directly from perception and sparse rewards.
I NTRODUCTION
Reinforcement learning (RL) aims to learn the optimal policy of a certain task by maximizing the cumulative reward acquired from the environment.
Recently, deep reinforcement learning has enjoyed great successes in a variety of domains with short-term and dense reward feedback (Mnih et al., 2016)
(e.g. Atari games, TORCS).
However, many real world problems are inherently based on sparse, binary rewards, where the agent needs to travel through a large number of states before a success (or failure) signal can be received.
For instance, in the grasping task of a robotic arm, the most accurate task reward would be a binary reward only if the agent successfully picked up the the object or not.
This kind of sparse-reward, goal-oriented tasks are considered the most difﬁcult challenges in reinforcement learning.
That is, environment only provides a ﬁnal success signal when the agent has accomplished the whole task.
The search space of these tasks may exponentially expand as state chain extends, which adds to the difﬁculty of reaching the ﬁnal goal with conventional reinforcement learning algorithms.
There have been multiple lines of approaches for tacking such sparse reward problems, including the ideas based on intrinsic motivation (Stadie et al., 2015; Bellemare et al., 2016; Houthooft et al., 2016), hierarchical reinforcement learning (Florensa et al., 2017a;
D.Kulkarni & Karthik Narasimhan, 2016) and curriculum learning (Karpathy & Van
De Panne, 2012; Sukhbaatar et al., 2017).
Recently, a particularly promising approach is the idea of reverse curriculum generation learning (Florensa et al., 2017b).
By reversing the traditional reinforcement learning process and gradually expanding the set of start states from the goal to the starting point, this method does achieve substantial progress in some simulation tasks.
However, the newly sampled start set mixes old and new states, which leads to inefﬁciency in training, as the agent spends a large amount of time reviewing rather than learning new skills.
This problem aggravates in large state space tasks where the expansion could be really slow, and the storing of history data is also impracticable due to the memory limitation.
To overcome the drawbacks mentioned above, we propose a Tendency Reinforcement
Learning (TRL) architecture based on a key idea: the agent should be targeted at learning what it cannot yet do, while its former experience can provide tendency hints to stabilize and accelerate the training process.
To achieve this, a dense binary reward (the “tendency”) is designed to provide a rough judgment on the success tendency of each state, and our agent learns from hybrid rewards that combine the ﬁnal goal reward with the hints outputted by tendency reward.
We deﬁne a set of start states as a phase, and continue extending it from the ﬁnal goal, until it reaches where the task starts.
In each phase, a tendency reward provides hints for the agent and inﬂuences its training.
Between phases, the tendency reward is updated using experience of the agent.
This mutual promotion between the agent and tendency reward contributes to a dramatic improvement in training efﬁciency.
In this paper, we introduce three novel techniques: First, we propose a hybrid reward design that combines the ﬁnal reward with tendency hints.
Secondly, with the introduction of a phase administrator, the agent is able to focus on exploiting new states, which increases the training efﬁciency.
Lastly, we develop an optional checkpoint scheme, that is, our framework is also compatible with additional checkpoints with no strict accuracy requirement, as long as they are effective at reducing irrelevant search space.
The major contribution of this work is that we present a reliable tendency reinforcement learning method that is capable of training agents to solve large state space tasks with only ﬁnal reward.
In our experiments, we apply and test the model in diverse domains, including a maze game, Super Mario Bros, a robotic grasping task, a crane conveyance task and a pick-and-place challenge, where it proves more efﬁcient and competitive than the prior works.
R ELATED WORK
Recently, great progress has been made in RL by approximating the policy and value functions through neural networks (Mnih et al., 2013; 2015; Schulman et al., 2015; Mnih et al., 2016).
However, these RL methods often suffer from extremely slow training progress or fail to learn in sparse reward tasks.
There have been several lines of approaches to tackling such problems, namely curriculum learning, hierarchical RL, intrinsic motivation and demonstration-augmented RL, each with its own constraints and assumptions.
Curriculum learning works by deconstructing a difﬁcult, long-horizon learning task into a sequence of easy to hard tasks (Karpathy & Van De Panne, 2012).
While classic curriculum is often deﬁned manually, recently in RL a number of work has shown promising results automated curriculum generation (Pinto et al., 2017;
Sukhbaatar et al., 2017; Wulfe & Hershey; Florensa et al., 2017b).
Our work is closest to this line of work.
In particular, reverse curriculum generation (Florensa et al., 2017a) has proposed a very similar curriculum generation through exploration backwards from the goal states, and has shown promising results in solving a variety of sparse reward problems.
However, the critical difference of our approach from theirs is that we propose the use of the tendency reward trained discriminatively to summarize the past histories of successful and unsuccessful states, allowing our algorithm not to store the whole history of starting sets and further provide a dense shaping reward.
Empirically our design choice is found to scale to larger state tasks with longer planning horizon better than this prior approach.
Hierarchical RL and intrinsic motivations are two other approaches for tackling the sparse reward problems.
Hierarchical RL (Sutton et al., 1999; D.Kulkarni & Karthik Narasimhan, 2016) learns to plan using a hierarchical of low-level policies, often directly provided to the high-level planner, to drastically reduce the search dimension of the problem, while intrinsic motivation often provides additional dense rewards in terms of novelty (Stadie et al., 2015; Houthooft et al., 2016; Bellemare et al., 2016) or some notion of empowerment (Mohamed & Rezende, 2015) to improve the exploration under no rewards.
However, both approaches assume forward-based exploration to solve the sparse reward task, which becomes increasingly difﬁcult as the dimensionality and the time horizon increase, unless strong structured prior is provided in HRL.
Our method, with the similar assumptions as in reverse curriculum learning, allows much more efﬁcient learning and exploration.
Lastly, there are a number of work that utilizes demonstrations to tackle such problems.
Inverse reinforcement learning infers the underlying reward function from expert demonstrations(Ng et al., 2000; Abbeel & Ng, 2004; Ziebart et al., 2008), and recent work has used such reward function as potential-based shaping reward to accelerate learning (Brys et al., 2015; Harutyunyan et al., 2015).
Another line of work directly made use of demonstrations to curriculize the training process (Popov et al., 2017).
These methods, however, make strong assumptions about the quality of demonstrations they receive.
By contrast, our optional checkpoint method only requires a small number of key states instead of several full demonstration trajectories, and is robust to noises.
Speciﬁcally, for a challenging robotic manipulation task from perception, we only required 5 key states to solve the task.
P RELIMINARIES AND A SSUMPTIONS
The deﬁnitions and symbols used in our model are listed in Appendix A
3. P RELIMINARIES
We revisit the conventional reinforcement learning setting where an agent interacts with the environment Env sequentially.
At each time step t , the agent receives an observation
O t and chooses an action a t from a set of action space A according to its policy π .
Then the environment will give the agent the next state s t +1 and a reward r t .
The goal of the agent is to maximize the expected return from each state s t through accumulating the reward by
R t = (cid:80) ∞ k =0 γ k r t + k .
3. A SSUMPTIONS FOR TENDENCY REINFORCEMENT LEARNING
The tendency reinforcement learning model is based on two assumptions that hold true for a variety of practical learning problems, especially for those learned in simulation.
Assumption 1.
When resetting, the agent can start from any state s ∈ P i .
Assumption 2.
The interaction between agent and environment follows a Markov chain based on deterministic transformations, that is, P
( s i +
| s i , a i ) ∈ { 0 , 1 } .
The ﬁrst assumption agrees with learning in simulation, and can be satisﬁed in many realworld problems (e.g. robotic manipulations).
If we could constrain the agent’s start states to a valid phase, the agent would be able to train from these recommended start states, which proves to increase learning efﬁciency (Kakade & Langford, 2002).
In case of longterm tasks where the phases extend with time, the number of possible start states might end up unnecessarily large.
Under such circumstances, additional checkpoint related methods can also be applied to assist training, with which the assumption is fully compatible.
The second assumption indicates that the environment runs in a non-stochastic manner and always follows some ﬁxed rules (e.g. laws of classical physics).
Because of this, it makes sense to determine tendency rewards with a binary classiﬁer, since there exists an absolute tendency for each state given a certain policy.
In deterministic environments, irreversible processes may cause the phase extension to stuck on some absorbing states.
However, our checkpoint scheme is able to solve this problem, details will be discussed in
Appendix E.
T HE MODEL
Na¨ıve applications of conventional RL solver tend to fail in goal-oriented tasks with large state space.
This kind of tasks are common in real world and can lead to an immense search
[Figure]
space.
Our Tendency RL is able to tackle this problem by reducing aimless searching with the application of 1) a novel tendency reward design that utilizes the agent’s former experience; 2) a phase administrator that assigns customized training phases to the agent; 3) a powerful checkpoint scheme that shrinks search space and accelerates learning (optional).
Our Insight Our key idea is to let the agent utilize previous experiences through a tendency classiﬁer trained on history trajectories in a supervised manner (Sec. 4.2).
We deﬁne each phase as a set of start states.
In training, phases are automatically generated by a phase administrator (Sec. 4.1) and gradually shift away from the ﬁnal goal.
In each phase, the agent learns from rewards given by the tendency classiﬁer, and trains the classiﬁer with its new experiences.
Algorithm 1: TendencyRL input : Goal state s g , starting phase P 1 , task start phase P task .
output: Trained policy π .
Phase index i ← 1 ; if P 1 = ∅ then
P 1 ← SampleNearby ( s g , K ) ;
Train policy π beginning from P 1 with environmental reward;
Collect trajectories & train Tendency Reward T ( · ) (Sec. 4.2); while P i ∩ P task = ∅ do
P i +1 ← PhaseAdministrator
( P i , N i ) (Alg. 2);
Train policy π beginning from P i +1 with hybird Reward (Eq.
Update Tendency Reward; i ← i + 1 ; return π ;
The input P 1 (cid:54) = ∅ only when the optional checkpoint scheme is used, which will be discussed in Sec.
4. P HASE A DMINISTRATOR
Each time when the agent has made some progress in current phase (being able to accomplish the task starting from 80% of states in P current ), the next phase P current +1 would be generated by the phase administrator based on the agent’s performance in current phase.
Normally, during the i th phase extension process, we let the agent start from P i and randomly take x i steps.
The newly reached states would then be included into P
i +1 if the agent’s success rate on that state is between a lower-bound α and an upper-bound β .
The introduction of α and β helps to limit the new states to a proper difﬁculty ( α is 0.1 and β is 0.9 in our experiments), and thus the agent is able to concentrate on exploiting new states and learn faster.
In each phase extension, a reasonable extension step size x i is required.
Generally, as x i becomes larger, less phases are used to train the agent.
However, the increased search space actually causes more training iterations.
Therefore, in order to strike a balance, the phase administrator adjusts x i according to the number of training iterations N i in P i .
This adjustment is achieved with a sigmoid function shown in equation (1).
[Equation]
where ¯ N is the average of N 1 , N 2 , . .
, N i − 1 .
The procedure of the phase administrator is sketched in Algorithm 2.
Algorithm 2: Phase Administrator input : Current phase P i , number of training iterations in the i th phase N i .
output: The next phase P
i +1 .
Calculate x i from N i with Equation (1); P i +1 ← ∅ ; while |P i +1 | <
K do s old ← Sample ( P i ) ; s new ← RandomWalk ( s old , x i ) ; if α < SuccessRate ( s new ) <
β and s new / ∈ P
i +1 then
P i +1 ← P
i +1 ∪ { s new } ;
4. T ENDENCY R EWARD
The tendency reward in our model is a binary classiﬁer (e.g. SVM, NN) that takes an observation O as input and outputs a judgment on whether that state leads to the ﬁnal success.
We name it tendency reward mainly because it provides slight guiding hints to speed up the training process in each phase and those large state space tasks thus become trainable.
In training, each time when the agent steps onto a state, it would refer to the tendency reward for a judgment.
If the state is evaluated as “favorable”, the agent would receive a positive tendency hint as a reward.
Otherwise, a negative reward would be issued.
To prevent the agent from stalling at a favorable state for unlimited tendency rewards, we deﬁned that such revisited states do not provide any reward, and applied a Gaussian kernel (Appendix A.3) to detect repeated states.
The binary tendency classiﬁer trains on both success and failure trajectories and we use logistic loss as the classiﬁer loss.
The deﬁnition of the hybrid reward is:
[Equation]
where γ is a scale factor ( 10 − 3 in our experiments) based on the following hypothesis: a positive reward received from the environment should be more important than the output of T ( · ) .
Since the tendency hints are very subtle compared to the ﬁnal reward, the agent’s desire to achieve the ﬁnal goal won’t be interfered with.
4. O PTIONAL CHECKPOINT SCHEME FOR EXTREME LARGE STATE SPACE TASK
In tasks with a vast state space, it is usually more difﬁcult for P to reach any state s nearby
P task .
Those domains are difﬁcult for both prior approaches and tendency RL.
In those cases, we show that by providing just a few checkpoints , our tendency RL algorithm can take advantage of them to shrink the search space.
Ideally, the checkpoints should be precise and indicate important states for the task, and in practice ﬁne hand-engineering is often involved.
However, with our tendency RL method, some rough key states would be sufﬁcient.
The agent still extends its phases with a regular phase administrator, while after each extension, it additionally checks whether the newly generated phase has covered some states nearby a certain key state.
If so, the next phase would be directly sampled from that key state rather than generated by the phase administrator.
In this way, the search space can decrease remarkably.
In experiments, our model also manifest robustness when encountered with misleading key states.
Given n as the number of checkpoints.
The agent ﬁrst extends new phases from s g with the tendency RL algorithm described in Sec. 4.
The extension would be ceased once the newly generated phase covers some states nearby s k i .
The criteria of judgment is based on a Gaussian kernel approximation function (Appendix A.3) as is described in Eq.
[Equation]
If J is larger than 30% of K , P current is believed to cover some states around s k i .
Then the next phase is sampled nearby s k i rather than extended from the current phase.
After that, we apply the tendency RL algorithm again and extend new phases from it.
The procedure will be repeated until P task is reached.
Algorithm 3: TendencyRL with optional checkpoints input : Goal state s g , key states s k 1 , . . .
, s k n , task start phase P task .
output: Trained policy π for the extreme large space task.
for j from 1 to n do
Q
j ← SampleNearby ( s k j , K ) ;
R ← n ∪ j =1 {Q j } ; while P current ∩ P task = ∅ do π ← TendencyRL (cid:18) s g , P current , ∪ Q∈R Q (cid:19)
(Alg. 1); for Q in R do if Q ∩ P current (cid:54) = ∅ then
R ← R \ {Q} ;
P current ← Q ; break ; return π ;
Note that our principle is distinct from conventional RL demonstration methods in that the agent is not eager to go after the checkpoints and does not get any additional reward from them.
In fact, the small number of checkpoints are only used to shrink the search space, and our method also shows robustness to imperfect checkpoint settings in experiment where we deliberately mix misleading states into the checkpoints, and found that the policy learned by the agent doesn’t get affected (Sec. 5.3).
E XPERIMENT
To evaluate our model, we design ﬁve experiments 1 with different levels of difﬁculties.
The details are elaborated in Appendix B and C.
We ﬁrst test the efﬁciency and validity of our model.
After testifying the correctness of the overall model, we visualize the distribution of tendency hints and the starting sets of phases.
Then we focus on the optional checkpoint scheme, verify its compatibility to our method and prove our method’s robustness to checkpoints of different qualities.
Finally,
[Figure]
we apply our model to long-term manipulation challenges with large state spaces to show our advantages.
5. V ALIDITY AND EFFICIENCY OF TRL
The training of the 40 × 40 maze task takes around 35000 training steps (1 hour andminutes) with 6 threads on GPU.
Then the agent can reach the exit from the start point of the maze with a 9 × 9 observation around it.
To investigate the inﬂuence of tendency reward on training, we conduct an experiment comparing the training effect under three conditions: using tendency reward without history phases, using history phases without tendency reward and using neither tendency reward nor history phases.
This is not only an ablation experiment but also a comparison with prior work, since the second condition is basically the reverse curriculum generation learning method (Florensa et al., 2017b).
The result shows that our model performs well (Fig. 3).
5. I NSPECTING THE LEARNING RESULTS
The agent takes nearly 40000 training steps (around 21 hours) with 8 threads on GPU to learn how to complete the level 1-1 of
Super Mario Bros.
The policy learned by the agent is shown in Fig.
In order to ﬁnd out the inﬂuence of the tendency reward, we traverse the whole states of some parts of the game and show both of the positive and negative tendency hints (Fig. 4).
As we can see in these ﬁgures, our agent has acquired the ability to ﬁnd enemies nearby and dodge or even kill them by jumping up to their head.
Besides, it also recognizes the danger of the pits and can choose the right time to jump over them.
These ﬁgures clearly show the guiding effect of the tendency reward, which does a great help for the agent when it is far away from the ﬁnal goal.
To ﬁgure out the phases generated by the phase administrator, we visualize the phases in
Super Mario Bros, and we ﬁnd such phases can effectively lead the agent to constantly learn
[Figure] [Figure]
new states farther from the reverse curriculum method, our method neither needs to keep track of starting sets nor keeps restarting from all phases at each iteration, which is the key to scaling to long horizon tasks with large state space.
5. C OMPATIBILITY AND ROBUSTNESS TO OPTIONAL CHECKPOINTS
After testing and inspecting our normal tendency RL model, we then focus on the optional checkpoint scheme.
We enlarge the maze to 100 × 100 also with a 9 × 9 observation around the agent.
With 8 key states as the checkpoints, the training takes around 50000 training steps (7 hours and 14 minutes).
We test our agent’s robustness to imperfect checkpoints, related results can refer to Appendix D.3.
We also compare our model with two methods using demonstrations: data-efﬁcient RL (Popov et al., 2017) and potential-based reward shaping (PBRS) from demonstration (Brys et al., 2015).
The former resets the agent randomly to someplace on an expert trajectory while the latter uses an artiﬁcial potential reward shaped based on demonstrations.
We test them with two kinds of checkpoints / demonstrations: “reasonable” and “unreasonable”.
As is shown in Fig.
6, data-efﬁcient RL may ﬂuctuate constantly when facing unreasonable demonstration.
PBRS from demonstration requires a lot of human involvement to achieve high performance.
Different from these two methods, our model only uses 10 evenly distributed key states on the trajectory as checkpoints and performs as good as PBRS from demonstration.
5. S UCCESS IN LARGE STATE SPACE PROBLEMS
The following experiments mainly focus on illustrating the practicality of our model on solving difﬁcult, real-world robotic manipulation problems with minimal assumptions.
When confronted with long-term challenges with large state space, to avoid wasting time on unnecessary states, our checkpoint scheme is applied to shrink the search space.
As is described in Sec. 4.3, several roughly designed key states are sufﬁcient to guide the agent through the large search space properly.
In the three experiments, we only add less thancheckpoints to assist training (5 in robotic arm, 5 in conveyance challenge of the crane and 14 in pick-and-place task).
The training process follows Alg. 3.
[Figure]
5.4. G RASPING T ASK
The grasping task on robotic arm simulation is a typical large state space problem, where the gripper can move anywhere within the range of the arm.
In our design, a camera is fastened to the forepart of the gripper and provides a 64 × 64 depth image as the agent’s observation.
We intuitively design 5 key states and train the whole task for 33360 iterations (4 hours 19 minutes) with 12 threads on GPU, and it turns out that our model can solve the task reliably.
To further investigate the relationship between training efﬁciency and scale of checkpoints, we conduct the same experiment with 5, 8, 11, 13 key points respectively, as is shown in
Appendix D.4.
5.4. C ONVEYANCE C HALLENGE
The conveyance challenge is also a 3D manipulation task with a larger state space.
The agent should pick up the ball, take it to correct basket, and drop it right into that basket to receive a positive reward.
The camera is also ﬁxed on the gripper that produces 64 ×
RGB images as input.
Our model is trained for 99240 iterations (5 hours 2 minutes) withthreads on GPU.
Since this task involves multiple operations and has a large state space, we roughly add ﬁve key states to help the agent learn different skills, as is shown in Appendix D.5.
With the intention of ﬁnding out the distribution of “favorable” and “adverse” states given by the tendency reward, we make the agent travel through the whole environment at an appropriate height under two conditions: 1) the gripper jaw is open with the target ball ﬁxed on the center of plane; 2) the ball is grasped by the gripper and moves with it.
Then we record the tendency reward at each position (Fig. 7).
As expected, areas close to the target basket are assigned the highest tendency hints, which is quite natural as these states are reached before of after a successful drop.
On the other hand, locations above those incorrect baskets tend to incur negative tendency hints.
More details can be found in Appendix D.5.
5.4. R ECURRENT P ICK - AND - PLACE C HALLENGE
Of all the three experiments in robotic manipulation, this challenge has the largest state space due to endless combinations of the agent’s location and orientation and enormous possible positions of plates.
This time, the agent observes through a panorama camera that provides 160 × 32 ﬁrst-person point of view.
The whole task can be divided into several identical pick-and-place operations that resemble the conveyance challenge.
We provide 14 intuitively designed checkpoints to accelerate training (Appendix D.6).
However, these checkpoints are not quite sufﬁcient in such a huge state space, and the agent shows confusion from time to time.
Our phase administrator plays a signiﬁcant role in phase arrangement, and actually makes up for the lack of expert data.
Since the agent performs well within each pick-and-place operation, new phases are more concentrated on the intervals between operations.
D ISCUSSION AND FUTURE WORK
We develop a tendency reinforcement learning algorithm that resolves complicated goaloriented tasks, and evaluate it with multiple experiments based on raw perceptions.
Our method consists of two core components: a binary tendency classiﬁer that provides dense hints on whether a state leads to success and a phase administrator that generates a reverse curriculum.
In ﬁve distinct environments, we ﬁnd that tendency RL method is efﬁcient and stable enough to converge on a reliable policy within hours even facing intractable problems, and does not require history data which is impossible to acquire in large space domains.
One limitation of our current approach is that our model requires the environment to be capable of resetting to some arbitrary states, which is also a limitation of the reverse curriculum generation method (Florensa et al., 2017b).
A promising future direction is to additionally train reset controllers that automatically reset the system to the initial state following each episode, together with a corresponding forward controller (Han et al., 2015).
In future work, our goal is to apply the tendency RL method to real world robotic manipulation tasks with perceptions.
Furthermore, we plan to extend the binary tendency classiﬁer into a predictor that outputs the expected success rate on current state, so that our model can also be applied to stochastic environments.
Additionally, we intend to release a subset of our code and the trained models to facilitate further research and comparisons
A D EFINITIONS AND FORMULAE A. T ERM DEFINITIONS
• Final goal : States that indicate the task is successfully ﬁnished (e
g. the exit in maze).
• Phase : A set of start states at a certain time period.
• Start state :
A state where the agent starts for the ﬁnal goal in a certain phase.
• Success state :
The states leading to the ﬁnal goal by current policy.
• Task start phase : States where the entire task begins, which is the ﬁnal aim of phases extensions (e.g. the start point in maze).
• Checkpoint :
A small number of key states artiﬁcially designed to shrink the search space in large space tasks.
A. S YMBOL DEFINITIONS
The symbols used in the tendency reinforcement learning model are described in table 1.
[Table]
A. G AUSSIAN K ERNEL
A Gaussian kernel (Eq.
(4)) is used in our experiments to detect repeated states.
The most d recent frames are used in the kernel, as is shown in Eq.
[Equation]
We assume that s now is a revisited state if ρ is larger than 0.9.
The parameter d is 20% of the max steps of each environment and σ = 1 & δ = 0 are constants in our experiments
[Table]
B E NVIRONMENT D ETAILS B. M AZE
Our ﬁrst environment is a maze task.
The size of the whole maze is 100 × 100 .
There is only one exit (ﬁnal goal) in the middle of the bottom line ( [100 , 50] ) and one starting point at [0 , 50] .
The agent is required to ﬁnd its own way to move from the starting point to exit with a limit observation covering a 9 × 9 area around it.
At each time-step, the agent has four actions, namely walk north , walk south , walk east and walk west , expressed as (N, S, E,
W), and transitions are made deterministically to an adjacent cell, unless there is a block, in which case no movement occurs.
The environment only gives 1 point reward when the agent has accomplished the ﬁnial goal.
On other conditions, the reward remains 0 .
B. S UPER M ARIO B ROS
The second environment is level 1-1 of Super Mario Bros (Brockman et al., 2016; koltafrickenfer, 2017; asfdfdfd, 2012).
The agent should complete the level from start point via a 16 × 13 observation, which indicates the nearby situation.
When the agent reaches its destination, it would receive 1 point reward, otherwise it receives 0 point.
The agent hasactions: jump , right , left , jump to right and jump to left , expressed as (J, R, L, JR, JL).
Mario could not achieve any reward by killing enemies or collecting coins.
B. G RASPING T ASK
The third environment is a grasping task on a robotic arm simulation in MuJoCo.
The end effector of the 6-DOF robotic arm is a two-ﬁnger gripper with a camera attached between the ﬁngers, which produces 64 × 64 overlooking depth images as the agent’s observations.
In order to manipulate in a more intuitive fashion, poses of joints have been mapped into 3D coordinates, enabling the gripper to move in 6 different directions.
There is also a special grasping action that closes the gripper jaws, by which the target cube can be grasped.
The agent is initialized randomly above a plane, and 1 point reward would be received if and only if the target cube placed on the plane is captured by the gripper.
B. C ONVEYANCE C HALLENGE
The fourth task is a conveyance challenge of a crane simulated by the physics engine MuJoCo.
The agent can interact with a ball placed on a plane using a 3-ﬁnger gripper connected to the crane, with 8 actions available (6 for free movements and 2 for gripper manipulations).
Surrounding the ball from a distance are some target baskets, among which one basket is randomly selected and colored green.
The agent is able to observe through a camera ﬁxed on the gripper that produces 64 × 64 RGB images, and only by dropping the ball into the green basket would the agent receive 1 point reward.
B. R ECURRENT P ICK - AND - PLACE
The ﬁfth task involves pick-and-place operations simulated in MuJoCo, which requires the agent to stack some plates into a neat pile.
This time, the agent observes through a panorama camera that provides 160 × 32 ﬁrst-person point of view.
Fixed before the camera is a electromagnet capable of attracting one plate at a time, which is used for plate transfer.
Available actions include turning left / right , stepping forward / backward , charging / releasing the electromagnet .
The ﬁnal goal is to gather and pile up all the plates into a straight stack, which is the only approach for the agent to obtain 1 point reward.
C N ETWORK A RCHITECTURES C. M AZE
The convolution neural network of the policy network has a ﬁrst layer with 12 4 × 4 ﬁlters of stride 1, followed by a layer with 24 3 × 3 ﬁlters of stride 1, followed by a layer with2 × 2 ﬁlters of stride 1.
Then we add a LSTM layer with 384 units which followed by a fully connected layer with 96 hidden units.
C. S UPER M ARIO B ROS
The convolution neural network of the policy network has a ﬁrst layer with 48 4 × 4 ﬁlters of stride 1, followed by a layer with 48 3 × 3 ﬁlters of stride 1, followed by a layer with2 × 2 ﬁlters of stride 1.
Then we add a LSTM layer with 960 units which followed by a fully connected layer with 240 hidden units.
C. E XPERIMENTS WITH MUJOCO E NGINE
The convolution neural network of the policy network has a ﬁrst layer with 12 4 × 4 ﬁlters of stride 1, followed by a layer with 24 3 × 3 ﬁlters of stride 1, followed by a layer with2 × 2 ﬁlters of stride 1, followed by a layer with 48 2 × 2 ﬁlters of stride 1.
Then we add a LSTM layer with 1536 units which followed by a fully connected layer with 96 hidden units.
Especially, in the pick-and-place challenge, there is an additional convolution layer with 96 2 × 2 ﬁlters of stride 1, and the LSTM layer contains 480 units.
D S UPPLEMENTARY MATERIALS FOR EXPERIMENTS
D.
S UPER M ARIO B ROS DETAILS
D.1.
T RAINING R ESULT
The training result of Super Mario Bros is shown in Fig.
D.1. C OMPARISON WITH CURIOSITY - DRIVEN METHOD
In Super Mario Bros, we compare our model with another one trained based on curiositydriven (Pathak et al., 2017).
We let the agents randomly move several steps at the beginning before running the models, then compare the completion degree 2 of two models.
With the environment only providing a reward when Mario reaches the ﬁnal goal, we ﬁnd that our method obtains absolute advantage (Table 2).
We also use the model trained in level 1-1 to run the level 1-2 and 1-3 of the game.
Since the latter two levels are not trained and require new skills that the ﬁrst level does not include (Fig. 9), both of the methods may suffer more failures.
[Table]
[Figure]
[Table]
[Figure] [Figure]
D. C OMPARISON WITH MAXIMUM ENTROPY IRL
We try the maximum entropy IRL (Ziebart et al., 2008) in the 100 × 100 maze task with more than two days’ training and about 44G virtual memory used, but it still failed to form a feasible reward function.
Since there are 10,000 states in the maze, the traditional IRL methods seem to suffer from the bottleneck of computing ability.
D. R OBUSTNESS TO IMPERFECT CHECKPOINTS
To test our model’s robustness to imperfect checkpoint settings, instead of arranging the checkpoints onto the optimal trajectory, we set some of them onto a byway.
We choosespecial phases sampled from the misleading checkpoints and show the policy the agent has learned in these phases.
As is shown in Fig.
10, in the sixth phase, the agent refuses to detour from the left side and ﬁnds a shortcut direct to the exit.
A possible explanation is that our tendency reward only provide slight guiding hints which do not bother the agent’s exploration for new policies.
[Figure]
D. T HE INFLUENCE OF CHECKPOINT SCALE ON TRAINING EFFICIENCY
We conduct the grasping experiment with 5, 8, 11, 13 key points respectively.
The inﬂuence of checkpoint scale on training efﬁciency can refer to Fig.
. D. C ONVEYANCE C HALLENGE DETAILS
An interesting observation in conveyance challenge is, the areas with positive tendency hints are more concentrated and the boundary between “favorable” and “adverse” states is more explicit when the gripper jaw is open.
This is because the states where the agent is not holding the ball are often related to grasping and dropping, which require more accuracy than just transferring the ball.
This experiment also proves that our agent has superb robustness to imperfect checkpoint settings.
In our previous assumptions, the agent should rise to a proper height where all baskets were completely in view, before moving towards the correct basket (Fig. 12).
However, it turns out that the agent learns to determine the target basket location at a lower height, where only portions of the baskets are in view.
[Figure] [Figure]
[Figure]
E O THER D ISCUSSIONS
The regular reverse curriculum algorithm usually fails if there exists an irreversible process in the system.
The irreversible process is deﬁned as:
[Equation]
In such cases, the states s and s (cid:48) are not connected, and an agent starting from s (cid:48) will never reach s since that probability is 0 .
We deﬁne a absorbing state s a as a state that satisﬁes
[Equation]
To be more generalized, we deﬁne a set S a of states to be a absorbing set if it satisﬁes
[Equation]
[Equation]
Consider a phase extension progress where P i +1 is generated from P i , if a large portion of states in P i +1 belong to some absorbing sets, it would be hard to for the new phase to include elements not in these absorbing sets.
Therefore, the training is likely to be contained within these sets and no actual progress could be made since the phase extension makes no sense.
However, with additional checkpoints, this limitation could be avoided even with irreversible processes and absorbing sets.
The mechanism is described as follows: When we sample states nearby a checkpoint that is not in any absorbing set, the sampled states might happen to belong to some absorbing set.
Although phase extensions are constrained within the absorbing sets, the generated phase might also cover some states sampled nearby a checkpoint.
Thus, according to the phase administrator algorithm (Alg. 2), that checkpoint could still be reached.