TRL: D ISCRIMINATIVE H INTS FOR S CALABLE R EVERSE C URRICULUM L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep reinforcement learning algorithms have proven successful in a variety of domains.
However, tasks with sparse rewards remain challenging when the state space is large.
Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the ﬁnal goal is accomplished.
In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum.
This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning.
We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario
Bros and show that our method performs more efﬁciently and robustly than prior approaches in tasks with long time horizons and large state space.
In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difﬁcult robot manipulation challenges directly from perception and sparse rewards.
I NTRODUCTION
Reinforcement learning (RL) aims to learn the optimal policy of a certain task by maximizing the cumulative reward acquired from the environment.
Recently, deep reinforcement learning has enjoyed great successes in many domains with short-term and dense reward feedback (Mnih et al., 2016)
(e.g. Atari games, TORCS).
However, many real world problems are inherently based on sparse, binary rewards, where the agent needs to travel through a large number of states before a success (or failure) signal can be received.
For instance, in the grasping task of a robotic arm, the most accurate task reward would be a binary reward only if the agent successfully picked up the the object or not.
This kind of goal-oriented tasks with sparse rewards are considered the most difﬁcult challenges in reinforcement learning.
That is, the environment only provides a ﬁnal success signal when the agent has accomplished the whole task.
The search space of these tasks may exponentially expand as state chain extends, which adds to the difﬁculty of reaching the ﬁnal goal with conventional reinforcement learning algorithms.
There have been multiple lines of approaches for tackling such sparse reward problems, including the ideas based on intrinsic motivation (Stadie et al., 2015; Bellemare et al., 2016; Houthooft et al., 2016), hierarchical reinforcement learning (Florensa et al., 2017a; Kulkarni et al., 2016), curriculum learning (Karpathy et al., 2012; Sukhbaatar et al., 2017) and experience-based off-policy learning (Andrychowicz et al., 2017).
Recently, a particularly promising approach is the idea of reverse curriculum generation (Florensa et al., 2017b).
By reversing the traditional reinforcement learning process and gradually expanding the set of start states from the goal to the starting point, this method does achieve substantial progress in some simulation tasks.
However, the newly sampled start set mixes old and new states, which leads to inefﬁciency in training, as the agent spends a large amount of time reviewing rather than learning new skills.
This problem aggravates in large state space tasks where the expansion could be really slow, and the storing of history data is also impracticable due to the memory limitation.
To solve goal-oriented tasks and avoid the drawbacks mentioned above, we propose a Tendency Reinforcement Learning (TRL) architecture which shapes a reward function with former experience and use it to stabilize and accelerate training.
To achieve this, a discriminative reward (the “tendency”) is shaped to output a judgment on the success tendency of each state, and the reward function for the agent is hybrid, which combines the ﬁnal goal reward with the tendency hints.
We deﬁne a set of start states as a phase, and continue extending it from the ﬁnal goal, until it reaches where the task starts.
In each phase, the tendency reward inﬂuences the agent’s training.
After training each phase, the tendency reward is updated using collected experience.
This mutual promotion contributes to a dramatic improvement in training efﬁciency and no longer requires keeping all history data.
In this paper, we introduce three novel techniques: First, we propose a hybrid reward design that combines the ﬁnal reward with tendency hints.
Secondly, with the introduction of a phase administrator, the curriculum for each phase is automated, which increases the efﬁciency.
Lastly, we develop an optional keyframe scheme, that is, our framework is also compatible with additional keyframes with no strict accuracy requirement, as long as they are effective at reducing irrelevant search space.
The major contribution of this work is that we present a reliable tendency reinforcement learning method that is capable of training agents to solve large state space tasks with only the ﬁnal reward, taking raw pixels as perception.
In our experiments, we apply and test the model in diverse domains, including a maze game, Super Mario Bros and 3 simulated tasks (a robotic grasping task, a crane conveyance task and a pick-and-place challenge), where it proves more efﬁcient and competitive than the prior works.
R ELATED WORK
Recently, great progress has been made in RL by approximating the policy and value functions through neural networks (Mnih et al., 2013; 2015; Schulman et al., 2015; Mnih et al., 2016).
However, these RL methods often suffer from extremely slow training progress or fail to learn in sparse reward tasks.
There have been several lines of approaches to tackling such problems, namely curriculum learning, hierarchical RL, intrinsic motivation and demonstration-augmented RL, each with its own constraints and assumptions.
Curriculum learning works by deconstructing a difﬁcult, long-horizon learning task into a sequence of easy to hard tasks (Karpathy et al., 2012).
While classic curriculum is often deﬁned manually, recently in RL a number of work has shown promising results automated curriculum generation (Pinto et al., 2017;
Sukhbaatar et al., 2017; Wulfe & Hershey, 2017; Florensa et al., 2017b).
Our work is closest to this line of work.
In particular, reverse curriculum generation (Florensa et al., 2017a) has proposed a very similar curriculum generation through exploration backwards from the goal states, and has shown promising results in solving a variety of sparse reward problems.
However, the critical difference of our approach from theirs is that we propose the use of the tendency reward trained discriminatively to summarize the past histories of successful and unsuccessful states, allowing our algorithm not to store the whole history of starting sets and further provide a dense shaping reward.
Empirically our design choice is found to scale to larger state tasks with longer planning horizon better than this prior approach.
Hierarchical RL and intrinsic motivations are two other approaches for tackling the sparse reward problems.
Hierarchical RL (Sutton et al., 1999; Kulkarni et al., 2016) learns to plan using a hierarchical of low-level policies, often directly provided to the high-level planner, to drastically reduce the search dimension of the problem, while intrinsic motivation often provides additional dense rewards in terms of novelty (Stadie et al., 2015; Houthooft et al., 2016; Bellemare et al., 2016) or some notion of empowerment (Mohamed & Rezende, 2015) to improve the exploration under no rewards.
However, both approaches assume forward-based exploration to solve the sparse reward task, which becomes increasingly difﬁcult as the dimensionality and the time horizon increase, unless strong structured prior is provided in HRL.
Our method, with the similar assumptions as in reverse curriculum learning, allows much more efﬁcient learning and exploration.
Lastly, there are a number of work that utilizes demonstrations to tackle such problems.
Inverse reinforcement learning infers the underlying reward function from expert demonstrations(Ng et al., 2000; Abbeel & Ng, 2004; Ziebart et al., 2008), and recent work has used such reward function as potential-based shaping reward to accelerate learning (Brys et al., 2015;
Harutyunyan et al., 2015).
Another line of work uses demonstrations that consist of keyframes to do Keyframe-based Learning (Akgun et al., 2012).
These methods, however, make strong assumptions about the quality of demonstrations they receive.
By contrast, our optional keyframe scheme only requires a small number of key states instead of several full demonstration trajectories [Table] and is robust to noises.
Speciﬁcally, for a challenging robotic manipulation task from perception, we only required 5 key states to solve the task [Table]
P RELIMINARIES AND A SSUMPTIONS
[Table]
Algorithm
[Table]
output
[Table] policy π [Table] Phase index i ←
[Table]
if P 1 = ∅ then
P 1 ← SampleNearby ( s g , K ) ;
Train policy π beginning from P 1 with environmental reward;
Collect trajectories
[Table]
Reward T ( · ) ; while P i ∩ P task = ∅ do
P i +1 ← PhaseAdministrator
( P i , N i ) ;
Train policy π beginning from P i + with hybird Reward;
Update Tendency Reward;
3. P RELIMINARIES
We revisit the conventional reinforcement learning setting where an agent interacts with the environment Env sequentially.
At each time step t , the agent receives an observation O t and chooses an action a t from a set of action space A according to its policy π .
Then the environment will give the agent the next state s t +1 and a reward r t .
The goal of the agent is to maximize the expected return from each state s t through accumulating the reward by R t = (cid:80) ∞ k =0 γ k r t + k .
3. A SSUMPTIONS
The tendency reinforcement learning model is based on three assumptions that follows (Florensa et al., 2017b) and hold true for a variety of practical learning problems.
Assumption 1.
We can arbitrarily reset the agent into any start state s ∈ P i .
Assumption 2.
At least one state s g is provided such that s g ∈ S g . Assumption 3. The Markov Chain induced by taking uniformly sampled random actions has a communicating class 1 including all start states in P task and the given goal state s g .
[-( s i +
 | s i , a i ) ∈ { 0 ,-]Assumption 1 can be satisﬁed in many robotic manipulation problems by storing the start states in P i with low-dimensional data (e.g. angle of joints and velocity of motors).
Acquiring these data and sending commands to let robotic apply these settings are widely provided by standard robotic APIs, and such functions are similar in game engines.
This assumption enables the agent to train from recommended start states, which proves to increase learning efﬁciency (Kakade & Langford, 2002).
Combining Assumption 1 with Assumption 2, we are able to reset the state to s g .
Assumptionensures that there exists a trajectory formed from a sequence of actions from any start state in P task to s g , and vice versa.
This assumption can be satisﬁed by many robotic manipulation problems and games, as long as there are no major irreversibilities in the system. Symbols used in our paper are listed in table 1 for reference.
T HE MODEL
Na¨ıve applications of conventional RL tend to fail in goal-oriented tasks with a large state space and sparse rewards.
However, this kind of tasks are common in real world.
TRL is able to tackle them by reducing aimless searching with the application of 1) a discriminative tendency reward shaping that utilizes the agent’s former experience; 2) a phase administrator that assigns customized curriculum to the agent; 3) a keyframe scheme that shrinks search space and accelerates learning in large state space multistage tasks (optional).
Our key idea is to let the agent utilize previous experiences through a tendency classiﬁer trained on history trajectories in a supervised manner (Sec. 4.1).
We deﬁne each phase as a set of start states.
In training, curriculum are automatically adjusted by a phase administrator (Sec. 4.2).
In each phase, the agent learns from rewards produced by the tendency classiﬁer, and trains the classiﬁer with new experiences.
The overview of training framework is shown in Fig.
1 and the corresponding algorithm is sketched in Alg.
[Figure]
4. T ENDENCY R EWARD
The tendency reward in our model is a binary classiﬁer (e.g. SVM, NN) that takes an observation O as input and outputs a judgment on whether that state leads to the ﬁnal success. We name it tendency reward mainly because it provides slight guiding hints to speed up the training process in each phase and those large state space tasks thus become trainable. In training, each time when the agent steps onto a state, it would refer to the tendency reward for a judgment.
If the state is evaluated as “favorable”, the agent would receive a positive tendency hint as a reward. Otherwise, a negative reward would be issued. To prevent the agent from stalling at a favorable state for unlimited tendency rewards, we deﬁned that such revisited states do not provide any reward, and applied a Gaussian kernel to detect them. The kernel is used with d most recent frames in a bid to detect repeated states, as is shown in Eq.
[Equation]
We assume that s now is a revisited state if ρ is larger than 0.9. The parameter d is 20% of the max steps of each environment and σ = 1 & δ = 0 are constants in our experiments. The binary tendency classiﬁer trains on both success and failure trajectories and we use logistic loss as the classiﬁer loss.
The hybrid reward is deﬁned in Eq.
[Equation]
where λ is a scale factor ( 10 − 3 in our experiments) based on the following hypothesis: a positive reward received from the environment should be more important than the output of T ( · ) .
Since the tendency hints are very subtle compared to the ﬁnal reward, the agent’s desire to achieve the ﬁnal goal won’t be interfered with.
4. P HASE A DMINISTRATOR
Each time when the agent has made some progress in P current (being able to accomplish the task starting from 80% of start set), the x current for the generation of P current +1 will be adjusted by the phase administrator based on the agent’s performance.
During the i th phase generation process, the agent start from P i and randomly take x i steps.
The newly reached states would then be included into P
i +1 if the agent’s success rate on that state is between a lower-bound α and an upper-bound β , which guarantee the new states to a proper difﬁculty ( α is 0.1 and β is 0.9 in our experiments).
In each phase generation, a reasonable x i is required.
Large x i means less phases, while it might increase training iterations in P
i +1 .
Therefore, in order to strike a balance, the phase administrator adjusts x i according to the number of training iterations N i in P i .
This adjustment is achieved with a sigmoid function shown in equation (4).
[Equation]
where ¯ N is the average of N 1 , N 2 , . .
, N i − 1 .
The pseudo code of the phase administrator is sketched in Algorithm 2 in Appendix.
B.
4. O PTIONAL KEYFRAME SCHEME FOR EXTREME LARGE STATE SPACE TASK
In tasks with a vast state space, it is usually more difﬁcult for P to reach any state s nearby P task .
Those domains are difﬁcult for both prior approaches and tendency RL.
In those cases, we show that by providing just a few keyframes , our tendency RL algorithm can take advantage of them to shrink the search space.
Ideally, the keyframes should be precise and indicate important states for the task, and in practice ﬁne hand-engineering is often involved.
However, for TRL, several rough key states would be sufﬁcient.
The agent still extends its phases with a regularly, while after each extension, it additionally checks whether the newly generated phase has covered some states nearby a certain key state (by a Gaussian kernel approximation function as is described in Eq.
If so, the next phase would be directly sampled from that key state rather than generated form P current .
In this way, the search space can decrease remarkably.
[Equation]
If J is larger than 30% of K , P current is believed to cover some states around s k i .
Then the next phase is sampled nearby s k i rather than extended from P current .
Pseudo code is shown in Alg.
3 in
Appendix.
B.
Note that our principle is distinct from LfD ( learn from demonstration ) methods in that the agent is not eager to follow the keyframes and doesn’t get additional reward from them.
In fact, these small number of keyframes are only used to shrink the search space, and our method also shows robustness to imperfect keyframe settings in experiment where we deliberately mix misleading states into the keyframes, and found that the inﬂuence is not signiﬁcant (Sec. 5.3).
E XPERIMENT
To evaluate our model, we design ﬁve experiments 2 with different levels of difﬁculties (Fig. 2).
In all of these experiments, we apply A3C (Mnih et al., 2016) as the RL solver.
The details of environments and network are elaborated in Appendix. C and D.
[Figure]
We ﬁrst test the efﬁciency and validity of our model.
After testifying the correctness of the overall model, we visualize the distribution of tendency hints and the starting sets of phases.
Then we focus on the optional keyframe scheme, verify its compatibility to our method and prove our method’s robustness to keyframes of different qualities.
Finally, we apply our model to long-term manipulation challenges with large state spaces to show our advantages.
5. V ALIDITY AND EFFICIENCY OF TRL
The training of the 40 × 40 maze task takes around 35000 training steps (1 hour and 40 minutes) with 6 threads on GPU.
Then the agent can reach the exit from the start point of the maze with a 9 × 9 observation around it.
To investigate the inﬂuence of tendency reward on training, we conduct an experiment comparing the training effect under three conditions: using tendency reward without history phases, using history phases without tendency reward and using neither tendency reward nor history phases.
This is not only an ablation experiment but also a comparison with prior work, since the second condition is basically the reverse curriculum generation learning method (Florensa et al., 2017b).
The result shows that our model performs well (Fig. 3).
We also show the improvement when using phase administrator which let the curriculum automated (Fig. 4).
5. I NSPECTING THE LEARNING RESULTS
The agent takes nearly 40000 training steps (around 21 hours) with 8 threads on GPU to learn how to complete the level 1-1 of
Super Mario Bros.
The policy learned by the agent is shown in Appendix.
E.1.2. In order to ﬁnd out the inﬂuence of the tendency reward, we traverse the whole states of some parts of the game and show both of the positive and negative tendency hints (Fig. 5).
[Figure]
As we can see in these ﬁgures, our agent has acquired the ability to ﬁnd enemies nearby and dodge or even kill them by jumping up to their heads.
Besides, it also recognizes the danger of the pits and can choose the right time to jump over them.
These ﬁgures show the guiding effect of the tendency reward, which does a great help for the agent when it is far away from the ﬁnal goal.
We also visualize parts of the phases in Mario, which effectively lead the agent to constantly learn new states farther from goal state (Fig. 5).
5. C OMPATIBILITY AND ROBUSTNESS TO OPTIONAL KEYFRAMES
We then enlarge the Maze to 100 × 100 also with a 9 × 9 observation.
With 8 key states as the keyframes, the training takes around 50000 training steps (7 hours and 14 minutes).
We also test our agent’s robustness to imperfect keyframes, related results can be referred in Appendix. E.3.
We also compare our model with two LfD methods: data-efﬁcient RL (Popov et al., 2017) and potential-based reward shaping (PBRS) from demonstration (Brys et al., 2015).
The former resets the agent randomly to someplace on an expert trajectory while the latter uses an artiﬁcial potential reward shaped based on demonstrations.
We test them with both reasonable and unreasonable keyframes / demonstrations.
As is shown in Fig.
6, data-efﬁcient RL is quite sensitive to the demonstration data and ﬂuctuates constantly facing unreasonable demonstrations.
PBRS from demonstration can achieve similar performance to TRL , but requires a lot more human efforts of hand-engineering its reward function (otherwise it just wouldn’t work).
Different from these methods, TRL does not need expert demonstration (10 labeled keyframes are sufﬁcient) and requires no further human elaboration. Also, the training of TRL is the most stable of three all.
5. S UCCESS IN LARGE STATE SPACE PROBLEMS
The following experiments mainly focus on illustrating the practicality of our model on solving difﬁcult, real-world robotic manipulation problems with minimal assumptions.
When confronted with long-term challenges with large state space, to avoid wasting time on unnecessary states, our keyframe scheme is applied to shrink the search space.
In the three experiments, we add less than 15 keyframes to assist training (5 in robotic arm, 5 in conveyance challenge of the crane and 14 in pick-and-place task), and the results are quite satisfying.
5.4. G RASPING T ASK
The grasping task on robotic arm simulation is a typical large state space problem, where the gripper can move anywhere within the range of the arm.
In our design, a camera is fastened to the forepart of the gripper and provides a 64 × 64 depth image as the agent’s observation.
We intuitively design [Figure] 5 key states and train the whole task for 33360 iterations (4 hours 19 minutes) with 12 threads on
GPU, and it turns out that our model can solve the task reliably.
To further investigate the relationship between training efﬁciency and scale of keyframes, we conduct the same experiment with 5, 8, 11, 13, 15, 18 key points respectively, as is shown in Appendix. E.4.
5.4. C ONVEYANCE C HALLENGE
The conveyance challenge is also a 3D manipulation task with a larger state space.
The agent should pick up the ball, take it to correct basket, and drop it right into that basket to receive a positive reward.
The camera is also ﬁxed on the gripper that produces 64 × 64 RGB images as input.
Our model is trained for 99240 iterations (5 hours 2 minutes) with 12 threads on GPU.
Since this task involves multiple operations and has a large state space, we roughly add ﬁve key states to help the agent learn different skills, as is shown in Appendix. E.5.
With the intention of ﬁnding out the distribution of “favorable” and “adverse” states given by the tendency reward, we make the agent travel through the whole environment at an appropriate height under two conditions: 1) the gripper jaw is open with the target ball ﬁxed on the center of plane; 2) the ball is grasped by the gripper and moves with it.
Then we record the tendency reward at each position (Fig. 7).
As expected, areas close to the target basket are assigned the highest tendency hints, which is quite natural as these states are reached before of after a successful drop.
On the other hand, locations above those incorrect baskets tend to incur negative tendency hints.
More details can be found in Appendix. E.5.
5.4. R ECURRENT P ICK - AND - PLACE C HALLENGE
Of all the three experiments in robotic manipulation, this challenge has the largest state space due to endless combinations of the agent’s location and orientation and enormous possible positions of plates.
This time, the agent observes through a panorama camera that provides 160 × 32 ﬁrst-person point of view.
The whole task can be divided into several identical pick-and-place operations that resemble the conveyance challenge.
We provide 14 intuitively designed keyframes to accelerate training (Appendix. E.6).
However, these keyframes are not quite sufﬁcient in such a huge state space, and the agent shows confusion from time to time.
Our phase administrator plays a signiﬁcant role in phase arrangement, and actually makes up for the lack of expert data.
Since the agent performs well within each pick-and-place operation, new phases are more concentrated on the intervals between operations.
D ISCUSSION AND FUTURE WORK
We develop a tendency reinforcement learning algorithm that resolves complicated goal-oriented tasks, and evaluate it with multiple experiments based on raw perceptions.
Our method consists of two core components: a binary tendency classiﬁer that provides dense hints on whether a state leads to success and a phase administrator that generates a reverse curriculum.
In ﬁve distinct environments, we ﬁnd that tendency RL method is efﬁcient and stable enough to converge on a reliable policy within hours even facing intractable problems, and does not require history data which is impossible to acquire in large space domains.
One limitation of our current approach is that our model requires the environment to be capable of resetting to some arbitrary states, which is also a limitation of the reverse curriculum generation method (Florensa et al., 2017b).
A promising future direction is to additionally train reset controllers that automatically reset the system to the initial state following each episode, together with a corresponding forward controller (Han et al., 2015).
In future work, our goal is to apply the tendency RL method to real world robotic manipulation tasks with perceptions.
Furthermore, we plan to extend the binary tendency classiﬁer into a predictor that outputs the expected success rate on current state, so that our model can also be applied to stochastic environments.
Additionally, we intend to release a subset of our code and the trained models to facilitate further research and comparisons.
A D EFINITIONS AND FORMULAE A. T ERM DEFINITIONS
• Final goal : States that indicate the task is successfully ﬁnished (e.g. the exit in maze).
• Phase : A set of start states at a certain time period.
• Start state :
A state where the agent starts for the ﬁnal goal in a certain phase.
• Success state :
The states leading to the ﬁnal goal by current policy.
• Task start phase : States where the entire task begins, which is the ﬁnal aim of phases extensions (e.g. the start point in maze).
• Keyframe : A small number of key states artiﬁcially designed to shrink the search space in large space tasks.
B O THER A LGORITHMS B. P HASE A DMINISTRATOR Algorithm
2: Phase Administrator Calculate x i from N i with Equation (4);
P i +1 ← ∅ ; while |P i +1 | <
K do s old ← Sample ( P i ) ; s new ← RandomWalk ( s old , x i ) ; if α < SuccessRate ( s new ) < β and s new / ∈ P i +1 then
P i +1 ← P i +1 ∪ { s new } ; B. T ENDENCY
WITH for do Q
j ← SampleNearby ( s k R ← n ∪ j =1 {Q j } ; while P current ∩ P task = ∅ do π ← TendencyRL (cid:18) s g , P current , ∪ Q∈R Q (cid:19) ; for Q in R do if Q P current (cid:54)
= then R P current ; break ; return π ;
C E NVIRONMENT D ETAILS C. M AZE
Our ﬁrst environment is a maze task.
The size of the whole maze is 100 × 100 .
There is only one exit (ﬁnal goal) in the middle of the bottom line ( [100 , 50] ) and one starting point at [0 , 50] .
The agent is required to ﬁnd its own way to move from the starting point to exit with a limit observation covering a 9 × 9 area around it.
At each time-step, the agent has four actions, namely walk north , walk south , walk east and walk west , expressed as (N, S, E, W), and transitions are made deterministically to an adjacent cell, unless there is a block, in which case no movement occurs.
The environment only gives 1 point reward when the agent has accomplished the ﬁnial goal.
On other conditions, the reward remains 0 .
C. S UPER M ARIO B ROS
The second environment is level 1-1 of Super Mario Bros (Brockman et al., 2016; koltafrickenfer, 2017; asfdfdfd, 2012).
The agent should complete the level from start point via a 16 × 13 observation, which indicates the nearby situation.
When the agent reaches its destination, it would receive 1 point reward, otherwise it receives 0 point.
The agent has 5 actions left L JR, JL).
Mario could not achieve any reward by killing enemies or collecting coins.
C. G RASPING T ASK
The third environment is a grasping task on a robotic arm simulation in MuJoCo.
The end effector of the 6-DOF robotic arm is a two-ﬁnger gripper with a camera attached between the ﬁngers, which produces 64 × 64 overlooking depth images as the agent’s observations.
In order to manipulate in a more intuitive fashion, poses of joints have been mapped into 3D coordinates, enabling the gripper to move in 6 different directions.
There is also a special grasping action that closes the gripper jaws, by which the target cube can be grasped.
The agent is initialized randomly above a plane, andpoint reward would be received if and only if the target cube placed on the plane is captured by the gripper.
C. C ONVEYANCE C HALLENGE
The fourth task is a conveyance challenge of a crane simulated by the physics engine MuJoCo.
The agent can interact with a ball placed on a plane using a 3-ﬁnger gripper connected to the crane, with 8 actions available (6 for free movements and 2 for gripper manipulations).
Surrounding the ball from a distance are some target baskets, among which one basket is randomly selected and colored green.
The agent is able to observe through a camera ﬁxed on the gripper that produces 64 ×
RGB images, and only by dropping the ball into the green basket would the agent receive 1 point reward.
C. R ECURRENT P ICK - AND - PLACE
The ﬁfth task involves pick-and-place operations simulated in MuJoCo, which requires the agent to stack some plates into a neat pile.
This time, the agent observes through a panorama camera that provides 160 × 32 ﬁrst-person point of view.
Fixed before the camera is a electromagnet capable of attracting one plate at a time, which is used for plate transfer.
Available actions include turning left / right , stepping forward / backward , charging / releasing the electromagnet .
The ﬁnal goal is to gather and pile up all the plates into a straight stack, which is the only approach for the agent to obtain 1 point reward.
D N ETWORK A RCHITECTURES D. M AZE
The convolution neural network of the policy network has a ﬁrst layer with 12 4 × 4 ﬁlters of stride 1, followed by a layer with 24 3 × 3 ﬁlters of stride 1, followed by a layer with 48 2 × 2 ﬁlters of stride 1.
Then we add a LSTM layer with 384 units which followed by a fully connected layer with 96 hidden units.
D. S UPER M ARIO B ROS
The convolution neural network of the policy network has a ﬁrst layer with 48 4 × 4 ﬁlters of stride 1, followed by a layer with 48 3 × 3 ﬁlters of stride 1, followed by a layer with 96 2 × 2 ﬁlters of stride 1.
Then we add a LSTM layer with 960 units which followed by a fully connected layer with 240 hidden units.
D. E XPERIMENTS WITH MUJOCO E NGINE
The convolution neural network of the policy network has a ﬁrst layer with 12 4 × 4 ﬁlters of stride 1, followed by a layer with 24 3 × 3 ﬁlters of stride 1, followed by a layer with 48 2 × 2 ﬁlters of stride 1, followed by a layer with 48 2 × 2 ﬁlters of stride 1.
Then we add a LSTM layer with 1536 units which followed by a fully connected layer with 96 hidden units.
Especially, in the pick-and-place challenge, there is an additional convolution layer with 96 2 × 2 ﬁlters of stride 1, and the LSTM layer contains 480 units.
E S UPPLEMENTARY MATERIALS FOR EXPERIMENTS E. S UPER M ARIO B ROS DETAILS E.1. T RAINING R ESULT
The training result of Super Mario Bros is shown in Fig.
E.1. C OMPARISON WITH CURIOSITY - DRIVEN METHOD
In Super Mario Bros, we compare our model with another one trained based on curiosity-driven (Pathak et al., 2017).
We let the agents randomly move several steps at the beginning before run- ning the models, then compare the completion degree 3 of two models.
With the environment only providing a reward when Mario reaches the ﬁnal goal, we ﬁnd that our method obtains absolute advantage (Table 2).
We also use the model trained in level 1-1 to run the level 1-2 and 1-3 of the game.
Since the latter two levels are not trained and require new skills that the ﬁrst level does not include (Fig. 9), both of the methods may suffer more failures.
[Table]
[Figure]
[Table]
[Figure]
E. C OMPARISON WITH MAXIMUM ENTROPY IRL
We try the maximum entropy IRL (Ziebart et al., 2008) in the 100 × 100 maze task with more than two days’ training and about 44G virtual memory used, but it still failed to form a feasible reward function.
Since there are 10,000 states in the maze, the traditional IRL methods seem to suffer from the bottleneck of computing ability
. E. R OBUSTNESS TO IMPERFECT KEYFRAMES
To test our model’s robustness to imperfect keyframe settings, instead of arranging the keyframes onto the optimal trajectory, we set some of them onto a byway.
We choose 8 special phases sampled from the misleading keyframes and show the policy the agent has learned in these phases.
As is shown in Fig.
10, in the sixth phase, the agent refuses to detour from the left side and ﬁnds a shortcut direct to the exit.
A possible explanation is that our tendency reward only provide slight guiding hints which do not bother the agent’s exploration for new policies. [Figure]
E. T HE INFLUENCE OF KEYFRAME SCALE ON TRAINING EFFICIENCY
We conduct the grasping experiment with 5, 8, 11, 13, 15 and 18 keyframes respectively.
The inﬂuence of keyframe scale on training efﬁciency can refer to Fig. [Figure]
E. C ONVEYANCE C HALLENGE DETAILS
An interesting observation in conveyance challenge is, the areas with positive tendency hints are more concentrated and the boundary between “favorable” and “adverse” states is more explicit when the gripper jaw is open.
This is because the states where the agent is not holding the ball are often related to grasping and dropping, which require more accuracy than just transferring the ball.
This experiment also proves that our agent has superb robustness to imperfect keyframe settings.
In our previous assumptions, the agent should rise to a proper height where all baskets were completely in view, before moving towards the correct basket (Fig. 12).
However, it turns out that the agent learns to determine the target basket location at a lower height, where only portions of the baskets are in view.
[Figure]
E. P ICK - AND - PLACE DETAILS
[Figure]
F O THER D ISCUSSIONS
The regular reverse curriculum algorithm usually fails if there exists an irreversible process in the system.
The irreversible process is deﬁned as:
[Equation]
In such cases, the states s and s (cid:48) are not connected, and an agent starting from s (cid:48) will never reach s since that probability is 0 .
We deﬁne a absorbing state s a as a state that satisﬁes
[Equation]
To be more generalized, we deﬁne a set S a of states to be a absorbing set if it satisﬁes
[Equation]
Consider a phase extension progress where P i +1 is generated from P i , if a large portion of states in P i +1 belong to some absorbing sets, it would be hard to for the new phase to include elements not in these absorbing sets.
Therefore, the training is likely to be contained within these sets and no actual progress could be made since the phase extension makes no sense.
However, with additional keyframes, this limitation could be avoided even with irreversible processes and absorbing sets.
The mechanism is described as follows: When we sample states nearby a keyframe that is not in any absorbing set, the sampled states might happen to belong to some absorbing set.
Although phase extensions are constrained within the absorbing sets, the generated phase might also cover some states sampled nearby a keyframe.
Thus, according to the phase administrator algorithm (Alg. 2), that keyframe could still be reached.