P ROGRESSIVE R EINFORCEMENT L EARNING WITH D ISTILLATION FOR M ULTI -S KILLED M OTION C ONTROL
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems, including agents that can move with skill and agility through their environment.
An open problem in this setting is that of developing good strategies for integrating or merging policies for multiple skills, where each individual skill is a specialist in a speciﬁc skill and its associated state distribution.
We extend policy distillation methods to the continuous action setting and leverage this technique to combine expert policies, as evaluated in the domain of simulated bipedal locomotion across different classes of terrain.
We also introduce an input injection method for augmenting an existing policy network to exploit new input features.
Lastly, our method uses transfer learning to assist in the efﬁcient acquisition of new skills.
The combination of these methods allows a policy to be incrementally augmented with new skills.
We compare our progressive learning and integration via distillation (PLAID) method against two alternative baselines
I NTRODUCTION
As they gain experience, humans develop rich repertoires of motion skills that are useful in different contexts and environments. Recent advances in reinforcement learning provide an opportunity to understand how motion repertoires can best be learned, recalled, and augmented.
Inspired by studies on the development and recall of movement patterns useful for different locomotion contextsRoemmich and Bastian (2015), we develop and evaluate an approach for learning multi-skilled movement repertoires. In what follows below, we refer to the proposed method as PLAID: Progressive Learning and Integration via Distillation.
For long lived applications of complex control tasks a learning system may need to acquire and integrate additional skills.
Accordingly, our problem is deﬁned by the sequential acquisition and integration of new skills. Given an existing controller that is capable of one-or-more skills, we wish to: (a) efﬁciently learn a new skill or movement pattern in a way that is informed by the existing control policy, and (b) to reintegrate that into a single controller that is capable of the full motion repertoire. This process can then be repeated as necessary.
In the process of acquiring a new skill, we also allow for a control policy to be augmented with additional inputs, without adversely impacting its performance.
This is a process we refer to as input injection . Understanding the time course of sensorimotor learning in human motor control is an open research problem Wolpert and Flanagan (2016) that exists concurrently with recent advances in deep reinforcement learning.
Issues of generalization, context-dependent recall, transfer or "savings" in fast learning, forgetting, and scalability are all in play for both human motor control models and the learning curricula proposed in reinforcement learning. While the development of hierarchical models for skills offers one particular solution that supports scalability and that avoids problems related to forgetting, we eschew this approach in this work and instead investigate a progressive approach to integration into a control policy deﬁned by a single deep network.
Distillation refers to the problem of combining the policies of one or more experts in order to create one single controller that can perform the tasks of a set of experts .
It can be cast as a supervised regression problem where the objective is to learn a model that matches the output distributions of all expert policies Parisotto et al.
Teh et al.
(2017); Rusu et al.
However, given a new task for which an expert is not given, it is less clear how to both learn the new task while successfully integrating this new skill in the pre-existing repertoire of the control policy for an agent.
One well-known technique in machine learning to signiﬁcantly improve sample efﬁciency across similar tasks is to use Transfer Learning (TL) Pan and Yang (2010), which seeks to reuse knowledge learned from solving a previous task to efﬁciently learn a new task.
However, transferring knowledge from previous tasks to new tasks may not be straightforward; there can be negative transfer wherein a previously-trained model can take longer to learn a new task via ﬁne-tuning than would a randomly-initialized model Rajendran et al.
Additionally, while learning a new skill, the control policy should not forget how to perform old skills.
The core contribution of this paper is a method (PLAID) to repeatedly expand and integrate a motion repertoire. The main building blocks consist of policy transfer and multi-task policy distillation , and the method is evaluated in the context of a continuous motor control problem, that of robust locomotion over distinct classes of terrain.
We evaluate the method against two alternative baselines.
We also introduce input injection , a convenient mechanism for adding inputs to control policies in support of new skills, while preserving existing capabilities.
R ELATED W ORK
Transfer learning and distillation are of broad interest in machine learning and RL Pan and Yang (2010);
Taylor and Stone (2009); Teh et al.
Here we outline some of the most relevant work in the area of Deep Reinforcement Learning (DRL) for continuous control environments.
Distillation Recent works have explored the problem of combining multiple expert policies in the reinforcement learning setting.
A popular approach uses supervised learning to combine each policy by regression over the action distribution.
This approach yields model compression Rusu et al.
(2015) as well as a viable method for multi-task policy transfer Parisotto et al.
(2015) on discrete action domains including the Arcade Learning Environment Bellemare et al.
We adopt these techniques and extend them for the case of complex continuous action space tasks and make use of them as building block.
Transfer Learning Transfer learning exploits the structure learned from a previous task in learning a new task. Our focus here is on transfer learning in environments consisting of continuous control tasks.
The concept of appending additional network structure while keeping the previous structure to reduce catastrophic forgetting has worked well on Atari games Rusu et al.
; Parisotto et al. (2015); Rusu et al.
Chen et al.
Other methods reproduce data from all tasks to reduce the possibility of forgetting how to perform previously learned skills e.g, Shin et al.
; Li and Hoiem (2016).
Recent work seeks to mitigate this issue using selective learning rates for speciﬁc network parameters Kirkpatrick et al.
A different approach to combining policies is to use a hierarchical structure Tessler et al.
In this setting, previously-learned policies are available as options to execute for a policy trained on a new task.
However, this approach assumes that the new tasks will be at least a partial composition of previous tasks, and there is no reintegration of newly learned tasks.
A recent promising approach has been to apply metalearning to achieve control policies that can quickly adapt their behavior according to current rewards Finn et al.
This work is demonstrated on parameterized task domains.
Hierarchical RL further uses modularity to achieve transfer learning for robotic tasks Tessler et al.
This allows for the substitution of network modules for different robot types over a similar tasks Devin et al.
Other methods use Hierarchical Reinforcement Learning (HRL) as a method for simplifying a complex motor control problem, deﬁning a decomposition of the overall task into smaller tasks Kulkarni et al.
; Heess et al. (2016); Peng et al.
While these methods examine knowledge transfer, they do not examine the reintegration of policies for related tasks and the associated problems such as catastrophic forgetting . Recent work examines learned motions can be shaped by prior mocap clips Merel et al. (2017), and that these can then be integrated in a hierarchical controller.
F RAMEWORK
In this section we outline the details of the Reinforcement Learning (RL) framework.
We also give an introduction to the concepts of TL and distillation .
3. R EINFORCEMENT L EARNING
Leveraging the framework of reinforcement learning, we frame the problem as an Markov Decision Processes (MDP): at each time step t , the world (including the agent) is in a state s t ∈ S , wherein the agent is able to perform actions a t ∈ A , sampled from a policy π ( s t , a t ) =
p ( a t | s t ) and resulting in state s t +1 ∈ S according to transition probabilities T ( s t , a t , s t +1 ) .
Performing action a t from state s t produces a reward r t ; the expected cumulative reward earned from following some policy π may then be written as:
[Equation]
where T is the time horizon, and γ is the discount factor, deﬁning the planning horizon length.
The agent’s goal is to learn an optimal policy, π ∗ , maximizing J ( π ) .
If the policy has parameters θ π , then the goal may be reformulated to identify the optimal parameters θ ∗ π :
[Equation]
Our policy models a Gaussian distribution with a mean state dependent mean, µ θ t ( s t ) .
Thus, Our stochastic policy may be formulated as follows:
[Equation]
where Σ is a diagonal covariance matrix with entries σ 2 i on the diagonal, similar to Peng et al.
To optimize our policy, we use stochastic policy gradient methods, a well-established family of techniques for reinforcement learning Sutton et al.
The gradient of the expected reward with respect to the policy parameters, ∇ θ π J
( π ( ·| θ π )) , is given by:
[Equation]
where d θ = (cid:82) S (cid:80) [-Tt =
-]T t =0 γ t p 0 ( s 0 )( s [-
-]0 → s | t, π 0 ) ds 0 is the discounted state distribution, p 0 ( s ) represents the initial state distribution, and p 0 ( s 0 )( s 0 → s | t, π 0 ) models the likelihood of reaching state s by starting at state s 0 and following the policy π ( a, s | θ π ) for T steps Silver et al.
A π ( s, a ) represents an advantage function Schulman et al.
In this work, we use the Positive Temporal Differnce (PTD) update proposed by Van Hasselt (2012) for A π ( s, a ) :
[Equation]
where V π ( s )
=
E (cid:104)(cid:80)
Tt =
γ t r t | s
= s (cid:105) is the value function, which gives the expected discounted cumulative reward from following policy π starting in state s .
PTD has the beneﬁt of being insensitive to the advantage function scale.
Furthermore, limiting policy updates in this way to be only in the direction of actions that have a positive advantage has been found to increase the stability of learning Van Hasselt (2012).
Because the true value function is unknown, an approximation V π ( · | θ v ) with parameters θ v is learned, which is formulated as the regression problem:
[Equation]
3. P OLICY D ISTILLATION
Given a set expert agents that have solved/mastered different tasks we may want to combine the skills of these different experts into a single multi-skilled agent.
This process is referred to as distillation .
Distillation does not necessarily produce an optimal mix of the given experts but instead tries to produce an expert that best matches the action distributions produced by all experts .
This can be preferred because this method functions independent of the reward functions used to train each expert .
Distillation also scales well with respect to the number of tasks or experts that are being combined.
3. T RANSFER L EARNING
Given an expert that has solved/mastered a task we want to reuse that expert knowledge in order to learn a new task efﬁciently.
This problem falls in the area of Transfer Learning Pan and Yang (2010).
Considering the state distribution expert is skilled at solving, ( D ω i the source distribution) it can be advantageous to start learning a new , target task
ω
i +1 with target distribution D ω
i +1 using assistance from the expert .
The agent learning how to solve the target task with domain D ω i +1 is referred to as the student .
When the expert is used to assist the student in learning the target task it can be referred to as the teacher .
The success of these methods are dependent on overlap between the D ω i and D ω
i +1 state distributions.
P ROGRESSIVE L EARNING
Although we focus on the problem of being presented with tasks sequentially there exist other methods for learning a multi-skilled character. We considered 3 overall integration methods for learning multiple skills. The ﬁrst being a MultiTasker: controller that trys to learn multiple tasks at the same time (MultiTasker), where a number of skills are learned at the same time. It has been shown that learning many tasks together can be faster than learning each task separately Parisotto et al. (2015).
The curriculum for using this method is shown in Figure 1a were during a single RL simulation all tasks are learned together.
It is also possible to learn each task separately but in parallel and then combine the resulting policies Figure 1b.
We attempted to evaluate this method as well but we found that learning many skills from scratch was challenging, we were only able to get fair results for the ﬂat task.
Also, when a new task is to be learned with this model it would occur outside of the original parallel learning, leading to a more sequential method.
The last version learns each task sequentially using TL from the previous, most skilled policy Figure 1c.
This method works well for both combining learned skills and learning new skills.
[Figure]
π
[Figure]
[Figure]
π
[Figure]
L
[Figure]
π
[Figure]
D
[Figure]
π
Lπ
Lπ
D ˆ π. . .
D ˆ . . .
L ω −π ω
D ˆ π ω
L ω π ω
+
D ˆ π ω +
[Figure]
[Figure]
AND I NTEGRATION VIA D ISTILLATION
In this section we detail our proposed learning framework for continual policy transfer and distillation (PLAID).
In the acquisition (TL) step, we are interested in learning a new task ω
i +1 .
Here transfer can be beneﬁcial if the task structure is somewhat similar to previous tasks
ω i .
We adopt the TL strategy of using an existing policy network and ﬁne-tuning it to a new task.
Since we are not concerned with retaining previous skills in this step, we can update this policy without concern for forgetting.
As the agent learns it will develop more skills, the addition of every new skill can increase the probability of transferring knowledge to assist the learning of the next skill.
In the integration ( distillation ) step, we are interested in combining all past skills ( π 0 , . .
, π i ) with the newly acquired skill π i +1 .
Traditional approaches have used policy regression where data is generated by collecting trajectories of the expert policy on a task.
Training the student on these trajectories does not always result in robust behaviour.
This poor behaviour is because the student experiences a different distribution of trajectories than the expert during evaluation.
To compensate for this distribution difference, portions of the trajectories should be generated by the student , this allows the expert to suggest behaviour that will pull the state distribution of the student closer to the expert ’s. This is a common problem in learning a model to reproduce a given distribution of trajectories Ross et al.
; Bengio et al. (2015);
Martinez et al.
; Lamb et al.
We use a method similar to the DAgger algorithm Ross et al. (2010) which has been found to be useful for distilling policies Parisotto et al.
(2015).As our RL algorithm is an actor-critic method, we also perform regression on the critic by ﬁtting both in the same step.
4. H IGH L EVEL E XPERIMENT D ESIGN
The results presented in this work cover a range of tasks that share a similar action space and state space. Our focus is to demonstrate continual learning between related tasks.
However, the conceptual framework allows for extensions that would permit differing state spaces, described later in Section: 5.2.
R ESULTS
In this experiment, our set of tasks consists of 5 different terrains that a 2D humanoid walker learns to traverse.
A bipedal character is trained to navigate multiple types of terrain, including ﬂat in (Figure 2a), incline (Figure 2b), steps (Figure 2c), slopes (Figure 2d), gaps (Figure 2e) and a combination of all terrains mixed (Figure 2f) that are not trained on.
The goal in these tasks is to maintain a consistent forward velocity traversing various terrains, while also matching a motion capture clip of a natural human walking gait on ﬂat ground, similar to Peng and van de Panne (2016).
The 2D humanoid receives as input both a character and eventually a terrain state representation.
A detailed description of the experimental setup is included in Section: 8.4.
The tasks are presented to the agent sequentially and the goal is to progressively learn to traverse all terrain types.
[Figure]
We evaluate our approach against two baselines.
Firstly, we compare the above learning curriculum from learning new tasks in PLAID with learning new tasks from Scratch: randomly initilized controller (Scratch).
This will demonstrate that knowledge from previous tasks can be effectively transferred after distillation steps.
Second, we compare to the MultiTasker, to demonstrate that iterated distillation is effective for retention of learned skills. The MultiTasker is also used as a baseline for comparing learning speed.
5. T RANSFER L EARNING
First, the pd-biped is trained to produce a walking motion on ﬂat ground ( ﬂat ).
In Figure 3a PLAID is compared to the two baselines for training on incline .
The Scratch method learns the slowest as it is given no information about how to perform similar skills.
The ﬁrst MultiTasker for the incline task is initialized from a terrain injected controller that was trained to walk on ﬂat ground.
Any subsequent MultiTasker is initialized from the ﬁnal MultiTasker model of the preceding task.
This controller has to learn multiple tasks together, which can complicate the learning process, as simulation for each task is split across the training and the overall RL task can be challenging.
This is in contrast to using PLAID, that is also initialized with the same policy trained on ﬂat , that will integrate skills together after each new skill is learned.
In Figure 3b the MultiTasker is learning the new task ( steps ) with similar speed to PLAID.
However, after adding more tasks the MultiTasker is beginning to struggle in Figure 3c and fails in in Figure 3d, with the number of tasks it must learn at the same time.
While PLAID learns the new tasks faster and is able to integrate the new skill required to solve the task robustly.
5. F EATURE I NJECTION
An appealing property of using distillation in PLAID is that the combined policy model need not resemble that of the individual expert controllers.
For example, two different experts lacking state features and trained without a local map of the terrain can be combined into a single policy that has new state features for the terrain.
These new terrain features can assist the agent in identifying which task domain it is operating in.
We introduce the idea of feature injection for this purpose.
We augment a policy with additional input features while allowing it to retain its original functional behaviour similar to Chen et al. This is [Figure]
achieved by adding additional inputs to the neural network and initializing the connecting layer weights and biases to 0 .
By only setting the weights and biases in the layer connecting the new features to the original network to 0 , the gradient can still propagate to any lower layers which are initialized random without changing the functional behavior.
This performed when distilling the ﬂat and incline experts .
5. D ISTILLING M ULTIPLE P OLICIES
Training over multiple tasks at the same time my help the agent learn skills quicker but this may not scale with respect to the number of tasks.
When training the MultiTasker over two or even three tasks (Figure 4a) the method displays good results, however when learning a fourth or more tasks the method struggles, as shown in Figure 4b and 4b.
Part of the reason for this struggle is when new tasks are added the MultiTasker has to make trade-offs between more tasks to maximizes.
As more tasks are added this trade-off becomes increasingly complex resulting in the MultiTasker favouring easier tasks.
Using PLAID to combine the skills of many policies appears to scale better with respect to the number of skills being integrated.
This is likely because distillation is a semi-supervised method which is more stable than the un-supervised RL solution.
This can be seen in Figure 4d, 4e and especially in 4f where PLAID combines the skills faster, and can ﬁnd higher value policies in practice.
PLAID also presents zero-shot training on tasks it has never trained on. In Figure 5 this generalization is shown as the agent navigate across the mixed environment.
There are some indications that distillation is hindering during the initial few training iterations.
We are initializing the network used in distillation with the most recently learning policy after TL.
The large change in the initial state distribution from the previous seen distribution during TL could be causing larger gradients to appear, disrupting some of the structure learned during the TL step, shown in Figure 4d and 4e.
There also might not exist a smooth transition in policy space between the newly learned policy and the previous policy distribution.
Forgetting To-do: Need that TL only, forget baseline D ISCUSSION
Transfer Learning: Because we are using an actor-critic learning method we also studied the possibility of using the value functions for TL.
We did not discover any empirical evidence that this assisted the learning process.
When transferring to a new task the state distribution has changed and the reward function maybe be completely different.
This makes it unlikely the value function will be accurate on this new task.
Also, value functions are in general easier and faster to learn than policies, implying that it is less important to transfer.
We also ﬁnd that helpfulness of TL depends on not only the task difﬁculty task but the reward function as well.
Two tasks may overlap in state space but the area they overlap could be easily reachable.
In this case TL may not give signiﬁcant beneﬁt because the overall RL problem is easy.
The greatest beneﬁt is gained from TL when the state space that overlaps for two tasks is difﬁcult to reach and in that difﬁcult to reach area is where the highest rewards are achieved.
To-do: I wonder if the distillation also provides some form of bias reduction that helps generalization and transfer learning... [Figure]
c and PLAID trained gaps [Figure] MultiTasker vs PLAID: The MultiTasker may be able to produce a policy that has higher overall average reward but in practice constraints can keep the method from combining skills gracefully.
If the reward functions are different between tasks the MultiTasker can favour the task with higher rewards, as this tasks may receive higher advantage.
It is also a non-trivial task to normalize the reward functions for each task in order to combine them.
The MultiTasker my also favour tasks that are in general easier than other tasks.
We have shown that the distiller can scale better to the number of tasks than the MultiTasker.
The tasks used in this analysis could be considered too similar, with respect to state space and reward functions.
We believe PLAID would further outperform the MultiTasker if the tasks were more difﬁcult and the reward functions dissimilar.
In our evaluation we compare the number of iterations PLAID uses to the number the MultiTasker uses on only the new task, this is not necessarily fair.
The MultiTasker gains its beneﬁts from training on the other tasks together.
If the idea is to reduce the number of simulation samples that are needed to learn new tasks then the MultiTasker would fall far behind.
Distillation is also very efﬁcient with respect to the number of simulation steps needed.
Data could be collected from the simulator in groups and learned from in many batches before more data is needed as is common for behavioral cloning.
We did not perform a study into how effective distillation could be by conservatively collecting data from the simulation in order to keep comparatively similar learning conditions between distillation and the MultiTasker. 6. L IMITATIONS : In our transfer learning results we could be over ﬁtting the initial expert for the particular task it was learning. Making it more challenging for the policy to learning a new task, resulting in negative transfer .
After learning many new tasks the previous tasks may not receive a large enough potion of the distillation training process to preserve the experts skill well enough.
How best to chose which data should be trained on next to best preserve the behaviour of experts is a general problem with multi-task learning.
Distillation treats all tasks equally independent of their reward.
This can result in very low value tasks receiving potentially more distribution than desired and high value tasks receiving not enough.
We haven’t needed the use a one-hot vector to indicate what task the agent is performing.
We want the agent to be able to recognize which task it is being given but we do realize that some tasks could be too similar to differentiate between.
For example, walking vs jogging on ﬂat ground.
6. F UTURE W ORK :
It could be helpful to perform some kind of task prioritization during the distillation step.
This could assist the agent with forgetting issues or help with relearning tasks.
Here we use the Mean Squared Error (MSE) to pull the distributions of student policies in line with expert polices for distillation .
It could be more advantageous to use a better metric for distance between policies.
Previous methods have used KL
Divergence in the discrete action space domain where the value function, encodes the policy, for example, with Deep Q-Netowrk (DQN).
In this work we are not focusing on producing the best policy from a mixture of experts , but instead matching the distributions from a number of experts .
The difference is subtle but in practice it can be more difﬁcult to balance many experts with respect to their reward functions.
It could also be beneﬁcial to use some kind of KL penalty while performing distillation .
Potentially something similar to the work in Teh et al.
(2017), that will help keep the policy from shifting to fast during training.
C ONCLUSION
Using PLAID we gain the beneﬁts of distillation as a supervised method to integrate policies together.
This being more efﬁcient then learning all skills together, and is a favourable method for integrating multiple skills together.
Ideally, the more skilled an agent is the easier it should be to learn new skills.
A PPENDIX 8. N ETWORK M ODELS
We used two different Network models for the experiments in this paper.
The ﬁrst model is a blind model that does not have any terrain features.
The blind policy is a Neural Network with 2 hidden layers ( 512 × 256 ) with ReLU activations.
The output layer of the policy network has linear activations.
The network used for the value function has the same design except there is 1 output on the ﬁnal layer.
This design is used for the ﬂat and incline tasks.
We augment the blind network design by adding features for terrain to create an agent with sight .
This network with terrain features has a single convolution layer with 8 ﬁlters of width 3 .
This constitutional layer is followed by a dense layer of 32 units.
The dense layer is then concatenated twice, once along each of the original two hidden layers in the blind version of the policy.
8. H YPER P ARAMETERS AND T RAINING
The policy network models a Gaussian distribution by outputting a state dependant mean.
We use a state independent standard deviation that normalized with respect to the action space and multiplied by 0 .
We also use a version of epsilon greedy exploration where with (cid:15) probability an exploration action is generated.
For all of our experiments we linearly anneal (cid:15) from 0 . 2 to 0 . 1 in 100 , 000 iterations and leave it from that point on. Each training simulation takes approximately 5 hours across 8 threads.
For network training we use Stochastic Gradient Decent (SGD) with momentum.
During the distillation step we use gradually anneal the probability of selecting an expert action from 1 to 0 over 10 , 000 iterations.
For the evaluation of each model on a particular task we use the average reward achieved by the agent over at most 100 seconds of simulation time.
We average this over running the agent over a number of randomly generated simulation runs.
8. I NPUT F EATURE I NJECTION
In order to add input features to network we construct a new network. This new network has a portion of it that is the same design as the previous network plus additional parameters. First initialize the new network with random parameters.
Then we copy over the values from the previous network into the new one for the portion of the network design the matches the old. Then the weight for the layers that connect the old portion of the network to the new are set to 0 . This will allow the network to preserve the previous distribution it modeled. Having the parameters from the old network will also help generate gradients to train the new 0 valued network parameters.
The agent used in the simulation models the dimensions and masses of the average adult. The size of the character state is 50 parameters that include the relative position and velocity of the links in the agent. The action space consists of 11 parameters that indicate target joint positions for the agent.
The target joint positions (pd-targets) are turned into joint torques via proportional derivative controllers at each joint.
The reward function for the agent consists of 3 primary terms.
The ﬁrst is a velocity term the rewards the agent for going at velocity of 1 m/s The second term is the difference between the pose of the agent and the current pose of a kinematic character controlled via a motion capture clip. The difference between the agent and the clip consists of the rotational difference between each corresponding joint and the difference in angular velocity. The angular velocity for the clip is approximated via ﬁnite differences between the current pose of the clip and it’s last pose.
The last term is an L2 penalty on the torques generated by the agent to help reduce spastic motions.
We also impose torque limits on the joints to reduce unrealistic behaviour, limits: Hips 150 , knees 125 , ankles 100 , shoulders 100 , elbows 75 and neck 50 N/m .
How does pd-biped work details on reward function
Terrain Types All terrain types are randomly generated per episode, except for the ﬂat terrain.
The incline terrain is slanted and the slant of the terrain is randomly sampled between 20 and 25 degrees. The steps terrain consists of ﬂat segments with widths randomly sampled from 1 . 0 m to 1 . 5 m followed by sharp steps that have randomly generated heights between 5 cm and 15 cm . The slopes terrain is randomly generated by updating the slope of the previous point in the ground with a value sampled from − 20 and 20 degrees to generate a new portion of the ground every 10 cm .
The gaps terrain generate gaps of width 25 - 30 cm separated by ﬂat segments of widths sampled from 2 . 0 m to 2 . 5 m . The mixed terrain is a combination of the above terrains where a portion is randomly chosen from the above terrain types. 8. M ULTITASKER
In certain cases the MultiTasker can learn new task faster than PLAID. In Figure 6a we present the MultiTasker and compare it to PLAID. In this case the MultiTasker splits its training time across multiple tasks, here we compare the two methods with respect to the time spent learning on the single new task.
This is a good baseline to compare our method against but in some way this is not fair.
If the real measure of how efﬁcient a learning method is the number of simulation samples that are needed to learn then the MultiTasker would fall far behind because it needs to train across all tasks to gain the beneﬁts of improving a single task. [Figure]