P ROGRESSIVE R EINFORCEMENT L EARNING WITH D ISTILLATION FOR M ULTI -S KILLED M OTION C ONTROL
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems, including agents that can move with skill and agility through their environment.
An open problem in this setting is that of developing good strategies for integrating or merging policies for multiple skills, where each individual skill is a specialist in a speciﬁc skill and its associated state distribution.
We extend policy distillation methods to the continuous action setting and leverage this technique combine expert policies, as evaluated in the domain of simulated bipedal locomotion across different classes of terrain.
We also introduce an input injection method for augmenting an existing policy network to exploit new input features.
Lastly, our method uses transfer learning to assist in the efﬁcient acquisition of new skills.
The combination of these methods allows a policy to be incrementally augmented with new skills.
We compare our progressive learning and integration via distillation (PLAID) method against two alternative baselines.
I NTRODUCTION
To-do: Really need to stress that our method with integrating NEW skills, most other methods are just composing many polices without ideas how to best learning a NEW skill
Reinforcement Learning has been successfully used for complex control tasks in recent years.
There has been multiple solutions for many difﬁcult control tasks, however, little work has been done to combine learned skills.
For long lived applications of complex control tasks a learning system may need to acquire and integrate additional skills.
This can be seen in motor development of children, where there are a number of stages that are progressed though to learn robust motor skills Adolph and Berger (2007).
Continual learning could offer gains in learning speed and allow the development of a single policy capable of performing a wide range of tasks.
How to best learn a robust method for combining previously learned skills to create a mutli-skilled controller that is more capable than the individual controllers remains an open question.
This is a challenging problem for a number of reasons; scalability, forgetting, how best to combine policies and understand which expert to use when.
Distillation refers to the problem of combining the policies of one or more experts in order to create one single controller that can perform the tasks of a set of experts .
Distillation treats the problem as a supervised regression problem where the objective is to learn a model that matches the output distributions of all expert policies Parisotto et al.
Teh et al.
(2017); Rusu et al.
However, what if a new task for which there is no expert is given, what is a reasonable method for learning the new task and including this new skill in the repertoire of the agent?
One well-known technique in machine learning to signiﬁcantly improve sample efﬁciency across similar tasks is to use Transfer Learning (TL) Pan and Yang (2010), which seeks to reuse knowledge learned from solving a previous task to efﬁciently learn a new task.
However, transferring knowledge from previous tasks to new tasks is not always straightforward; there can be negative transfer , in some cases, a previously-trained model can take longer to learn a new task via ﬁne-tuning than would a randomly-initialized model Rajendran et al.
Here we investigate methods to continually learn and integrate new tasks into an existing policy in an efﬁcient and robust fashion.
Importantly, to learn these new skills while not forgetting how to perform old skills.
The main building blocks in this work are policy transfer and multi-task policy distillation .
First, an existing policy is trained on a new task using TL, TL should increase the speed at which new skills is learned.
Then the resulting policy that has learned the new task can be distilled with the existing policy, resulting in a new policy that can perform both tasks, with marginal forgetting .
In the distillation step we can also change the policy model design.
For example, after learning to walk on ﬂat and inclined ground we can add state features for the local terrain.
Here we introduce the option to modify the sensory-motor transform , a model that maps from sensory input to motor or action outputs, by augmenting the policy network to include additional state features.
This augmented version of the network will have additional capability to detect subsets of the state space and perform more robust locomotion planning to travel in complex environments.
These multi-skilled policies will then be the basis for further integration of new tasks.
The combination of these methods constructs a scalable method to integrate new skills into a single policy.
R ELATED W ORK
Knowledge reuse has been a popular area of machine learning and AI for some time
Pan and Yang (2010);
Taylor and Stone (2009).
There has been a signiﬁcant amount of work in the area including methods to combine models that have both learned to solve a different task and methods for using information from learning other tasks while learning a new one Teh et al.
Here we outline some of the most relevant work in the area of Deep Reinforcement Learning (DRL) for continuous control environments.
Distillation Recent works have explored the problem of combining multiple expert policies in the reinforcement learning setting.
A popular approach is to use supervised learning to combine each policy by regression over the action distribution.
This approach has been shown to both yield model compression
Rusu et al.
(2015) and a viable method for multi-task policy transfer Parisotto et al.
(2015) on discrete action domains including the Arcade Learning Environment Bellemare et al.
We adopt these techniques and extend them for the case of complex continuous action space tasks and make use of them as building block.
Transfer Learning There is limited work done in this area on environments with continuous control tasks.
The work in Rajendran et al.
(2015) proposes that it may be possible but does not show any examples of this.
The concept of appending additional network structure while keeping the previous structure to ensure there is no forgetting has worked well on the Atari games Rusu et al.
; Parisotto et al. (2015);
Rusu et al.
; Chen et al.
Similar work uses methods to reproduce data from all tasks to reduce the possibility of forgetting how to perform previously learned skills e.g, Shin et al.
; Li and Hoiem (2016).
There is also the challenge of catastrophic forgetting , where as new skills are learned old ones are forgotten.
Some, recent work seeks to negate this issue using selective learning rates for speciﬁc network parameters Kirkpatrick et al.
A different approach to combining policies is to use a hierarchical structure Tessler et al.
In this setting, previously-learned policies are available as options to execute for a policy trained on a new task.
However, their approach assumes that the new tasks will be at least a partial composition of previous tasks, and they do not demonstrate reintegration of the newly learned tasks.
Hierarchical RL Another related line of work uses modularity to achieve transfer learning for robotic tasks.
This modularity enables the swapping of network modules for different robot types over a similar task that was train with one particular robot Devin et al.
Other works use Hierarchical Reinforcement
Learning (HRL) as a method for simplifying a complex motor control problem, deﬁning a decomposition of the overall task into smaller tasks Kulkarni et al.
; Heess et al. (2016); Peng et al.
Introducing more modularity to enable transfer between differing task domains is a promising direction of research.
Although, these works examines knowledge transfer it does not examine transfer and reintegration of policies for related tasks and associated problems such as catastrophic forgetting .
F RAMEWORK
In this section we outline the details of the Reinforcement Learning (RL) framework.
We also give an introduction to the concepts of TL and distillation .
3. R EINFORCEMENT L EARNING
Leveraging the framework of reinforcement learning, we frame the general problem as a Markov Decision
Processes (MDP): at each time step t , the world (including the agent) is in a state s t ∈ S , wherein the agent is able to perform actions a t ∈ A , sampled from a policy π ( s t , a t ) =
p ( a t | s t ) and resulting in state s t +1 ∈ S according to transition probabilities T ( s t , a t , s t +1 ) .
Performing action a t from state s t produces a reward r t ; the expected cumulative reward earned from following some policy π may then be written as:
[Equation]
where T is a possibly-inﬁnite horizon, and γ is the discount factor, capturing the intuition that future rewards become less impactful as they get further in the future.
The agent’s goal is to learn an optimal policy, π ∗ , maximizing J ( π ) .
If the policy has parameters θ π ∈ Θ , then the goal may be reformulated to identify the optimal parameters θ [Equation] π :
[Equation]
We assume that actions follow a Gaussian distribution around a mean determined by the current state, µ θ t ( s t ) .
Thus, Our stochastic policy may be formulated as follows:
[Equation]
where Σ is a diagonal covariance matrix with entries σ 2 i on the diagonal, similar to Peng et al.
To optimize our policy, we employ policy gradient methods, a well-established family of techniques for reinforcement learning Sutton et al.
The gradient of the expected reward with respect to the policy parameters, ∇ θ π J
( π ( ·| θ π )) , is given by:
[Equation]
where d θ = (cid:82) S (cid:80) Tt =
γ t p 0 ( s 0 )( s
→
s | t, π 0 ) ds 0 is the discounted state distribution, p 0 ( s ) represents the initial state distribution, and p 0 ( s 0 )( s 0 → s | t, π 0 ) models the likelihood of reaching state s by starting at state s 0 and following the policy π ( a, s | θ π ) for T steps Silver et al.
A π ( s, a ) represents an advantage function Schulman et al.
In this work, we adapt the Positive Temporal Differnce (PTD) update proposed by Van Hasselt (2012):
[Equation]
otherwise
[Equation]
where V π ( s )
=
E (cid:104)(cid:80)
Tt =
γ t r t | s
= s (cid:105) is the value function, which gives the expected discounted cumulative reward from following policy π starting in state s .
PTD has the beneﬁt of being insensitive to the advantage function scale.
Furthermore, limiting policy updates in this way to be only in the direction of actions that have a positive advantage has been found to increase the stability of learning Van Hasselt (2012).
Because the true value function is unknown, an approximation V π ( ·| θ v ) with parameters θ v can be learned, which can be formulated as a regression problem: minimize
[Equation]
3. P OLICY D ISTILLATION
When there exist many agents that have solved/mastered different tasks we may want to combine the skills of these different experts into a single expert agent.
This process is referred to as distillation .
Distillation does not necessarily produce an optimal mix of the given experts but instead tries to produce an expert that best matches the action distributions produced by the experts .
This can be preferred because this method can function independent of the separate reward functions used when training each expert .
Here we also favour distillation , as the method should scale better to the number of tasks or experts that are being combined.
3. T RANSFER L EARNING
Given an expert that has solved/mastered a task we want to reuse the knowledge the expert has in order to learn a new task.
This problem falls in the area of Transfer Learning Pan and Yang (2010).
When considering the distribution of states the existing expert is skilled at solving, ( D ω i the source distribution) it may be advantageous to start learning the new, target task
ω
i +1 with target distribution D ω
i +1 using assistance from the existing expert .
The agent learning how to solve the target task with domain D ω
i +1 is sometimes referred to as the student .
When the expert is used to assist the student in learning the target task and can be referred to as the teacher .
The success of these methods rely on the assumption that there is some overlap between the D ω i and D ω
i +1 state distributions.
P ROGRESSIVE L EARNING AND I NTEGRATION
In this section we detail our proposed learning framework for continual policy transfer and distillation (Progressive Learning and Integration via Distillation (PLAiD)).
In the acquisition (TL) step, we are interested in learning a new task
ω
i +1 .
Here we assume the task to be somewhat similar to previous tasks ω
i such that transfer can be beneﬁcial.
The more skilled an agent is the more likely it will have some knowledge that will assist in learning a new task.
We adopt the straightforward transfer learning strategy of using an existing policy network and ﬁne-tuning it to the new task.
Since we are not interested in retaining any of our previous skills in this step, we can update this policy without concern for forgetting.
In the integration ( distillation ) step, we are interested in combining all past skills ( π 0 , . .
, π i ) with the newly acquired one π i +1 .
Traditional approaches have commonly used policy regression where the tuples for the regression are generated by running trajectories of the expert policy on its task.
Training the student on these trajectories does not always result in robust behaviour.
This poor behaviour is because the student experiences a different state distribution than the expert during evaluation.
To compensate for this state distribution difference sections of the trajectories should be generated by the student , this allows the expert to suggest behaviour that will pull the student state distribution closer to the expert ’s. This is a common problem in creating a model to reproduce a given distribution of trajectories Ross et al.
; Bengio et al.
; Martinez et al.
; Lamb et al.
We use the DAgger algorithm and demonstrate that for our locomotion problems, it is crucial to the robustness of distilled policies Parisotto et al.
(2015).As our reinforcement learning algorithms is an actor-critic method, we also perform regression on the critic by ﬁtting both in the same step.
4. H IGH L EVEL E XPERIMENT D ESIGN
For the results presented in this work, the range of tasks to be solved share a similar action space and state space, as our main focus is to demonstrate continual learning between related tasks.
However, the conceptual framework allows for simple extensions that would permit differing action or state spaces, since we can use a new network model as the student policy in the distillation step.
In Figure 1 a ﬂow diagram of
PLAiD, showing how new skills are learned and integrated into a new policy, is given.
[Figure]
ˆ
R ESULTS
The method in this work is evaluated on a humanoid robotic simulation.
Here a bipedal character is trained to navigate multiple types of terrain, including ﬂat and incline , shown in Figure 2a and Figure 2b as well as steps (Figure 2c), slopes (Figure 2d), gaps (Figure 2e) and a combination of all the terrains mixed (Figure 2f).
The environment used is similar to the environments used in Peng and van de Panne (2016) where the goal is to learn the pd-targets to establish a robust walking gait.
We add additional terrain types to this problem as well.
An example of a biped trained to walk on an steps is shown in Figure 2c.
In this experiment, our set of tasks consists of To-do: 5 different terrains that a 2D humanoid walker learns to traverse.
The details of the learning tasks as well as the reinforcement learning algorithm closely follow those outlined in Peng and van de Panne (2016) for ﬂat terrain only.
The goal in these tasks is to maintain a consistent forward velocity traversing various terrains, we further use a supplied motion capture clip of a natural human walking gait on ﬂat ground.
The 2D humanoid receives as input both a character and terrain state representation.
A detailed description of the experimental setup is included in Section: 8.4.
The tasks
[Figure]
are presented to the continual learner sequentially and the goal is to progressively learn to traverse all terrain types.
Although we focus on the problem of being presented with tasks sequentially there exist other methods for learning a multi-skilled character.
We considered 3 overall integration methods for learning multiple skills.
The ﬁrst being a MultiTasker: controller that trys to learn multiple tasks at the same time (MultiTasker), where a number of skills are learned at the same time.
It has been shown that learning may tasks together can be faster than learning each on separately Parisotto et al. (2015).
The curriculum for using this method is shown in Figure 3a were during a single RL simulation time is spent learning each task.
It is also possible to learn each separately but in parallel and then combine the resulting policies Figure 3b.
We attempted to evaluate this method as well but we found that learning many skills from scratch was challenging, we were only able to get fair results for the ﬂat task.
Also, when a new task is learned with this model it would occur outside of the original parallel learning, leading to a more sequential method.
The last version learns each task sequentially using TL from the previous, most skilled policy Figure 3c.
This method works well for both combining learned skills and learning new skills.
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
We evaluate our approach against two baselines.
Firstly, we compare the above learning curriculum from learning new tasks in PLAiD with learning new tasks from Scratch: randomly initilized controller (Scratch).
This will demonstrate that knowledge from previous tasks can be effectively transferred after distillation steps.
Second, we compare to the MultiTasker, to demonstrate that iterated distillation is effective for retention of learned skills.
5. T RANSFER L EARNING
The pd-biped is trained to produce a walking motion on ﬂat ground ( ﬂat ).
We then initialize training on a similar task with slanted ground at random inclines ( incline ), and compare it to training a controller with two other methods.
The Scratch version is trained to learn the task from a randomly initialized policy.
The distiller version initializes its policy with one that was trained to walk on ﬂat ground.
The MultiTasker version initialized its policy from one trained to walk on ﬂat ground but also alternates between ﬂat and incline during training.
The MultiTasker version is designed to evaluate the affects of adding skills to an agent while training on the old skills to prevent forgetting.
This is in contrast to the PLAiD method that will integrate skills together after each new skill is learned.
In Figure 4a PLAiD is compared to the two baselines for ﬂat + incline .
The Scratch method learns the slowest as it is given no information about how to perform similar skills.
The MultiTasker method is initialized from a controller that was trained to walk on ﬂat ground.
This controller has to learn multiple tasks together, which complicates the learning process as simulation for each task is split across the training process and the overall RL task can be challenging.
This is in contrast to using PLAiD that will integrate skills together after each new skill is learned.
However, it has been shown that training across many tasks at once can increase overall learning speed and potentially reduce forgetting Teh et al.
We can see this in Figure 4b where the MultiTasker is learning the new task ( steps ) with similar speed. However, after adding more tasks, we can see in Figure 4c the MultiTasker beginning to struggle with learning many tasks at the same time. On the other hand PLAiD learns the new task faster and is able to integrate the new skill required to solve the task robustly.
While training, the MultiTasker splits its time across the number of tasks equally.
If we only consider the time taken to learn the new task the MultiTasker can learn quicker.
However, as we will show later PLAiD will begin to out perform the MultiTasker as the number of tasks increases.
[Figure]
5. F EATURE I NJECTION
An appealing property of using distillation in PLAiD is that the combined policy model need not resemble that of the individual expert controllers.
Two different experts lacking state features and trained without a local map of the terrain can be combined into a single policy that has new state features for the terrain.
These new terrain features assist the agent in identifying which task domain it is operating in.
In our framework we wish to initialize the distillation step with the previous policy.
We introduce the idea of feature injection for this purpose.
We augment a policy with additional input features while allowing it to retain its original functional behaviour in a way similar to Chen et al.
This is achieved by adding additional inputs to the neural network and initializing the weights and biases associated with these new features to 0 .
this approach is demonstrated in the third step of the experiment ( π 3 in Figure 3c) by combining the blind ﬂat and incline expert policies via distillation.
5. D ISTILLING M ULTIPLE P OLICIES
Learning multiple tasks at the same time my help the agent learn tasks quicker but this method may not scale with respect to the number of tasks.
When training the MultiTasker over two or even three tasks (Figure 5a) the method shows improvement, however when learning a fourth task the method struggles, as shown in Figure 5c.
Using PLAiD to combine the skills of many policies appears to scale better with respect to the number of skills being integrated.
This is likely because distillation is a semi-supervised method which is more stable than the un-supervised RL solution.
This can be seen in Figure 5b and 5d where PLAiD combines the skills faster, and can ﬁnd higher value policies in practice.
[Figure]
Forgetting There are some indications that distillation is hindering during the initial few training steps.
We are initializing the network used in the distillation step with the most recently learning policy after TL.
The large change in the initial state distribution from the previous seen distribution during TL could be causing larger gradients to appear, disrupting some of the structure learned during the TL step, shown in Figure 5b and 5d.
There also might not exist a smooth transition in policy space between the newly learning policy and the previous learned policy distribution.
D ISCUSSION
Transfer Learning: For our transfer learning results we could be over ﬁtting.
The initial expert could have been over trained for the particular task it was learning.
Making it more challenging for the policy to learning a new task, resulting in negative transfer .
As we are using an actor-critic learning method we also studied the possibility of using the value functions for TL as well.
We did not discover any empirical evidence that this assisted the learning process.
When transferring to a new task the state distribution has changed and the reward function maybe be completely different.
This makes it unlikely the value function will be accurate on this new task.
Also, value functions are in general easier and faster to learn than policies, implying that it is less important to transfer knowledge from these function approximations.
We also ﬁnd that helpfulness of TL depends on not only the task difﬁculty task but the rewards as well.
Two tasks may overlap in state space but the area they overlap could be easily to reachable.
In this case TL may not give signiﬁcant beneﬁt because the overall RL problem is easy.
The greatest beneﬁt is gained from TL when the state space that overlaps for two tasks is difﬁcult to reach and in that difﬁcult to reach area is where the highest rewards are achieved.
To-do: I wonder if the distillation also provides some form of bias reduction that helps generalization and transfer learning...
MultiTasker vs PLAiD The MultiTasker may be able to produce a policy that has higher overall average reward but in practice constraints can keep the method from combining skills gracefully.
If the reward functions are different between tasks the MultiTasker can favour the task with higher rewards, as this tasks may receive higher advantage.
It is also a non-trivial task to normalize the reward functions for each task in order to combine them.
We have shown that the distiller can scale better to the number of tasks than the MultiTasker.
The tasks used in this analysis could be considered too similar, with respect to state space and reward functions.
We believe PLAiD would further outperform the MultiTasker if the tasks were more difﬁcult and the reward functions dissimilar.
6. L IMITATIONS
After learning many new tasks the previous tasks may not receive a large enough potion of the distillation training process to preserve the experts skill well enough.
How best to chose which data should be trained on next to best preserve the behaviour of experts is a general problem with multi-task learning.
Distillation treats all tasks equally independent of their reward.
This can result in very low value tasks receiving potentially more distribution than desired and high value tasks recieving not enough. We haven’t needed the use a one-hot vector to indicate what task to the agent is performing. We want the agent to be able to recognize which task it is being given but we do realize that some tasks could be too similar to differentiate between. For example, walking vs jogging on ﬂat ground.
6. F UTURE W ORK
It could be helpful to perform some kind of task prioritization during the distillation step.
This could assist the agent with forgetting or help relearn tasks.
Here we use the Mean Squared Error (MSE) to pull the distributions of the student policies in line with the expert polices for distillation .
It could be more advantageous to use a better metric for distance between the policies.
Previous methods have used KL Divergence in the discrete action space domain where the value function, in this case Deep Q-Netowrk (DQN), encodes the policy.
This could improve the TL after distillation step.
In this work we are not focusing on producing the best policy from a mixture of experts , but instead matching the distributions from a number of experts .
The difference is subtle but in practice it can be too difﬁcult to balance many reward functions. It could also be beneﬁcial to use some kind of KL penalty while performing distillation .
Something similar to the work in Teh et al.
(2017), that will help keep the policy from shifting too much / to fast during training.
C ONCLUSION
Using PLAiD we gain the beneﬁts of distillation as a supervised method to integrate policies together.
This being more efﬁcient then learning all skills together, and is a favourable method for integrating many skills together.
Ideally, the more skilled an agent is the easier it should be to learn new skills.
A PPENDIX 8. N ETWORK M ODELS
We used two different Network models for the experiments in this paper.
The ﬁrst model is a blind model that does not have any terrain features.
The blind policy is a Neural Network with 2 hidden layers ( 512 × 256 ) with ReLU activations.
The output layer of the policy network has linear activations.
The network used for the value function has the same design except there is 1 output on the ﬁnal layer.
This design is used for the ﬂat and incline tasks.
We augment the blind network design by adding features for terrain to create an agent with sight .
This network with terrain features has a single convolution layer with 8 ﬁlters of width 3 .
This constitutional layer is followed by a dense layer of 32 units.
The dense layer is then concatenated twice, once along each of the original two hidden layers in the blind version of the policy.
8. H YPER P ARAMETERS
The policy network models a Gaussian distribution by outputting a state dependant mean.
We use a state independent standard deviation.
Each policy uses an epsilon-greedy, ﬁxed standard deviation Gaussian for action exploration.
For all of our experiments we ﬁx the standard deviation to 0.1 and linearly anneal epsilon from 0.2 to 0.1 in 100, iterations and leave it from that point on.
For the learning we use SGD with momentum with the same settings for all experiments.
For DAGGER we gradually anneal the probability of selecting an expert action from 1 to 0 over 10000 iterations.
8. M ORE DETAILS ON TERRAIN I NJECTION 8. M ORE DETAILS ON TRAINING
For reinforcement learning, a time limit cap of 100 seconds is used.
To-do: Should discuss the quarks of hyper parameter changes for TL.
For example, how to initialize the exploration rate.
To-do: Which network to we initialize the distillation from?
How does pd-biped work details on reward function
Terrain Types To-do: Need Videos and images for each terrain type, gaps , steps , slopes and ﬂat .
To-do: ﬁlmstrip of mixed terrain 8.
M ORE ??
[Figure]
To-do: Need policy evaluation comparison for with and without DAGGER.
To-do: DUmping this here for now