Paper:
TRANSFER FROM MULTIPLE SOURCES IN THE SAME DOMAIN
Transferring knowledge from prior source tasks in solving a new target task can be useful in several learning applications. The application of transfer poses two serious challenges which have not been adequately addressed. First, the agent should be able to avoid negative transfer, which happens when the transfer hampers or slows down the learning instead of helping it. Second, the agent should be able to selectively transfer, which is the ability to select and transfer from different and multiple source tasks for different parts of the state space of the target task. We propose A2T (Attend, Adapt and Transfer), an attentive deep architecture which adapts and transfers from these source tasks. Our model is generic enough to effect transfer of either policies or value functions. Empirical evaluations on different learning algorithms show that A2T is an effective architecture for transfer by being able to avoid negative transfer while transferring selectively from multiple source tasks in the same domain.
Transferring knowledge from prior source tasks in solving a new target task can be useful in several learning applications. The application of transfer poses two serious challenges which have not been adequately addressed. First, the agent should be able to avoid negative transfer, which happens when the transfer hampers or slows down the learning instead of helping it. Second, the agent should be able to selectively transfer, which is the ability to select and transfer from different and multiple source tasks for different parts of the state space of the target task. We propose A2T (Attend, Adapt and Transfer), an attentive deep architecture which adapts and transfers from these source tasks. Our model is generic enough to effect transfer of either policies or value functions. Empirical evaluations on different learning algorithms show that A2T is an effective architecture for transfer by being able to avoid negative transfer while transferring selectively from multiple source tasks in the same domain.
1 INTRODUCTION
One of the goals of Artificial Intelligence (AI) is to build autonomous agents that can learn and adapt to new environments. Reinforcement Learning (RL) is a key technique for achieving such adaptability. The goal of RL algorithms is to learn an optimal policy for choosing actions that maximize some notion of long term performance. Transferring knowledge gained from tasks solved earlier to solve a new target task can help, either in terms of speeding up the learning process or in terms of achieving a better solution, among other performance measures. When applied to RL, transfer could be accomplished in many ways (see Taylor & Stone (2009; 2011) for a very good survey of the field). One could use the value function from the source task as an initial estimate in the target task to cut down exploration [Sorg & Singh (2009)]. Alternatively one could use policies from the source task(s) in the target task. This can take one of two forms - (i) the derived policies can be used as initial exploratory trajectories [Atkeson & Schaal (1997); Niekum et al. (2013)] in the target task and (ii) the derived policy could be used to define macro-actions which may then be used by the agent in solving the target task [Mannor et al. (2004); Brunskill & Li (2014)].
∗Authors contributed equally
While transfer in RL has been much explored, there are two crucial issues that have not been adequately addressed in the literature. The first is negative transfer, which occurs when the transfer results in a performance that is worse when compared to learning from scratch in the target task. This severely limits the applicability of many transfer techniques only to cases for which some measure of relatedness between source and target tasks can be guaranteed beforehand. This brings us to the second problem with transfer, which is the issue of identifying an appropriate source task from which to transfer. In some scenarios, different source tasks might be relevant and useful for different parts of the state space of the target task. As a real world analogy, consider multiple players (experts) who are good at different aspects of a game (say, tennis). For example, Player 1 is good at playing backhand shots while Player 2 is good at playing forehand shots. Consider the case of a new player (agent) who wants to learn tennis by selectively learning from these two experts. We handle such a situation in our architecture by allowing the agent to learn how to pick and use solutions from multiple and different source tasks while solving a target task, selectively applicable for different parts of the state space. We call this selective transfer. Our agent can transfer knowledge from Player 1 when required to play backhand shots and Player 2 for playing forehand shots. Further, let us consider consider the situation that both Player 1 and Player 2 are bad at playing drop shots. Apart from the source tasks, we maintain a base network that learns from scratch on the target task. The agent can pick and use the solution of the base network when solving the target task at the parts of the state space where transferring from the source tasks is negative. Such a situation could arise when the source task solutions are irrelevant for solving the target task over a specific portion of the state space, or when the transferring from the source tasks is negative over a specific portion of the state space (for example, transferring the bad drop shot abilities of Players 1 and 2). This situation also entails the first problem of avoiding negative transfer. Our framework allows an agent to avoid transferring from both Players 1 and 2 while learning to play drop shots, and rather acquire the drop shot skill by learning to use the base network. The architecture is trained such that the base network uses not just the experience obtained through the usage of its solutions in the target task, but the overall experience acquired using the combined knowledge of the source tasks and itself. This enables the base network solutions to get closer to the behavior of the overall architecture (which uses the source task solutions as well). This makes it easier for the base network to assist the architecture to fine tune the useful source task solutions to suit the target task perfectly over time.
The key contribution in the architecture is a deep attention network, that decides which solutions to attend to, for a given input state. The network learns solutions as a function of current state thereby aiding the agent in adopting different solutions for different parts of the state space in the target task.
To this end, we propose A2T: Attend, Adapt and Transfer, an Attentive Deep Architecture for Adaptive Transfer, that avoids negative transfer while performing selective transfer from multiple source tasks in the same domain. In addition to the tennis example, A2T is a fairly generic framework that can be used to selectively transfer different skills available from different experts as appropriate to the situation. For instance, a household robot can appropriately use skills from different experts for different household chores. This would require the skill to transfer manipulation skills across objects, tasks and robotic actuators. With a well developed attention mechanism, the most appropriate and helpful combination of object-skill-controller can be identified for aiding the learning on a related new task. Further, A2T is generic enough to effect transfer of either action policies or actionvalue functions, as the case may be. We also adapt different algorithms in reinforcement learning as appropriate for the different settings and empirically demonstrate that the A2T is effective for transfer learning for each setting.
2 RELATED WORK
As mentioned earlier, transfer learning approaches could deal with transferring policies or value functions. For example, Banerjee & Stone (2007) describe a method for transferring value functions by constructing a Game tree. Similarly, Sorg & Singh (2009) use the value function from a source task as the initial estimate of the value function in the target task.
Another method to achieve transfer is to reuse policies derived in the source task(s) in the target task. Probabilistic Policy Reuse as discussed in Fernández & Veloso (2006) maintains a library of policies and selects a policy based on a similarity metric, or a random policy, or a max-policy from the knowledge obtained. This is different from the proposed approach in that the proposed approach
can transfer policies at the granularity of individual states which is not possible in policy-reuse rendering it unable to learn customized policy at that granularity.Atkeson & Schaal (1997); Niekum et al. (2013) evaluated the idea of having the transferred policy from the source tasks as explorative policies instead of having a random exploration policy. This provides better exploration behavior provided the tasks are similar. Talvitie & Singh (2007) try to find the promising policy from a set of candidate policies that are generated using different action mapping to a single solved task. In contrast, we make use of one or more source tasks to selectively transfer policies at the granularity of state. Apart from policy transfer and value transfer as discussed above, Ferguson & Mahadevan (2006) discuss representation transfer using Proto Value Functions.
The idea of negative and selective transfer have been discussed earlier in the literature. For example, Lazaric & Restelli (2011) address the issue of negative transfer in transferring samples for a related task in a multi-task setting. Konidaris et al. (2012) discuss the idea of exploiting shared common features across related tasks. They learn a shaping function that can be used in later tasks.
The two recent works that are very relevant to the proposed architecture are discussed in Parisotto et al. (2015) and Rusu et al. (2016). Parisotto et al. (2015) explore transfer learning in RL across Atari games by trying to learn a multi-task network over the source tasks available and directly finetune the learned multi-task network on the target task. However, fine-tuning as a transfer paradigm cannot address the issue of negative transfer which they do observe in many of their experiments. Rusu et al. (2016) try to address the negative transfer issue by proposing a sequential learning mechanism where the filters of the network being learned for an ongoing task are dependent through lateral connections on the lower level filters of the networks learned already for the previous tasks. The idea is to ensure that dependencies that characterize similarity across tasks could be learned through these lateral connections. Even though they do observe better transfer results than direct fine-tuning, they are still not able to avoid negative transfer in some of their experiments.
3 PROPOSED ARCHITECTURE
Let there be N source tasks and let K1,K2, . . .KN be the solutions of these source tasks 1, . . . N respectively. Let KT be the solution that we learn in the target task T . Source tasks refer to tasks that we have already learnt to perform and target task refers to the task that we are interested in learning now. These solutions could be for example policies or state-action values. Here the source tasks should be in the same domain as the target task, having the same state and action spaces. We propose a setting where KT is learned as a function of K1, . . . ,KN ,KB , where KB is the solution of a base network which starts learning from scratch while acting on the target task. In this work, we use a convex combination of the solutions to obtain KT .
KT (s) = wN+1,sKB(s) + N∑ i=1 wi,sKi(s) (1)
N+1∑ i=1 wi,s = 1, wi,s ∈ [0, 1] (2)
wi,s is the weight given to the ith solution at state s.
The agent uses KT to act in the target task. Figure 1a shows the proposed architecture. While the source task solutionsK1, . . . ,KN remain fixed, the base network solutions are learnt and henceKB can change over time. There is a central network which learns the weights (wi,s, i ∈ 1, 2, . . . , N+1), given the input state s. We refer to this network as the attention network. The [0, 1] weights determine the attention each solution gets allowing the agent to selectively accept or reject the different solutions, depending on the input state. We adopt a soft-attention mechanism whereby more than one weight can be non-zero [Bahdanau et al. (2014)] as opposed to a hard-attention mechanism [Mnih et al. (2014)] where we are forced to have only one non-zero weight.
wi,s = exp (ei,s) N+1∑ j=1 exp (ej,s) , i ∈ {1, 2, . . . , N + 1} (3)
(e1,s, e2,s, . . . , eN+1,s) = f(s; θa) (4)
Here, f(s; θa) is a deep neural network (attention network), which could consist of convolution layers and fully connected layers depending on the representation of input. It is parametrised by θa and takes as input a state s and outputs a vector of length N + 1, which gives the attention scores for the N + 1 solutions at state s. Eq.(3) normalises this score to get the weights that follow Eq.(2).
If the ith source task solution is useful at state s, then wi,s is set to a high value by the attention network. Working at the granularity of states allows the attention network to attend to different source tasks, for different parts of the state space of the target task, thus giving it the ability to perform selective transfer. For parts of the state space in the target task, where the source task solutions cause negative transfer or where the source task solutions are not relevant, the attention network learns to give high weight to the base network solution (which can be learnt and improved), thus avoiding negative transfer.
Depending on the feedback obtained from the environment upon following KT , the attention network’s parameters θa are updated to improve performance.
As mentioned earlier, the source task solutions, K1, . . . ,KN remain fixed. Updating these source task’s parameters would cause a significant amount of unlearning in the source tasks solutions and result in a weaker transfer, which we observed empirically. This also enables the use of source task solutions, as long as we have the outputs alone, irrespective of how and where they come from.
Even though the agent follows KT , we update the parameters of the base network that produces KB , as if the action taken by the agent was based only on KB . Due to this special way of updating KB , apart from the experience got through the unique and individual contribution of KB to KT in parts of the state space where the source task solutions are not relevant, KB also uses the valuable experience got by using KT which uses the solutions of the source tasks as well.
This also means that, if there is a source task whose solution Kj is useful for the target task in some parts of its state space, then KB tries to replicate Kj in those parts of the state space. In practise, the source task solutions though useful, might need to be modified to suit perfectly for the target task. The base network takes care of these modifications required to make the useful source task solutions perfect for the target task. The special way of training the base network assists the architecture in achieving this faster. Note that the agent could follow/useKj throughKT even when KB does not attain its replication in the corresponding parts of the state space. This allows for a good performance of the agent in earlier stages training itself, when a useful source task is available and identified.
Since the attention is soft, our model has the flexibility to combine multiple solutions. The use of deep neural networks allow the model to work even for large, complex RL problems. The deep attention network, allows the agent to learn complex selection functions, without worrying about
representation issues a priori. To summarise, for a given state, A2T learns to attend to specific solutions and adapts this attention over different states, hence attaining useful transfer. A2T is general and can be used for transfer of solutions such as policy and value.
3.1 POLICY TRANSFER
The solutions that we transfer here are the source task policies, taking advantage of which, we learn a policy for the target task. Thus, we have K1, . . . ,KN ,KB ,KT ← π1, . . . πN , πB , πT . Here π represents a stochastic policy, a probability distribution over all the actions. The agent acts in the target task, by sampling actions from the probability distribution πT . The target task policy πT is got as described in Eq.(1) and Eq.(2). The attention network that produces the weights for the different solutions, is trained by the feedback got after taking action following πT . The base network that produces πB is trained as if the sampled action came from πB (though it originally came from πT ), the implications of which were discussed in the previous section. When the attention network’s weight for the policy πB is high, the mixture policy πT is dominated by πB , and the base network learning is nearly on-policy. In the other cases, πB undergoes off-policy learning. But if we look closely, even in the latter case, since πB moves towards πT , it tries to be nearly on-policy all the time. Empirically, we observe that πB converges. This architecture for policy transfer can be used alongside any algorithm that has an explicit representation of the policy. Here we describe two instantiations of A2T for policy transfer, one for direct policy search using REINFORCE algorithm and another in the Actor-Critic setup.
3.1.1 POLICY TRANSFER IN REINFORCE ALGORITHMS USING A2T:
REINFORCE algorithms [Williams (1992)] can be used for direct policy search by making weight adjustments in a direction that lies along the gradient of the expected reinforcement. The full architecture is same as the one shown in Fig.1a with K ← π. We do direct policy search, and the parameters are updated using REINFORCE. Let the attention network be parametrized by θa and the base network which outputs πB be parametrized by θb. The updates are given by:
θa ← θa + αθa(r − b) ∂ ∑M t=1 log(πT (st, at))
∂θa (5)
θb ← θb + αθb(r − b) ∂ ∑M t=1 log(πB(st, at))
∂θb (6)
where αθa , αθb are non-negative factors, r is the return obtained in the episode, b is some baseline and M is the length of the episode. at is the action sampled by the agent at state st following πT . Note that while πT (st, at) is used in the update of the attention network, πB(st, at) is used in the update of the base network.
3.1.2 POLICY TRANSFER IN ACTOR-CRITIC USING A2T:
Actor-Critic methods [Konda & Tsitsiklis (2000)] are Temporal Difference (TD) methods that have two separate components, viz., an actor and a critic. The actor proposes a policy whereas the critic estimates the value function to critique the actor’s policy. The updates to the actor happens through TD-error which is the one step estimation error that helps in reinforcing an agent’s behaviour.
We use A2T for the actor part of the Actor-Critic. The architecture is shown in Fig.1b. The actor, A2T is aware of all the previous learnt tasks and tries to use those solution policies for its benefit. The critic evaluates the action selection from πT on the basis of the performance on the target task. With the same notations as REINFORCE for st, at, θa, θb, αθa , αθb , πB , πT ; let action at dictated by πT lead the agent to next state st+1 with a reward rt+1 and let V (st) represent the value of state st and γ the discount factor. Then, the update equations for the actor are as below:
δt = rt+1 + γV (st+1)− V (st) (7)
θa ← θa + αθaδt ∂ log πT (st,at) ∂θa∣∣∣∂ log πT (st,at)∂θa ∣∣∣ (8)
θb ← θb + αθbδt ∂ log πB(st,at) ∂θb∣∣∣∂ log πB(st,at)∂θb ∣∣∣ (9) Here, δt is the TD error. The state-value function V of the critic is learnt using TD learning.
3.2 VALUE TRANSFER
In this case, the solutions being transferred are the source tasks’ action-value functions, which we will call as Q functions. Thus, K1, . . . ,KN ,KB ,KT ← Q1, . . . , QN , QB , QT . Let A represent the discrete action space for the tasks and Qi(s) = {Q(s, aj) ∀ aj ∈ A}. The agent acts by using QT in the target task, which is got as described in Eq.(1) and Eq.(2). The attention network and the base network of A2T are updated as described in the architecture.
3.2.1 VALUE TRANSFER IN Q LEARNING USING A2T:
The state-action value Q function is used to guide the agent to selecting the optimal action a at a state s, whereQ(s, a) is a measure of the long-term return obtained by taking action a at state s. One way to learn optimal policies for an agent is to estimate the optimal Q(s, a) for the task. Q-learning [Watkins & Dayan (1992)] is an off-policy Temporal Difference (TD) learning algorithm that does so. The Q-values are updated iteratively through the Bellman optimality equation [Puterman (1994)] with the rewards obtained from the task as below:
Q(s, a)← E[r(s, a, s′) + γmaxa′Q(s′, a′)]
In high dimensional state spaces, it is infeasible to update Q-value for all possible state-action pairs. One way to address this issue is by approximating Q(s, a) through a parametrized function approximator Q(s, a; θ),thereby generalizing over states and actions by operating on higher level features [Sutton & Barto (1998)]. The DQN [Mnih et al. (2015)] approximates the Q-value function with a deep neural network to be able to predict Q(s, a) over all actions a, for all states s.
The loss function used for learning a Deep Q Network is as below: L(θ) = Es,a,r,s′ [ ( yDQN −Q(s, a; θ) )2 ],
with yDQN = ( r + γmaxa′Q(s′, a′, θ−) ) Here, L represents the expected TD error corresponding to current parameter estimate θ. θ− represents the parameters of a separate target network, while θ represents the parameters of the online network. The usage of a target network is to improve the stability of the learning updates. The gradient descent step is shown below:
∇θL(θ) = Es,a,r,s′ [(yDQN −Q(s, a; θ))∇θQ(s, a)]
To avoid correlated updates from learning on the same transitions that the current network simulates, an experience replay [Lin (1993)] D (of fixed maximum capacity) is used, where the experiences are pooled in a FIFO fashion.
We use DQN to learn our expertsQi, i ∈ 1, 2 . . . N on the source tasks. Q-learning is used to ensure QT (s) is driven to a good estimate of Q functions for the target task. Taking advantage of the offpolicy nature of Q-learning, both QB and QT can be learned from the experiences gathered by an -greedy behavioral policy based on QT . Let the attention network that outputs w be parametrised by θa and the base network outputting QB be parametrised by θb. Let θa− and θb− represent the parameters of the respective target networks. Note that the usage of target here is to signify the parameters (θ−a , θ − b ) used to calculate the target value in the Q-learning update and is different from its usage in the context of the target task. The update equations are:
yQT = (r + γmaxa′QT (s′, a′; θa−, θb−)) (10)
LQT (θa, θb) = Es,a,r,s′ [(yQT −QT (s, a; θa, θb))2] (11)
LQB (θb) = Es,a,r,s′ [(yQT −QB(s, a; θb))2] (12)
∇θaLQT = E[(yQT −QT (s, a))∇θaQT (s, a)] (13)
∇θbLQB = E[(yQT −QB(s, a))∇θbQR(s, a)] (14) θa and θb are updated with the above gradients using RMSProp. Note that the Q-learning updates for both the attention network (Eq.(11)) and the base network (Eq.(12)) use the target value generated by QT . We use target networks for both QB and QT to stabilize the updates and reduce the nonstationarity as in DQN training. The parameters of the target networks are periodically updated to that of the online networks.
4 EXPERIMENTS AND DISCUSSION
We evaluate the performance of our architecture A2T on policy transfer using two simulated worlds, viz., chain world and puddle world as described below. The main goal of these experiments is to test the consistency of results with the algorithm motivation. Chain world: Figure 2a shows the chain world where the goal of the agent is to go from one point in the chain (starting state) to another point (goal state) in the least number of steps. At each state the agent can choose to either move one position to the left or to the right. After reaching the goal state the agent gets a reward that is inversely proportional to the number of steps taken to reach the goal.
Puddle worlds: Figures 2b and 2c show the discrete version of the standard puddle world that is widely used in Reinforcement Learning literature. In this world, the goal of the agent is to go from a specified start position to the goal position, maximising its return. At each state the agent can choose one of these four actions: move one position to the north, south, east or west.With 0.9 probability the agent moves in the chosen direction and with 0.1 probability it moves in a random direction irrespective of its choice of action. On reaching the goal state, the agent gets a reward of +10. On reaching other parts of the grid the agent gets different penalties as mentioned in the legend of the figures. . We evaluate the performance of our architecture on value transfer using the Arcade Learning Environment (ALE) platform [Bellemare et al. (2012)]. Atari 2600: ALE provides a simulator for Atari 2600 games. This is one of the most commonly used benchmark tasks for deep reinforcement learning algorithms [Mnih et al. (2015), Mnih et al. (2016), Parisotto et al. (2015), Rusu et al. (2016)]. We perform our adaptive transfer learning experiments on the Atari 2600 game Pong.
4.1 ABILITY TO DO SELECTIVE TRANSFER
In this section, we consider the case when multiple partially favorable source tasks are available such that each of them can assist the learning process for different parts of the state space of the target task. The objective here is to first show the effectiveness of the attention network in learning to focus only on the source task relevant to the state the agent encounters while trying to complete the target task and then evaluating the full architecture with an additional randomly initialised base network.
This is illustrated for the Policy Transfer setting using the chain world shown in (Fig. 2a). Consider that the target task LT is to start in A or B with uniform probability and reach C in the least number of steps. Now, consider that two learned source tasks, viz., L1 and L2, are available. L1 is the source task where the agent has learned to reach the left end (A) starting from the right end (B). In contrast, L2 is the source task where the agent has learned to reach the right end (B) starting from the left end (A). Intuitively, it is clear that the target task should benefit from the policies learnt for tasks L1 and L2. We learn to solve the task LT using REINFORCE given the policies learned for L1 and L2. Figure 3a (i) shows the weights given by the attention network to the two source task policies for different parts of the state space at the end of learning. We observe that the attention network has learned to ignore L1, and L2 for the left, and right half of the state space of the target task, respectively. Next, we add base network and evaluate the full architecture on this task. Figure 3a (ii) shows the weights given by the attention network to the different source policies for different parts of the state space at the end of learning. We observe that the attention network has learned to ignore L1, and L2 for the left, and right half of the state space of the target task, respectively. As the base network replicates πT over time, it has a high weight throughout the state space of the target task.
We also evaluate our architecture in a relatively more complex puddle world shown in Figure 2c. In this case, L1 is the task of moving from S1 to G1, and L2 is the task of moving from S2 to G1. In the target task LT , the agent has to learn to move to G1 starting from either S1 or S2 chosen with uniform probability. We learn the task LT using Actor-Critic method, where the following are available (i) learned policy for L1 (ii) learned policy for L2 and (iii) a randomly initialized policy network (the base network). Figure 3b shows the performance results. We observe that actor-critic using A2T is able to use the policies learned for L1, and L2 and performs better than a network learning from scratch without any knowledge of source tasks.
We do a similar evaluation of the attention network, followed by our full architecture for value transfer as well. We create partially useful source tasks through a modification of the Atari 2600 game Pong. We take inspiration from a real world scenario in the sport Tennis, where one could imagine two different right-handed (or left) players with the first being an expert player on the forehand but weak on the backhand, while the second is an expert player on the backhand but weak on the forehand. For someone who is learning to play tennis with the same style (right/left) as the experts, it is easy to follow the forehand expert player whenever he receives a ball on the forehand and follow the backhand expert whenever he receives a ball on the backhand.
We try to simulate this scenario in Pong. The trick is to blur the part of the screen where we want to force the agent to be weak at returning the ball. The blurring we use is to just black out all pixels in the specific region required. To make sure the blurring doesn’t contrast with the background, we modify Pong to be played with a black background (pixel value 0) instead of the existing gray (pixel value 87). We construct two partially helpful source task experts L1 and L2. L1 is constructed by
training a DQN on Pong with the upper quadrant (the agent’s side) blurred, while L2 is constructed by training a DQN with the lower quadrant (the agent’s side) blurred. This essentially results in the ball being invisible when it is in the upper quadrant for L1 and lower quadrant for L2. We therefore expect L1 to be useful in guiding to return balls on the lower quadrant, and L2 for the upper quadrant. The goal of the attention network is to learn suitable filters and parameters so that it will focus on the correct source task for a specific situation in the game. The source task experts L1 and L2 scored an average of 9.2 and 8 respectively on Pong game play with black background. With an attention network to suitably weigh the value functions of L1 and L2, an average performance of 17.2 was recorded just after a single epoch (250,000 frames) of training. (The score in Pong is in the range of [−21, 21]). This clearly shows that the attention mechanism has learned to take advantage of the experts adaptively. Fig. 4 shows a visualisation of the attention weights for the same.
We then evaluate our full architecture (A2T) in this setting, i.e with an addition of DQN learning from scratch (base network) to the above setting. The architecture can take advantage of the knowledge of the source task experts selectively early on during the training while using the expertise of the base network wherever required, to perform well on the target task. Figure 5 summarizes the results, where it is clear that learning with both the partially useful experts is better than learning with only one of them which in turn is better than learning from scratch without any additional knowledge.
4.2 ABILITY TO AVOID NEGATIVE TRANSFER AND ABILITY
TO TRANSFER FROM FAVORABLE TASK
We first consider the case when only one learned source task is available such that its solution K1 (policy or value) can hamper the learning process of the new target task. We refer to such a source task as an unfavorable source task. In such a scenario, the attention network shown in Figure 1a should learn to assign a very low weight (ignore) to K1 . We also consider a modification of this setting by adding another source task whose solution K2 is favorable to the target task. In such a scenario, the attention network should learn to assign high weight (attend) to K2 while ignoringK1.
We now define an experiment using the puddle world from Figure 2b for policy transfer. The target task in our experiment is to maximize the return in reaching the goal state G1 starting from any one of the states S1, S2, S3, S4. We artificially construct an unfavorable source task by first learning to solve the above task and then negating the weights of the topmost layer of the actor network. We then add a favorable task to the above setting. We artificially construct a favorable source task
simply by learning to solve the target task and using the learned actor network. Figure 6 shows the results. The target task for the value transfer experiment is to reach expert level performance on Pong. We construct two kinds of unfavorable source tasks for this experiment. Inverse-Pong: A DQN on Pong trained with negated reward functions, that is with R′(s, a) = −R(s, a) where R(s, a) is the reward provided by the ALE emulator for choosing action a at state s. Freeway: An expert DQN on another Atari 2600 game, Freeway, which has the same range of optimal value functions and same action space as Pong. We empirically verified that the Freeway expert DQN leads to negative transfer when directly initialized and fine-tuned on Pong which makes this a good proxy for a negative source task expert even though the target task Pong has a different state space.
We artificially construct a favorable source task by learning a DQN to achieve expertise on the target task (Pong) and use the learned network. Figure 7a compares the performance of the various scenarios when the unfavorable source task is Inverse-Pong, while Figure 7b offers a similar comparison with the negative expert being Freeway.
From all the above results, we can clearly see that A2T does not get hampered by the unfavorable source task by learning to ignore the same and performs competitively with just a randomly initialized learning on the target task without any expert available. Secondly, in the presence of an additional source task that is favorable, A2T learns to transfer useful knowledge from the same while ignoring the unfavorable task, thereby reaching expertise on the target task much faster than the other scenarios.
4.3 VISUALIZATION: EVOLUTION OF ATTENTION WEIGHTS WITH ONE POSITIVE AND ONE NEGATIVE EXPERT
We present the evolution of attention weights for the experiment described in Section 4.2 where we focus on the efficacy of the A2T framework in providing an agent the ability to avoid negative transfer and transfer from a favorable source task (perfect expert). Figure 8 depicts the evolution of
the attention weights (normalised in the range of [0, 1]) during the training of the A2T framework. The corresponding experiment is the case where the target task is to solve Pong, while there are two source task experts, one being a perfect Pong playing trained DQN (to serve as positive expert), and the other being the Inverse-Pong DQN trained with negated reward functions (to serve as negative expert). Additionally, there’s also the base network that learns from scratch using the experience gathered by the attentively combined behavioral policy from the expert networks, the base network and itself.
We train the framework for 30 epochs, and the plot illustrates the attention weights every second epoch. We clearly see from figure 8 that there is no weird co-adaptation that happens in the training, and the attention on the negative expert is uniformly low throughout. Initially, the framework needs to collect some level of experience to figure out that the positive expert is optimal (or close to optimal). Till then, the attention is mostly on the base network, which is learning from scratch. The attention then shifts to the positive expert which in turn provides more rewarding episodes and transition tuples to learn from. Finally, the attention drifts slowly to the base network from the positive expert again, after which the attention is roughly random in choosing between the execution of positive expert and the base network. This is because the base network has acquired sufficient expertise as the positive expert which happens to be optimal for the tar-
get task. This visualization clearly shows that A2T is a powerful framework in ignoring a negative expert throughout and using a positive expert appropriately to learn quickly from the experience gathered and acquire sufficient expertise on the target task.
4.4 WHEN A PERFECT EXPERT IS NOT AVAILABLE AMONG THE SOURCE TASKS
In our experiments in the previous subsection dealing with prevention of negative transfer and using a favorable source task, we consider the positive expert as a perfect (close to optimal) expert on the same task we treat as the target task. This raises the question of relying on the presence of a perfect expert as a positive expert. If we have such a situation, the obvious solution is to execute each of the experts on the target task and vote for them with probabilities proportional to the average performance of each.
The A2T framework is however generic and not intended to just do source task selection. We illustrate this with an additional baseline experiment, where the positive source task is an imperfect expert on the target task. In such a case, just having a weighted average voting among the available source task networks based on their individual average rewards is upper bounded by the
performance of the best available positive expert, which happens to be an imperfect expert on the target task. Rather, the base network has to acquire new skills not present in the source task networks. We choose a partially trained network on Pong, that scores an average of 8 (max: 21). The graph in figure 9 clearly shows that the A2T framework with a partial Pong expert and a negative expert performs better than i) learning from scratch, ii) A2T with only one negative expert, and performs worse than A2T with one perfect positive expert and one negative expert. This is expected because
a partial expert cannot provide as much of expert knowledge as a perfect expert, but still provides some useful knowledge in speeding the process of solving the target task. An important conclusion from this experiment is that the A2T framework is capable of discovering new skills not available among any of the experts when such skills are required for optimally solving the target task. To maintain consistency, we perform the same number of runs for averaging scores and experimented with both learning rates and pick the better performing one (0.00025).
5 CONCLUSION AND FUTURE WORK
In this paper we present a very general deep neural network architecture, A2T, for transfer learning that avoids negative transfer while enabling selective transfer from multiple source tasks in the same domain. We show simple ways of using A2T for policy transfer and value transfer. We empirically evaluate its performance with different algorithms, using simulated worlds and games, and show that it indeed achieves its stated goals. Apart from transferring task solutions, A2T can also be used for transferring other useful knowledge such as the model of the world.
While in this work we focused on transfer between tasks that share the same state and action spaces and are in the same domain, the use of deep networks opens up the possibility of going beyond this setting. For example, a deep neural network can be used to learn common representations [Parisotto et al. (2015)] for multiple tasks thereby enabling transfer between related tasks that could possibly have different state-action spaces. A hierarchical attention over the lower level filters across source task networks while learning the filters for the target task network is another natural extension to transfer across tasks with different state-action spaces. The setup from Progressive Neural Networks [Rusu et al. (2016)] could be borrowed for the filter transfer, while the A2T setup can be retained for the policy/value transfer. Exploring this setting for continuous control tasks so as to transfer from modular controllers as well avoid negative transfer is also a potential direction for future research.
The nature of tasks considered in our experiments is naturally connected to Hierarchical Reinforcement Learning and Continual Learning. For instance, the blurring experiments inspired from Tennis based on experts for specific skills like Forehand and Backhand could be considered as learning from sub-goals (program modules) like Forehand and Backhand to solve a more complex and broader task like Tennis by invoking the relevant sub-goals (program modules). This structure could be very useful to build a household robot for general purpose navigation and manipulation whereby specific skills such as manipulation of different objects, navigating across different source-destination points, etc could be invoked when necessary. The attention network in the A2T framework is essentially a soft meta-controller and hence presents itself as a powerful differentiable tool for Continual and Meta Learning. Meta-Controllers have typically been been designed with discrete decision structure over high level subgoals. This paper presents an alternate differentiable meta-controller with a soft-attention scheme. We believe this aspect can be exploited for differentiable meta-learning architectures for hierarchical reinforcement learning. Over all, we believe that A2T is a novel way to approach different problems like Transfer Learning, Meta-Learning and Hierarchical Reinforcement Learning and further refinements on top of this design can be a good direction to explore.
ACKNOWLEDGEMENTS
Thanks to the anonymous reviewers of ICLR 2017 who have provided thoughtful remarks and helped us revise the paper. We would also like to thank Sherjil Ozair, John Schulman, Yoshua Bengio, Sarath Chandar, Caglar Gulchere and Charu Chauhan for useful feedback about the work.
APPENDIX A: DETAILS OF THE NETWORK ARCHITECTURE IN VALUE TRANSFER EXPERIMENTS
For the source task expert DQNs, we use the same architecture as [Mnih et al. (2015)] where the input is 84 × 84 × 4 with 32 convolution filters, dimensions 8 × 8, stride 4 × 4 followed by 64 convolution filters with dimensions 4× 4 and stride 2× 2, again followed by 64 convolution filters of size 3×3 and stride 1×1. This is then followed by a fully connected layer of 512 units and finally by a fully connected output layer with as many units as the number of actions in Pong (Freeway) which is 3. We use ReLU nonlinearity in all the hidden layers.
With respect to the A2T framework architecture, we have experimented with two possible architectures:
• The base and attention networks following the NIPS architecture of Mnih et al. (2013) except that the output layer is softmax for the attention network.
• The base and attention networks following the Nature architecture of Mnih et al. (2015) with a softmax output layer for the attention network.
Specifically, the NIPS architecture of Mnih et al. (2013) takes in a batch of 84 × 84 × 4 inputs, followed by 16 convolution filters of dimensions 8× 8 with stride 4× 4, 32 convolution filters with dimensions 4 × 4 and stride 2 × 2, a fully connected hidden layer of 256 units, followed by the output layer. For the Selective Transfer with Blurring experiments described in Section 4.1, we use the second option above. For the other experiments in Section 4.2 and the additional experiments in Appendix, we use the first option. The attention network has N + 1 outputs where N is the number of source tasks.
APPENDIX B: TRAINING DETAILS
TRAINING ALGORITHM
For all our experiments in Value Transfer, we used RMSProp as in [Mnih et al. (2015)] for updating gradient. For Policy Transfer, since the tasks were simple, stochastic gradient descent was sufficient to provide stable updates. We also use reward clipping, target networks and experience replay for our value transfer experiments in exactly the same way (all hyper parameters retained) as [Mnih et al. (2015)]. A training epoch is 250,000 frames and for each training epoch, we evaluate the networks with a testing epoch that lasts 125,000 frames. We report the average score over the completed episodes for each testing epoch. The average scores obtained this way are averaged over 2 runs with different random seeds. In the testing epochs, we use = 0.05 in the -greedy policy.
LEARNING RATE
In all our experiments, we trained the architecture using the learning rates, 0.0025 and 0.0005. In general, the lower learning rate provided more stable (less variance) training curves. While comparing across algorithms, we picked the best performing learning rate out of the two (0.0025 and 0.0005) for each training curve.
APPENDIX C: BLURRING EXPERIMENTS ON PONG
The experts are trained with blurring (hiding the ball) and black background as illustrated in APPENDIX A. Therefore, to compare the learning with that of a random network without any additional knowledge, we ran the baseline DQN on Pong with a black background too. Having a black background provides a rich contrast between the white ball and the black background, thereby making training easier and faster, which is why the performance curves in that setting are different to the other two settings reported for Inverse Pong and Freeway Negative transfer experiments where no blacking is done and Pong is played with a gray background. The blurring mechanism in Pong is illustrated in APPENDIX E.
APPENDIX E: BLURRING MECHANISM IN PONG - DETAILS
APPENDIX D: BLURRING EXPERIMENTS ON BREAKOUT
Similar to our Blurring experiment on Pong, we additionally ran another experiment on the Atari 2600 game, Breakout, to validate the efficiency of our attention mechanism. We consider a setup with two experts L1 and L2 along with our attention network. The experts L1 and L2 were trained by blurring the lower left and right quadrants of the breakout screen respectively. We don’t have to make the background black like in the case of Pong because the background is already black in Breakout and direct blurring is sufficient to hiding the ball in the respective regions without any contrasts introduced. We blur only the lower part so as to make it easy for the agent to at least anticipate the ball based on the movement at the top. We empirically observed that blurring the top half (as well) makes it hard to learn any meaningful partially useful experts L1 and L2.
The goal of this experiment is to show that the attention network can learn suitable filters so as to dynamically adapt and learn to select the expert appropriate to the situation (game screen) in the task. The expert L1 which was blurred on the left bottom half is bound to weak at returning balls on that region while L2 is expected to be weak on the right. This is in the same vein as the forehandbackhand example in Tennis and its synthetic simulation for Pong by blurring the upper and lower quadrants. During game play, the attention mechanism is expected to ignore L2 when the ball is on the bottom right half (while focusing on L1) and similarly ignore L2 (while focusing on L1) when the ball is on the left bottom half. We learn experts L1 and L2 which score 42.2 and 39.8 respectively. Using the attention mechanism to select the correct expert, we were able to achieve a score of 94.5 after training for 5 epochs. Each training epoch corresponds to 250, 000 decision steps, while the scores are averaged over completed episodes run for 125, 000 decision steps. This shows that the attention mechanism learns to select the suitable expert. Though the performance is limited by the weaknesses of the respective experts, our goal is to show that the attention paradigm is able to take advantage of both experts appropriately. This is evident from the scores achieved by standalone experts and the attention mechanism. Additionally, we also present a visualization of the attention mechanism weights assigned to the experts L1 and L2 during game play in APPENDIX G. The weights assigned are in agreement with what we expect in terms of selective attention. The blurring mechanism is visually illustrated in APPENDIX F.
APPENDIX G: BLURRING ATTENTION VISUALIZATION ON BREAKOUT
APPENDIX F: BLURRING MECHANISM IN BREAKOUT - DETAILS
APPENDIX J: CASE STUDY OF TARGET TASK PERFORMANCE LIMITED BY DATA AVAILABILITY

Review:
In this paper a well known soft mixture of experts model is adapted for, and applied to, a specific type of transfer learning problem in reinforcement learning (RL), namely transfer of action policies and value functions between similar tasks. Although not treated as such, the experimental setup is reminiscent of hierarchical RL works, an aspect which the paper does not consider at length, regrettably.
One possible implication of this work is that architecture and even learning algorithm choices could simply be stated in terms of the objective of the target task, rather than being hand-engineered by the experimenter. This is clearly an interesting direction of future work which the paper illuminates.
Pros:
The paper diligently explains how the network architecture fits in with various widely used reinforcement learning setups, which does facilitate continuation of this work.
The experiments are good proofs of concept, but do not go beyond that i.m.h.o. 
Even so, this work provides convincing clues that collections of deep networks, which were trained on not entirely different tasks, generalize better to related tasks when used together rather than through conventional transfer learning (e.g. fine-tuning).
Cons:
As the paper well recounts in the related work section, libraries of fixed policies have long been formally proposed for reuse while learning similar tasks. Indeed, it is well understood in hierarchical RL literature that it can be beneficial to reuse libraries of fixed (Fernandez & Veloso 2006) or jointly learned policies which may not apply to the entire state space, e.g. options (Pricop et. al). What is not well understood is how to build such libraries, and this paper does not convincingly shed light in that direction, as far as I can tell.
The transfer tasks have been picked to effectively illustrate the potential of the proposed architecture, but the paper does not tackle negative transfer or compositional reuse in well known challenging situations outlined in previous work (e.g. Parisotto et. al 2015, Rusu el. al 2015, 2016).
Since the main contributions are of an empirical nature, I am curious how the results shown in figures 6 & 7 look plotted against wall-clock time, since relatively low data efficiency is not a limitation for achieving perfect play in Pong (see Mnih. et al, 2015). It would be more illuminating to consider tasks where final performance is plausibly limited by data availability. It would also be interesting if the presented results were achieved with reduced amounts of computation, or reduced representation sizes compared to learning from scratch, especially when one of the useful source tasks is an actual policy trained on the target task.
Finally, it is perhaps underwhelming that it takes a quarter of the data required for learning Pong from scratch just to figure out that a perfect Pong policy is already in the expert library. Simply evaluating each expert for 10 episodes and using an  average-score-weighted majority vote to mix action choices would probably achieve the same final performance for a smaller fraction of the data.