Original: R OUTING N ETWORKS : A DAPTIVE S ELECTION OF N ON - LINEAR F UNCTIONS FOR M ULTI -T ASK L EARNING
Revised: R OUTING N ETWORKS : A DAPTIVE S ELECTION OF N ON - LINEAR F UNCTIONS FOR M ULTI -T ASK L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces transfer.
Revised: Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces transfer.

Original: To address this issue we introduce the routing network paradigm, a novel neural network unit and training algorithm.
Revised: To address this issue we introduce the routing network paradigm, a novel neural network unit and training algorithm.

Original: A routing network is a kind of selforganizing neural network consisting of two components: a router and a set of one or more function blocks .
Revised: A routing network is a kind of selforganizing neural network consisting of two components: a router and a set of one or more function blocks .

Original: A function block may be any neural network – for example a fully-connectedlayer or a a block of a wide residual network.
Revised: A function block may be any neural network – for example a fully-connected or a convolutional layer.

Original: Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when the router decides to stop or a ﬁxed recursion depth is reached.
Revised: Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when the router decides to stop or a ﬁxed recursion depth is reached.

Original: In this way the routing network dynamically composes different function blocks for each input.
Revised: In this way the routing network dynamically composes different function blocks for each input.

Original: We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks.
Revised: We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks.

Original: We evaluate our model on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets.
Revised: We evaluate our model on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets

Original: Our experiments demonstrate signiﬁcant improvement in accuracy with sharper convergence over both single task and joint training baselines for these tasks.
Revised: Our experiments demonstrate signiﬁcant improvement in accuracy with sharper convergence over both single task and joint training baselines for these tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-task learning (MTL) is a paradigm in which multiple tasks must be learned simultaneously.
Revised: Multi-task learning (MTL) is a paradigm in which multiple tasks must be learned simultaneously.

Original: Tasks are typically separate prediction problems, each with their own data distribution.
Revised: Tasks are typically separate prediction problems, each with their own data distribution.

Original: In an early formulation of the problem, (Caruana, 1997) describes the goal of MTL as improving generalization performance by “leveraging the domain-speciﬁc information contained in the training signals of related tasks.”
Revised: In an early formulation of the problem, (Caruana, 1997) describes the goal of MTL as improving generalization performance by “leveraging the domain-speciﬁc information contained in the training signals of related tasks.”

Original: This means a model must leverage commonalities in the tasks (positive transfer) while minimizing interference (negative transfer).
Revised: This means a model must leverage commonalities in the tasks (positive transfer) while minimizing interference (negative transfer).

Original: In this paper we propose a new architecture for
Revised: In this paper we propose a new architecture for

Original: MTL problems called a routing network , which consists of two trainable components: a router and a set of function block s. Given an input, the router selects a function block from the set, applies it to the input, and passes the result back to the Router, recursively.
Revised: MTL problems called a routing network , which consists of two trainable components: a router and a set of function block s. Given an input, the router selects a function block from the set, applies it to the input, and passes the result back to the Router, recursively.

Original: The router can decide at any point to stop routing and is in practice limited to a ﬁxed recursion limit.
Revised: The router can decide at any point to stop routing and is in practice limited to a ﬁxed recursion limit.

Original: Intuitively, the architecture allows the network to dynamically self-organize in response to the input, sharing function blocks for different tasks when positive transfer is possible, and using separate blocks to prevent negative transfer.
Revised: Intuitively, the architecture allows the network to dynamically self-organize in response to the input, sharing function blocks for different tasks when positive transfer is possible, and using separate blocks to prevent negative transfer.

Original: The architecture is very general allowing many possible Router implementations.
Revised: The architecture is very general allowing many possible Router implementations.

Original: For example, the router can condition its decision on both the input instance and a task label or just one or the other.
Revised: For example, the router can condition its decision on both the input instance and a task label or just one or the other.

Original: It can also condition on the depth (number of router invocations), ﬁltering the function module choices to allow layering.
Revised: It can also condition on the depth (number of router invocations), ﬁltering the function module choices to allow layering.

Original: In addition, it can condition its decision for one instance on what was historically decided for other instances, to encourage collaboration.
Revised: In addition, it can condition its decision for one instance on what was historically decided for other instances, to encourage collaboration.

Original: The function blocks may be simple fullyconnected neural network layers or whole networks as long as the dimensionality of each function block allows composition with the previous function block choice.
Revised: The function blocks may be simple fullyconnected neural network layers or whole networks as long as the dimensionality of each function block allows composition with the previous function block choice.

Original: They needn’t even be the same type of unit.
Revised: They needn’t even be the same type of unit.

Original: Any neural network or part of a network can be “routed” by adding its layers to the set of function blocks, making the architecture applicable to a wide range of problems.
Revised: Any neural network or part of a network can be “routed” by adding its layers to the set of function blocks, making the architecture applicable to a wide range of problems.

Original: Because the routers make a sequence of hard decisions, which are not differentiable, we use reinforcement learning (RL) to train them.
Revised: Because the routers make a sequence of hard decisions, which are not differentiable, we use reinforcement learning (RL) to train them.

Original: We discuss the training algorithm in Section 3.1, but one way we have modeled this as an RL problem is to create a separate RL agent for each task (assuming task labels are available in the dataset).
Revised: We discuss the training algorithm in Section 3.1, but one way we have modeled this as an RL problem is to create a separate RL agent for each task (assuming task labels are available in the dataset).

Original: Each such task agent learns its own policy for routing instances of that task to one of the function blocks.
Revised: Each such task agent learns its own policy for routing instances of that task to one of the function blocks.

Original: To evaluate this architecture we have created a “routed” version of the convnet architecture used in (Ravi & Larochelle, 2017) and use three image classiﬁcation datasets adapted for MTL learning: a multi-task MNIST dataset that we created, a Mini-imagenet data split as introduced in (Vinyals et al., 2016), and the 20 superclasses of CIFAR-100 (Krizhevsky, 2009) each treated as different tasks.
Revised: To evaluate this architecture we have created a “routed” version of the convnet architecture used in (Ravi & Larochelle, 2017) and use three image classiﬁcation datasets adapted for MTL learning: a multi-task MNIST dataset that we created, a Mini-imagenet data split as introduced in (Vinyals et al., 2016), and the 20 superclasses of CIFAR-100 (Krizhevsky, 2009) each treated as different tasks.

Original: We conduct extensive experiments comparing against both single task and the popular strategy of joint training with layer sharing as described in Caruana (1997).
Revised: We conduct extensive experiments comparing against both single task and the popular strategy of joint training with layer sharing as described in Caruana (1997).

Original: Our results indicate a signiﬁcant improvement in accuracy over these strong baselines with a speedup in convergence.
Revised: Our results indicate a signiﬁcant improvement in accuracy over these strong baselines with a speedup in convergence.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Work on multi-task deep learning (Caruana, 1997) traditionally includes signiﬁcant hand design of neural network architectures, attempting to ﬁnd the right static mix of task-speciﬁc and shared parameters.
Revised: Work on multi-task deep learning (Caruana, 1997) traditionally includes signiﬁcant hand design of neural network architectures, attempting to ﬁnd the right static mix of task-speciﬁc and shared parameters.

Original: For example, many architectures share low-level features like those learned in shallow layers of deep convolutional networks or word embeddings across tasks and add task speciﬁc architectures in later layers.
Revised: For example, many architectures share low-level features like those learned in shallow layers of deep convolutional networks or word embeddings across tasks and add task speciﬁc architectures in later layers.

Original: By contrast, in the Routing Networks paradigm, we learn a fully dynamic, compositional model which can adjust its structure differently for each task.
Revised: By contrast, in the Routing Networks paradigm, we learn a fully dynamic, compositional model which can adjust its structure differently for each task.

Original: Routing Networks share a common goal with techniques for automated selective transfer learning using attention (Rajendran et al., 2017) and learning gating mechanisms between representations (Stollenga et al., 2014), (Misra et al., 2016), (Ruder et al., 2017).
Revised: Routing Networks share a common goal with techniques for automated selective transfer learning using attention (Rajendran et al., 2017) and learning gating mechanisms between representations (Stollenga et al., 2014), (Misra et al., 2016), (Ruder et al., 2017).

Original: However, these techniques have not been shown to scale to large numbers of routing decisions and tasks.
Revised: However, these techniques have not been shown to scale to large numbers of routing decisions and tasks.

Original: Our work is also related to mixtures of experts architectures (Jacobs et al., 1991), (Jordan & Jacobs, 1994) as well as their modern attention based (Riemer et al., 2016) and sparse (Shazeer et al., 2017) variants.
Revised: Our work is also related to mixtures of experts architectures (Jacobs et al., 1991), (Jordan & Jacobs, 1994) as well as their modern attention based (Riemer et al., 2016) and sparse (Shazeer et al., 2017) variants.

Original: The gating network in a typical mixtures of experts model takes in the input and chooses an appropriate weighting for the output of each expert network.
Revised: The gating network in a typical mixtures of experts model takes in the input and chooses an appropriate weighting for the output of each expert network.

Original: This is generally implemented as a soft mixture decision as opposed to a hard routing decision, allowing the choice to be differentiable.
Revised: This is generally implemented as a soft mixture decision as opposed to a hard routing decision, allowing the choice to be differentiable.

Original: Although the sparse and layer-wise variant presented in (Shazeer et al., 2017) does save some computational burden, the proposed end-to-end differentiable model is only an approximation and doesn’t model important effects such as exploration vs. exploitation tradeoffs, despite their impact on the system.
Revised: Although the sparse and layer-wise variant presented in (Shazeer et al., 2017) does save some computational burden, the proposed end-to-end differentiable model is only an approximation and doesn’t model important effects such as exploration vs. exploitation tradeoffs, despite their impact on the system.

Original: Mixtures of experts have recently been considered in the transfer learning setting (Aljundi et al., 2016), however, the decision process is modelled by an autoencoder reconstruction error based heuristic and is not scaled to a large number of tasks.
Revised: Mixtures of experts have recently been considered in the transfer learning setting (Aljundi et al., 2016), however, the decision process is modelled by an autoencoder reconstruction error based heuristic and is not scaled to a large number of tasks.

Original: In the use of dynamic representations, our work is also related to single task and multi-task models that learn to generate weights for an optimal neural network (Ha et al., 2016), (Ravi & Larochelle, 2017), (Munkhdalai & Yu, 2017).
Revised: In the use of dynamic representations, our work is also related to single task and multi-task models that learn to generate weights for an optimal neural network (Ha et al., 2016), (Ravi & Larochelle, 2017), (Munkhdalai & Yu, 2017).

Original: While these models are very powerful, they have trouble scaling to deep models with a large number of parameters (Wichrowska et al., 2017) without tricks to simplify the formulation.
Revised: While these models are very powerful, they have trouble scaling to deep models with a large number of parameters (Wichrowska et al., 2017) without tricks to simplify the formulation.

Original: In contrast, we demonstrate that Routing Networks can be applied to create dynamic network architectures for architectures like convnets by routing some of their layers.
Revised: In contrast, we demonstrate that Routing Networks can be applied to create dynamic network architectures for architectures like convnets by routing some of their layers.

Original: Our work extends an emerging line of recent research focused on automated architecture search.
Revised: Our work extends an emerging line of recent research focused on automated architecture search.

Original: In this work, the goal is to reduce the burden on the practitioner by automatically learning black box algorithms that search for optimal architectures and hyperparameters.
Revised: In this work, the goal is to reduce the burden on the practitioner by automatically learning black box algorithms that search for optimal architectures and hyperparameters.

Original: These include techniques based on reinforcement learning (Zoph & Le, 2017), (Baker et al., 2017), evolutionary algorithms (Miikkulainen et al., 2017), approximate random simulations (Brock et al., 2017), and adaptive growth (Cortes et al., 2016).
Revised: These include techniques based on reinforcement learning (Zoph & Le, 2017), (Baker et al., 2017), evolutionary algorithms (Miikkulainen et al., 2017), approximate random simulations (Brock et al., 2017), and adaptive growth (Cortes et al., 2016).

Original: To the best of our knowledge we are the ﬁrst to apply this idea to multitask learning.
Revised: To the best of our knowledge we are the ﬁrst to apply this idea to multitask learning.

Original: Our technique can learn to construct a very general class of architectures without the need for human intervention to manually choose which parameters will be shared and which will be kept task-speciﬁc.
Revised: Our technique can learn to construct a very general class of architectures without the need for human intervention to manually choose which parameters will be shared and which will be kept task-speciﬁc.

Original: Perhaps most related to our work is the literature on minimizing computation cost for single-task problems by conditional routing.
Revised: Perhaps most related to our work is the literature on minimizing computation cost for single-task problems by conditional routing.

Original: These include decisions trained with REINFORCE (Denoyer &
Revised: These include decisions trained with REINFORCE (Denoyer &

Original: Gallinari, 2014), (Bengio et al., 2015), (Hamrick et al., 2017), Q Learning (Liu & Deng, 2017), and actor-critic methods (McGill & Perona, 2017).
Revised: Gallinari, 2014), (Bengio et al., 2015), (Hamrick et al., 2017), Q Learning (Liu & Deng, 2017), and actor-critic methods (McGill & Perona, 2017).

Original: Our approach differs however in the introduction of several novel elements.
Revised: Our approach differs however in the introduction of several novel elements.

Original: Speciﬁcally, our work explores the multi-task learning setting, it uses a multi-agent reinforcement learning training algorithm, and it is structured as a recursive decision process.
Revised: Speciﬁcally, our work explores the multi-task learning setting, it uses a multi-agent reinforcement learning training algorithm, and it is structured as a recursive decision process.

Original: While our the Routing Networks paradigm is a novel artiﬁcial neural network formulation, the high level idea of task speciﬁc “routing” as a cognitive function is well founded in biological studies and theories of the human brain (Gurney et al., 2001), (Buschman & Miller, 2010), (Stocco et al., 2010).
Revised: While our the Routing Networks paradigm is a novel artiﬁcial neural network formulation, the high level idea of task speciﬁc “routing” as a cognitive function is well founded in biological studies and theories of the human brain (Gurney et al., 2001), (Buschman & Miller, 2010), (Stocco et al., 2010).

Original:  R OUTING N ETWORKS
Revised:  R OUTING N ETWORKS

Original: A routing network consists of two components: a router and a set of function block s, which can be any neural network.
Revised: A routing network consists of two components: a router and a set of function block s, which can be any neural network.

Original: At each iteration the router processes its input to select a block, applies the block to the input, and recursively processes the result.
Revised: At each iteration the router processes its input to select a block, applies the block to the input, and recursively processes the result.

Original: This process is illustrated in Figure 1.
Revised: This process is illustrated in Figure 1.

Original: The dashed line shows the router accepting the input and selecting a function f i .
Revised: The dashed line shows the router accepting the input and selecting a function f i .

Original: The solid line shows the application of f i to the input and the wide dashed line indicates the recurrent loop.
Revised: The solid line shows the application of f i to the input and the wide dashed line indicates the recurrent loop.

Original: If the function blocks are of different dimensions then the router is constrained to select dimensionally conformant blocks to apply.
Revised: If the function blocks are of different dimensions then the router is constrained to select dimensionally conformant blocks to apply.

Original: Algorithm 1 gives the algorithm for evaluating a routing network on an input.
Revised: Algorithm 1 gives the algorithm for evaluating a routing network on an input.

Original: If the routing network is run for d invocations then we say it has depth d .
Revised: If the routing network is run for d invocations then we say it has depth d .

Original: For N function blocks a routing network run to a depth d can select from N d distinct trainable functions.
Revised: For N function blocks a routing network run to a depth d can select from N d distinct trainable functions.

Original: Any neural network can be represented as a routing network by adding copies of its layers as routing network function blocks.
Revised: Any neural network can be represented as a routing network by adding copies of its layers as routing network function blocks.

Original: We can group the function blocks for each network layer and constrain the router to pick from layer 0 function blocks at depth 0, layer 1 blocks at depth 1, and so on.
Revised: We can group the function blocks for each network layer and constrain the router to pick from layer 0 function blocks at depth 0, layer 1 blocks at depth 1, and so on.

Original: If the number of function blocks differs from layer to layer in the original network, then the router may accommodate this by, for example, maintaining a separate decision function for each depth.
Revised: If the number of function blocks differs from layer to layer in the original network, then the router may accommodate this by, for example, maintaining a separate decision function for each depth.

Original: [Figure]
Revised: [Figure]

Original: Data: input x ; d the recurrence depth 1 output = x 2 for i in 1..
Revised: Data: input x ; d the recurrence depth 1 output = x 2 for i in 1..

Original: d do f = router(output) output = f (output) 5 return output
Revised: d do f = router(output) output = f (output) 5 return output

Original: Both the router and function blocks need training.
Revised: Both the router and function blocks need training.

Original: A high-level view of the training procedure is shown in Figure 2.
Revised: A high-level view of the training procedure is shown in Figure 2.

Original: First, given an input x (which may include a task label), we compute a route and output value ˆ y using the procedure given in Algorithm 1.
Revised: First, given an input x (which may include a task label), we compute a route and output value ˆ y using the procedure given in Algorithm 1.

Original: We then apply a loss function L and compute gradient updates using SGD/backprop.
Revised: We then apply a loss function L and compute gradient updates using SGD/backprop.

Original: To train the router we employ a reinforcement learning approach.
Revised: To train the router we employ a reinforcement learning approach.

Original: Speciﬁcally, we use the output ˆ y to compute a reward value of +1 if the prediction is correct and -1 if it is incorrect.
Revised: Speciﬁcally, we use the output ˆ y to compute a reward value of +1 if the prediction is correct and -1 if it is incorrect.

Original: We then accumulate this reward back through each of the routing decisions (along with any per-action rewards) to compute the cumulative discounted reward R i at each decision point i .
Revised: We then accumulate this reward back through each of the routing decisions (along with any per-action rewards) to compute the cumulative discounted reward R i at each decision point i .

Original: Algorithm 2 summarizes the procedure.
Revised: Algorithm 2 summarizes the procedure.

Original: Many reinforcement learning
Revised: Many reinforcement learning

Original: [Figure]
Revised: [Figure]

Original: = k k + γ ∗ algorithms are applicable in this setting.
Revised: = k k + γ ∗ algorithms are applicable in this setting.

Original: We experimented with Policy Gradient (PG), Q-Learning, and multi-agent (MARL) algorithms which we describe in more detail below.
Revised: We experimented with Policy Gradient (PG), Q-Learning, and multi-agent (MARL) algorithms which we describe in more detail below.

Original: As we have described it, the training of the router and function blocks is performed independently after computing the loss.
Revised: As we have described it, the training of the router and function blocks is performed independently after computing the loss.

Original: We have also experimented with adding the gradients from the router choices ( ∆ R ) to those for the function blocks which produce their input.
Revised: We have also experimented with adding the gradients from the router choices ( ∆ R ) to those for the function blocks which produce their input.

Original: We found no advantage but leave a more thorough investigation for future work.
Revised: We found no advantage but leave a more thorough investigation for future work.

Original: 3. R OUTER T RAINING USING RL
Revised: 3. R OUTER T RAINING USING RL

Original: [Figure]
Revised: [Figure]

Original: and the task label task .
Revised: and the task label task .

Original: α i is a routing agent; α d is a dispatching agent.
Revised: α i is a routing agent; α d is a dispatching agent.

Original: The simplest implementation of a router in our approach is as a single RL agent whose policy predicts the next function block given the output value of the previous layer (and possibly a task label).
Revised: The simplest implementation of a router in our approach is as a single RL agent whose policy predicts the next function block given the output value of the previous layer (and possibly a task label).

Original: This conﬁguration is shown in Figure 3(a).
Revised: This conﬁguration is shown in Figure 3(a).

Original: In practice we found it difﬁcult to achieve good performance in this setting (for reasons we discuss in Section 4) and our best results divide the decision making across multiple separate agents, one per task, as shown in Figure 3(b).
Revised: In practice we found it difﬁcult to achieve good performance in this setting (for reasons we discuss in Section 4) and our best results divide the decision making across multiple separate agents, one per task, as shown in Figure 3(b).

Original: Here the assignment of tasks to agents is ﬁxed, not learned during training.
Revised: Here the assignment of tasks to agents is ﬁxed, not learned during training.

Original: When a new instance is presented to the network, its task label is used to identify the corresponding agent which then makes the routing decision.
Revised: When a new instance is presented to the network, its task label is used to identify the corresponding agent which then makes the routing decision.

Original: It is also possible to learn to assign the tasks to agents as shown in Figure 3(c).
Revised: It is also possible to learn to assign the tasks to agents as shown in Figure 3(c).

Original: The routing prediction is now made hierarchically with one agent, the dispatcher agent, denoted α d in the ﬁgure, choosing from among the task-speciﬁc agents based on the task label and input value.
Revised: The routing prediction is now made hierarchically with one agent, the dispatcher agent, denoted α d in the ﬁgure, choosing from among the task-speciﬁc agents based on the task label and input value.

Original: We have experimented with all of these architectures and discuss the results in the Section 4.
Revised: We have experimented with all of these architectures and discuss the results in the Section 4.

Original: One important aspect of the router training is the design of a reward structure which incentivizes good function block choices.
Revised: One important aspect of the router training is the design of a reward structure which incentivizes good function block choices.

Original: We consider two rewards: a ﬁnal reward, and a per-action reward that we shape to motivate collaboration.
Revised: We consider two rewards: a ﬁnal reward, and a per-action reward that we shape to motivate collaboration.

Original: The ﬁnal reward is +1 if the instance is correctly classiﬁed and − 1 if not.
Revised: The ﬁnal reward is +1 if the instance is correctly classiﬁed and − 1 if not.

Original: We also only consider per-action rewards that are constrained by per-action reward ≤ ﬁnal reward · ρ for ρ ≤ 10 .
Revised: We also only consider per-action rewards that are constrained by per-action reward ≤ ﬁnal reward · ρ for ρ ≤ 10 .

Original: To incentivize the agents to use as few function blocks as possible we use a collaboration reward which rewards each action an amount proportional to the average agreement among all agents who took that action in the past.
Revised: To incentivize the agents to use as few function blocks as possible we use a collaboration reward which rewards each action an amount proportional to the average agreement among all agents who took that action in the past.

Original: We experimented with two versions: a policy-based reward, where the reward is proportional to the average over the probabilities (or
Revised: We experimented with two versions: a policy-based reward, where the reward is proportional to the average over the probabilities (or

Original: Q-Values) assigned to that action by any agent that took it in the past; and a count-based reward, where the reward is proportional to the average frequency with which that action was taken in the past.
Revised: Q-Values) assigned to that action by any agent that took it in the past; and a count-based reward, where the reward is proportional to the average frequency with which that action was taken in the past.

Original: There was no signiﬁcant difference between them and we use the policy-based reward for all of our experiments.
Revised: There was no signiﬁcant difference between them and we use the policy-based reward for all of our experiments.

Original: In the next sections we describe each RL setting used in the experiments in detail.
Revised: In the next sections we describe each RL setting used in the experiments in detail.

Original: We consider variations in both the policy representation (tabular or approximation function) and learning strategies (PG, Q, and WPL/MARL).
Revised: We consider variations in both the policy representation (tabular or approximation function) and learning strategies (PG, Q, and WPL/MARL).

Original: 3.1. RL A LGORITHMS
Revised: 3.1. RL A LGORITHMS

Original: We evaluate three reinforcement learning strategies: Policy Gradient (PG), Q-Learning, and the
Revised: We evaluate three reinforcement learning strategies: Policy Gradient (PG), Q-Learning, and the

Original: Weighted Policy Learner (WPL), a MARL strategy.
Revised: Weighted Policy Learner (WPL), a MARL strategy.

Original: Both the Q-Learning and Policy Gradient algorithms are applicable with tabular and approximation function policy representations.
Revised: Both the Q-Learning and Policy Gradient algorithms are applicable with tabular and approximation function policy representations.

Original: We use vanilla
Revised: We use vanilla

Original: PG with the tabular representation and REINFORCE (Williams, 1992) for approximation function representations.
Revised: PG with the tabular representation and REINFORCE (Williams, 1992) for approximation function representations.

Original: The tabular representation has the invocation depth as its row dimension and the function block as its column dimension with the entries containing the probability of choosing a given function block at a given depth.
Revised: The tabular representation has the invocation depth as its row dimension and the function block as its column dimension with the entries containing the probability of choosing a given function block at a given depth.

Original: For Q-Learning we use a similar table but store the expected return in the entries.
Revised: For Q-Learning we use a similar table but store the expected return in the entries.

Original: We use regular Q-Learning for the tabular representation and train the approximators to minimize the (cid:96) 2 norm of the temporal difference error.
Revised: We use regular Q-Learning for the tabular representation and train the approximators to minimize the (cid:96) 2 norm of the temporal difference error.

Original: Speciﬁcally, we deﬁne a state s to be the current output value of the last function block chosen up to that point (including a task label if one is provided with the input) and the current depth.
Revised: Speciﬁcally, we deﬁne a state s to be the current output value of the last function block chosen up to that point (including a task label if one is provided with the input) and the current depth.

Original: The depth is necessary to include when routing a layered network which requires ﬁltering function block choices based on depth.
Revised: The depth is necessary to include when routing a layered network which requires ﬁltering function block choices based on depth.

Original: If
Revised: If

Original: V ( s ) is the estimated return from state s and the agent has taken action a to reach state s (cid:48) , then the loss function tries to minimize: (( r + γV ( s (cid:48) ))
Revised: V ( s ) is the estimated return from state s and the agent has taken action a to reach state s (cid:48) , then the loss function tries to minimize: (( r + γV ( s (cid:48) ))

Original: − V ( s )) 2 , where α is the learning rate; r is the reward for taking action a ; and γ is the discount factor (set to 1 for all experiments).
Revised: − V ( s )) 2 , where α is the learning rate; r is the reward for taking action a ; and γ is the discount factor (set to 1 for all experiments).

Original: We have also experimented with a “combinatorial” version of tabular Q learning which maintains a separate row for each combination of function block choices.
Revised: We have also experimented with a “combinatorial” version of tabular Q learning which maintains a separate row for each combination of function block choices.

Original: Since this approach suffers from a combinatorial state explosion it is not scalable to high depths and we include it only for comparison.
Revised: Since this approach suffers from a combinatorial state explosion it is not scalable to high depths and we include it only for comparison.

Original: Params: decision actions a ; decision returns R ; expected decision return estimates ˆ R ; policy learning rate λ π 1 for each decision a k do ∆( a k ) ← R ( a k ) − R k if ∆( a k ) < 0 then ∆( a k ) ← ∆( a k )(1 − π ( a k )) else ∆( a k ) ← ∆( a k )( π ( a k ))
Revised: Params: decision actions a ; decision returns R ; expected decision return estimates ˆ R ; policy learning rate λ π 1 for each decision a k do ∆( a k ) ← R ( a k ) − R k if ∆( a k ) < 0 then ∆( a k ) ← ∆( a k )(1 − π ( a k )) else ∆( a k ) ← ∆( a k )( π ( a k ))

Original: π ← simplex-projection ( π + λ π ∆)
Revised: π ← simplex-projection ( π + λ π ∆)

Original: In the multi-agent reinforcement learning (MARL) setting multiple agents interact in the environment and the expected return for any given agent may change without any action on that agent’s part.
Revised: In the multi-agent reinforcement learning (MARL) setting multiple agents interact in the environment and the expected return for any given agent may change without any action on that agent’s part.

Original: Worse, for routing networks, the environment is non-stationary since the function blocks are being trained as well as the router policy.
Revised: Worse, for routing networks, the environment is non-stationary since the function blocks are being trained as well as the router policy.

Original: This makes the training considerably more difﬁcult than in the single-agent (MDP) setting.
Revised: This makes the training considerably more difﬁcult than in the single-agent (MDP) setting.

Original: But the MARL approach offers a helpful interpretation of the problem as a stochastic game in which each agent competes and collaborates in the use and training of function blocks.
Revised: But the MARL approach offers a helpful interpretation of the problem as a stochastic game in which each agent competes and collaborates in the use and training of function blocks.

Original: In this view, incompatible agents need to compete for blocks to train, since negative transfer will make collaboration unattractive, while compatible agents can gain in transfer by training the same function block.
Revised: In this view, incompatible agents need to compete for blocks to train, since negative transfer will make collaboration unattractive, while compatible agents can gain in transfer by training the same function block.

Original: The agent’s (locally) optimal policies will correspond to the game’s Nash equilibrium 2 .
Revised: The agent’s (locally) optimal policies will correspond to the game’s Nash equilibrium 2 .

Original: Vanilla policy gradient methods are less well adapted to the changing environment and changes in other agent’s behavior, which may degrade their performance in this setting.
Revised: Vanilla policy gradient methods are less well adapted to the changing environment and changes in other agent’s behavior, which may degrade their performance in this setting.

Original: One MARL algorithm speciﬁcally designed to address this problem, and which has also been shown to converge in nonstationary environments, is the weighted policy learner (WPL) algorithm (Abdallah & Lesser, 2006), shown in Algorithm 3.
Revised: One MARL algorithm speciﬁcally designed to address this problem, and which has also been shown to converge in nonstationary environments, is the weighted policy learner (WPL) algorithm (Abdallah & Lesser, 2006), shown in Algorithm 3.

Original: WPL is a PG algorithm designed to dampen oscillation and push the agents to converge more quickly.
Revised: WPL is a PG algorithm designed to dampen oscillation and push the agents to converge more quickly.

Original: This is done by scaling the gradient of the expected return for an action a
Revised: This is done by scaling the gradient of the expected return for an action a

Original: according the probability of taking that action π ( a ) (if the gradient is positive) or 1 − π ( a ) (if the gradient is negative).
Revised: according the probability of taking that action π ( a ) (if the gradient is positive) or 1 − π ( a ) (if the gradient is negative).

Original: Intuitively, this has the effect of slowing down the learning rate when the policy is moving away from a Nash equilibrium strategy and increasing it when it approaches one.
Revised: Intuitively, this has the effect of slowing down the learning rate when the policy is moving away from a Nash equilibrium strategy and increasing it when it approaches one.

Original: We The full WPL-Update algorithm is shown in Algorithm 3.
Revised: We The full WPL-Update algorithm is shown in Algorithm 3.

Original: Details, including convergence proofs and more examples giving the intuition behind the algorithm, can be found in (Abdallah & Lesser, 2006).
Revised: Details, including convergence proofs and more examples giving the intuition behind the algorithm, can be found in (Abdallah & Lesser, 2006).

Original: The WPL-Update algorithm is deﬁned only for the tabular setting.
Revised: The WPL-Update algorithm is deﬁned only for the tabular setting.

Original: It is future work to adapt it to work with function approximators.
Revised: It is future work to adapt it to work with function approximators.

Original: Algorithm Algorithm
Revised: Algorithm Algorithm

Original: Params: decision actions a ; decision returns R ; expected decision return estimate learning rate λ r ; policy learning rate λ π 1 ˆ R k ← (1 − λ r ) ∗
Revised: Params: decision actions a ; decision returns R ; expected decision return estimate learning rate λ r ; policy learning rate λ π 1 ˆ R k ← (1 − λ r ) ∗

Original: ˆ R k
Revised: ˆ R k

Original: + λ r ∗ R k 2 WPL-Update( a , R , ˆ R , λ π )
Revised: + λ r ∗ R k 2 WPL-Update( a , R , ˆ R , λ π )

Original: To apply Algorithm 3 in the routing network context, we use the WPL-Trainer shown in Algorithm 4, that computes a running average of the returns for each action which is then passed to the WPL-Update algorithm.
Revised: To apply Algorithm 3 in the routing network context, we use the WPL-Trainer shown in Algorithm 4, that computes a running average of the returns for each action which is then passed to the WPL-Update algorithm.

Original: The complete router training algorithm using
Revised: The complete router training algorithm using

Original: WPL is then as follows.
Revised: WPL is then as follows.

Original: Replace line 7 of Algorithm 2 with WPL-Trainer( a , R ), then initialize ˆ R k ← 0 for all actions a k and run Algorithm 2.
Revised: Replace line 7 of Algorithm 2 with WPL-Trainer( a , R ), then initialize ˆ R k ← 0 for all actions a k and run Algorithm 2.

Original: After the actions a and returns R are computed on line 6, WPL-Trainer will compute an update to the running return averages for each decision a k then compute the WPL policy update using WPL-Update.
Revised: After the actions a and returns R are computed on line 6, WPL-Trainer will compute an update to the running return averages for each decision a k then compute the WPL policy update using WPL-Update.

Original: WPL-Update uses these estimates of the expected return for each action to determine the gradient and update the policy.
Revised: WPL-Update uses these estimates of the expected return for each action to determine the gradient and update the policy.

Original: The function simplex-projection projects the updated policy values to make it a valid probability distribution.
Revised: The function simplex-projection projects the updated policy values to make it a valid probability distribution.

Original: The projection is deﬁned as: clip ( π ) / (cid:80) ( clip ( π )) , where clip ( x ) = max(0 , min (1 , x )) .
Revised: The projection is deﬁned as: clip ( π ) / (cid:80) ( clip ( π )) , where clip ( x ) = max(0 , min (1 , x )) .

Original:  Q UANTITATIVE R ESULTS
Revised:  Q UANTITATIVE R ESULTS

Original: We experiment with three datasets: multi-task versions of MNIST (MNIST-MTL)
Revised: We experiment with three datasets: multi-task versions of MNIST (MNIST-MTL)

Original: (Lecun et al., 1998), Mini-Imagenet (MIN-MTL) (Vinyals et al., 2016) as introduced by (Ravi & Larochelle, 2017), and CIFAR-100 (CIFAR-MTL) (Krizhevsky, 2009) where we treat the 20 superclasses as tasks.
Revised: (Lecun et al., 1998), Mini-Imagenet (MIN-MTL) (Vinyals et al., 2016) as introduced by (Ravi & Larochelle, 2017), and CIFAR-100 (CIFAR-MTL) (Krizhevsky, 2009) where we treat the 20 superclasses as tasks.

Original: In the binary MNIST-MTL dataset, the task is to differentiate instances of a given class c from non-instances.
Revised: In the binary MNIST-MTL dataset, the task is to differentiate instances of a given class c from non-instances.

Original: We create 10 tasks and for each we use 1000 instances of the positive class c and 1000 each of the remaining 9 negative classes for a total of 10,000 instances per task during training, which we then test on 200 samples per task.
Revised: We create 10 tasks and for each we use 1000 instances of the positive class c and 1000 each of the remaining 9 negative classes for a total of 10,000 instances per task during training, which we then test on 200 samples per task.

Original: MIN-MTL is a smaller version of ImageNet (Deng et al., 2009) which is easier to train in reasonable time periods.
Revised: MIN-MTL is a smaller version of ImageNet (Deng et al., 2009) which is easier to train in reasonable time periods.

Original: For mini-ImageNet we randomly choose 50 labels and create tasks from 10 random subsets of 5 labels each chosen from these.
Revised: For mini-ImageNet we randomly choose 50 labels and create tasks from 10 random subsets of 5 labels each chosen from these.

Original: Each label has 800 training instances and 50 testing instances – so 4000 training and 250 testing instances per task.
Revised: Each label has 800 training instances and 50 testing instances – so 4000 training and 250 testing instances per task.

Original: For all 10 tasks we have a total of 40000 training instances.
Revised: For all 10 tasks we have a total of 40000 training instances.

Original: Finally, CIFARhas coarse and ﬁne labels for its instances.
Revised: Finally, CIFARhas coarse and ﬁne labels for its instances.

Original: We follow existing work (Krizhevsky, 2009) creating one task for each of the 20 coarse labels and include 500 instances for each of the corresponding ﬁne labels.
Revised: We follow existing work (Krizhevsky, 2009) creating one task for each of the 20 coarse labels and include 500 instances for each of the corresponding ﬁne labels.

Original: There are 20 tasks with a total of 2500 instances per task; 2500 for training and 500 for testing.
Revised: There are 20 tasks with a total of 2500 instances per task; 2500 for training and 500 for testing.

Original: All results are reported on the test set and are averaged over 3 runs.
Revised: All results are reported on the test set and are averaged over 3 runs.

Original: The data are summarized in Table 1.
Revised: The data are summarized in Table 1.

Original: Each of these datasets has interesting characteristics which challenge the learning in different ways.
Revised: Each of these datasets has interesting characteristics which challenge the learning in different ways.

Original: CIFAR-MTL is a “natural” dataset whose tasks correspond to human categories.
Revised: CIFAR-MTL is a “natural” dataset whose tasks correspond to human categories.

Original: MIN-MTL is randomly generated so will have less task coherence.
Revised: MIN-MTL is randomly generated so will have less task coherence.

Original: And MNIST-MTL, while simple, has the difﬁcult property that the same instance can appear with different labels in different tasks, causing interference.
Revised: And MNIST-MTL, while simple, has the difﬁcult property that the same instance can appear with different labels in different tasks, causing interference.

Original: For example, in the “0 vs other digits” task, “0” appears with a positive label but in the “1 vs other digits” task it appears with a negative label.
Revised: For example, in the “0 vs other digits” task, “0” appears with a positive label but in the “1 vs other digits” task it appears with a negative label.

Original: Dataset
Revised: Dataset

Original: [Table]
Revised: [Table]

Original: Our experiments are conducted on a convnet architecture (SimpleConvNet) which appeared recently in (Ravi & Larochelle, 2017).
Revised: Our experiments are conducted on a convnet architecture (SimpleConvNet) which appeared recently in (Ravi & Larochelle, 2017).

Original: This model has 4 convolutional layers, each consisting of a 3x3 convolution and 32 ﬁlters, followed by batch normalization and a ReLU.
Revised: This model has 4 convolutional layers, each consisting of a 3x3 convolution and 32 ﬁlters, followed by batch normalization and a ReLU.

Original: The convolutional layers are followed byfully connected layers, with 128 hidden units each.
Revised: The convolutional layers are followed byfully connected layers, with 128 hidden units each.

Original: Our routed version of the network routes the 3 fully connected layers and for each routed layer we supply one randomly initialized function block per task in the dataset.
Revised: Our routed version of the network routes the 3 fully connected layers and for each routed layer we supply one randomly initialized function block per task in the dataset.

Original: We dedicate a special action to allow the agents to skip layers during training which leaves the current state unchanged.
Revised: We dedicate a special action to allow the agents to skip layers during training which leaves the current state unchanged.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We use the Adam optimization algorithm Kingma & Ba (2014).
Revised: We use the Adam optimization algorithm Kingma & Ba (2014).

Original: The collaboration reward parameter ρ = 0.3; learning rate =
Revised: The collaboration reward parameter ρ = 0.3; learning rate =

Original: 1e-3 annealed by dividing by 10 every 10 epochs; batch size = 10.
Revised: 1e-3 annealed by dividing by 10 every 10 epochs; batch size = 10.

Original: The SimpleConvNet has batch normalization layers but we use no dropout and perform no hyper-parameter search or other tuning.
Revised: The SimpleConvNet has batch normalization layers but we use no dropout and perform no hyper-parameter search or other tuning.

Original: All data are presented in Table 2 in the Appendix.
Revised: All data are presented in Table 2 in the Appendix.

Original: In the ﬁrst experiment, shown in Figure 4, we compare different RL training algorithms on CIFARMTL against the MARL:WPL algorithm which keeps a separate table for each depth to accommodate layering.
Revised: In the ﬁrst experiment, shown in Figure 4, we compare different RL training algorithms on CIFARMTL against the MARL:WPL algorithm which keeps a separate table for each depth to accommodate layering.

Original: We compare against an agent-per-task REINFORCE learner which maintains a separate approximation function (a two-layer MLP) for each layer; an agent-per-task Q learner with a separate approximation function (a two-layer MLP) per layer; a single-agent REINFORCE with a separate approximation function (a two-layer MLP) per layer; an agent-per-task Q learner with a separate table for each layer; and an agent-per-task Q learner with a “combinatorial” table which holds the decision history.
Revised: We compare against an agent-per-task REINFORCE learner which maintains a separate approximation function (a two-layer MLP) for each layer; an agent-per-task Q learner with a separate approximation function (a two-layer MLP) per layer; a single-agent REINFORCE with a separate approximation function (a two-layer MLP) per layer; an agent-per-task Q learner with a separate table for each layer; and an agent-per-task Q learner with a “combinatorial” table which holds the decision history.

Original: The best performer is the WPL algorithm which outperforms the nearest competitor (agent-per-task REINFORCE with a separate per-layer decision function) by 10%.
Revised: The best performer is the WPL algorithm which outperforms the nearest competitor (agent-per-task REINFORCE with a separate per-layer decision function) by 10%.

Original: We can see that (1) the WPL algorithm works better than a similar vanilla PG; (2) having multiple agents works better than having a single agent; and (3) with the exception of WPL, using a function approximator works better than tabular versions.
Revised: We can see that (1) the WPL algorithm works better than a similar vanilla PG; (2) having multiple agents works better than having a single agent; and (3) with the exception of WPL, using a function approximator works better than tabular versions.

Original: The next experiment compares the best performing algorithm MARL:WPL against other routing approaches.
Revised: The next experiment compares the best performing algorithm MARL:WPL against other routing approaches.

Original: The MARL:WPL algorithm has the full-connected layers of the SimpleConvNet routed using the layering approach we discussed earlier.
Revised: The MARL:WPL algorithm has the full-connected layers of the SimpleConvNet routed using the layering approach we discussed earlier.

Original: We now keep the WPL algorithm but experiment with variations on this setup.
Revised: We now keep the WPL algorithm but experiment with variations on this setup.

Original: To make the next comparison clear we rename this algorithm “routingall-fc” in Figure 5 to reﬂect the fact that it routes all the fully connected layers of the SimpleConvNet.
Revised: To make the next comparison clear we rename this algorithm “routingall-fc” in Figure 5 to reﬂect the fact that it routes all the fully connected layers of the SimpleConvNet.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We compare against several other approaches.
Revised: We compare against several other approaches.

Original: One approach, routing-fc-no-layering, has the same setup as routing-all-fc, but does not constrain the router to pick only from layer 0 function blocks at depth 0, etc.
Revised: One approach, routing-fc-no-layering, has the same setup as routing-all-fc, but does not constrain the router to pick only from layer 0 function blocks at depth 0, etc.

Original: It is allowed to choose any function block from two of the layers (since two of the layers have identical input and output dimensions).
Revised: It is allowed to choose any function block from two of the layers (since two of the layers have identical input and output dimensions).

Original: Another approach, soft-mixture-fc, is a soft version of the router architecture.
Revised: Another approach, soft-mixture-fc, is a soft version of the router architecture.

Original: This soft version utilizes the same function blocks as the routed version, but replaces the hard selection with a trained softmax average over the blocks.
Revised: This soft version utilizes the same function blocks as the routed version, but replaces the hard selection with a trained softmax average over the blocks.

Original: We also compare against the dispatched architecture shown in Figure 3(c).
Revised: We also compare against the dispatched architecture shown in Figure 3(c).

Original: Here there is not much difference between the routing-all-fc architecture and the non-layered version.
Revised: Here there is not much difference between the routing-all-fc architecture and the non-layered version.

Original: But both architectures are considerably better than the soft and dispatched versions.
Revised: But both architectures are considerably better than the soft and dispatched versions.

Original: The soft version has data only to 30 epochs because it is very time-consuming to train.
Revised: The soft version has data only to 30 epochs because it is very time-consuming to train.

Original: We use the routing-all-fc layer in the remainder of our experiments.
Revised: We use the routing-all-fc layer in the remainder of our experiments.

Original: Next we compare the best algorithm routing-all-fc on different domains against two challenging baselines: task speciﬁc-1-fc and task speciﬁc-all-fc.
Revised: Next we compare the best algorithm routing-all-fc on different domains against two challenging baselines: task speciﬁc-1-fc and task speciﬁc-all-fc.

Original: task-speciﬁc-1-fc has a separate last fully connected layer for each task and shares the rest of the layers.
Revised: task-speciﬁc-1-fc has a separate last fully connected layer for each task and shares the rest of the layers.

Original: task speciﬁc-all-fc has a separate set of all the fully connected layers for each task.
Revised: task speciﬁc-all-fc has a separate set of all the fully connected layers for each task.

Original: These baseline architectures allow considerable sharing of parameters but also grant the network private parameters for each task to avoid interference.
Revised: These baseline architectures allow considerable sharing of parameters but also grant the network private parameters for each task to avoid interference.

Original: The decision about what layers to share is static and does not change on a per-task basis.
Revised: The decision about what layers to share is static and does not change on a per-task basis.

Original: The results are shown in Figures 6, 7, and 8.
Revised: The results are shown in Figures 6, 7, and 8.

Original: In each case the routing net does better than the baselines, beating them by 7% on CIFAR-MTL, 2% on MIN-MTL.
Revised: In each case the routing net does better than the baselines, beating them by 7% on CIFAR-MTL, 2% on MIN-MTL.

Original: We surmise that the results are better on CIFAR because the task instances have more in common whereas the MIN-MTL tasks are randomly constructed, making sharing less proﬁtable.
Revised: We surmise that the results are better on CIFAR because the task instances have more in common whereas the MIN-MTL tasks are randomly constructed, making sharing less proﬁtable.

Original: On MNIST-MTL the random baseline is 90% and the routing net beats the nearest baseline by about 4%.
Revised: On MNIST-MTL the random baseline is 90% and the routing net beats the nearest baseline by about 4%.

Original: We also ran the soft version here since it was easy to train but it didn’t make progress.
Revised: We also ran the soft version here since it was easy to train but it didn’t make progress.

Original: In all cases routing makes a signiﬁcant difference over the baselines and we conclude that a dynamic policy which learns the function blocks to compose on a per-task basis yields better accuracy and sharper convergence than simple static decisions or the soft attention approach.
Revised: In all cases routing makes a signiﬁcant difference over the baselines and we conclude that a dynamic policy which learns the function blocks to compose on a per-task basis yields better accuracy and sharper convergence than simple static decisions or the soft attention approach.

Original: To explore why the multi-agent approach seems to do better than the single-agent, we compared their policy dynamics.
Revised: To explore why the multi-agent approach seems to do better than the single-agent, we compared their policy dynamics.

Original: In the cases we examined we found that the single agent often chose just 1 or 2 function blocks at each depth, and then routed all tasks to those.
Revised: In the cases we examined we found that the single agent often chose just 1 or 2 function blocks at each depth, and then routed all tasks to those.

Original: We suspect that there is simply too little signal available to the agent in the early, random stages, and once a bias is established its decisions suffer from a lack of diversity.
Revised: We suspect that there is simply too little signal available to the agent in the early, random stages, and once a bias is established its decisions suffer from a lack of diversity.

Original: [Figure]
Revised: [Figure]

Original: The routing network on the other hand learns a policy which, unlike the baseline static models, partitions the network quite differently for each task, and also achieves considerable diversity in its choices as can be seen in Figure 11.
Revised: The routing network on the other hand learns a policy which, unlike the baseline static models, partitions the network quite differently for each task, and also achieves considerable diversity in its choices as can be seen in Figure 11.

Original: This ﬁgure shows the routing decisions made over the whole MNIST MTL dataset.
Revised: This ﬁgure shows the routing decisions made over the whole MNIST MTL dataset.

Original: Each task is labeled at the top and the decisions for each of the three routed layers are shown below.
Revised: Each task is labeled at the top and the decisions for each of the three routed layers are shown below.

Original: We believe that because the routing network has separate policies for each task, it is less sensitive to a bias for one or two function blocks and each agent learns more independently what works for its assigned task.
Revised: We believe that because the routing network has separate policies for each task, it is less sensitive to a bias for one or two function blocks and each agent learns more independently what works for its assigned task.

Original:  Q UALITATIVE R ESULTS
Revised:  Q UALITATIVE R ESULTS

Original: To better understand the agent interaction we have created several views of the policy dynamics.
Revised: To better understand the agent interaction we have created several views of the policy dynamics.

Original: First, in Figure 9, we chart the policy over time for the ﬁrst decision.
Revised: First, in Figure 9, we chart the policy over time for the ﬁrst decision.

Original: Each rectangle labeled T i on the left represents the evolution of the agent’s policy for that task.
Revised: Each rectangle labeled T i on the left represents the evolution of the agent’s policy for that task.

Original: For each task, the horizontal axis is number of samples per task and the vertical axis is actions (decisions).
Revised: For each task, the horizontal axis is number of samples per task and the vertical axis is actions (decisions).

Original: Each vertical slice shows the probability distribution over actions after having seen that many samples of its task, with darker shades indicating higher probability.
Revised: Each vertical slice shows the probability distribution over actions after having seen that many samples of its task, with darker shades indicating higher probability.

Original: From this picture we can see that, in the beginning, all task agents have high entropy.
Revised: From this picture we can see that, in the beginning, all task agents have high entropy.

Original: As more samples are processed each agent develops several candidate function blocks to use for its task but eventually all agents converge to close to 100% probability for one particular block.
Revised: As more samples are processed each agent develops several candidate function blocks to use for its task but eventually all agents converge to close to 100% probability for one particular block.

Original: In the language of games, the agents ﬁnd a pure strategy for routing.
Revised: In the language of games, the agents ﬁnd a pure strategy for routing.

Original: In the next view of the dynamics, we pick one particular function block (block 7) and plot the probability, for each agent, of choosing that block over time.
Revised: In the next view of the dynamics, we pick one particular function block (block 7) and plot the probability, for each agent, of choosing that block over time.

Original: The horizontal axis is time (sample) and the vertical axis is the probability of choosing block 7.
Revised: The horizontal axis is time (sample) and the vertical axis is the probability of choosing block 7.

Original: Each colored curve corresponds to a different task agent.
Revised: Each colored curve corresponds to a different task agent.

Original: Here we can see that there is considerable oscillation over time until two agents, pink and green, emerge as the “victors” for the use of block 7 and each assign close to 100% probability for choosing it in routing their respective tasks.
Revised: Here we can see that there is considerable oscillation over time until two agents, pink and green, emerge as the “victors” for the use of block 7 and each assign close to 100% probability for choosing it in routing their respective tasks.

Original: It is interesting to see that the eventual winners, pink and green, emerge earlier as well as strongly interested in block 7.
Revised: It is interesting to see that the eventual winners, pink and green, emerge earlier as well as strongly interested in block 7.

Original: We have noticed this pattern in the analysis of other blocks and speculate that the agents who want to use the block are being pulled away from their early Nash equilibrium as other agents try to train the block away.
Revised: We have noticed this pattern in the analysis of other blocks and speculate that the agents who want to use the block are being pulled away from their early Nash equilibrium as other agents try to train the block away.

Original: [Figure]
Revised: [Figure]

Original: Finally, in Figure 11 we show a map of the routing for MNIST-MTL.
Revised: Finally, in Figure 11 we show a map of the routing for MNIST-MTL.

Original: Here tasks are at the top and each layer below represents one routing decision.
Revised: Here tasks are at the top and each layer below represents one routing decision.

Original: Conventional wisdom has it that networks will beneﬁt from sharing early, using the ﬁrst layers for common representations, diverging later to accommodate differences in the tasks.
Revised: Conventional wisdom has it that networks will beneﬁt from sharing early, using the ﬁrst layers for common representations, diverging later to accommodate differences in the tasks.

Original: This is the setup for our baselines.
Revised: This is the setup for our baselines.

Original: It is interesting to see that this is not what the network learns on its own.
Revised: It is interesting to see that this is not what the network learns on its own.

Original: Here we see that the agents have converged on a strategy which ﬁrst uses 7 function blocks, then compresses to just 4, then again expands to use 5.
Revised: Here we see that the agents have converged on a strategy which ﬁrst uses 7 function blocks, then compresses to just 4, then again expands to use 5.

Original: It is not clear if this is an optimal strategy but it does certainly give improvement over the static baselines.
Revised: It is not clear if this is an optimal strategy but it does certainly give improvement over the static baselines.

Original:  F UTURE W ORK
Revised:  F UTURE W ORK

Original: We have presented a general architecture for routing and multi-agent router training algorithm which performs signiﬁcantly better than baselines and non-multi-agent approaches.
Revised: We have presented a general architecture for routing and multi-agent router training algorithm which performs signiﬁcantly better than baselines and non-multi-agent approaches.

Original: The paradigm can eas- ily be applied to a state-of-the-art network to allow it to learn to dynamically adjust its representations.
Revised: The paradigm can eas- ily be applied to a state-of-the-art network to allow it to learn to dynamically adjust its representations.

Original: As described in the section on Routing Networks the state space to be learned grows exponentially with the depth of the routing, making it challenging to scale the routing to deeper networks in their entirety.
Revised: As described in the section on Routing Networks the state space to be learned grows exponentially with the depth of the routing, making it challenging to scale the routing to deeper networks in their entirety.

Original: It would be interesting to try hierarchical RL techniques (Barto & Mahadevan (2003)) here.
Revised: It would be interesting to try hierarchical RL techniques (Barto & Mahadevan (2003)) here.

Original: Our most successful experiments have used the multi-agent architecture with one agent per task, trained with WPL-Trainer.
Revised: Our most successful experiments have used the multi-agent architecture with one agent per task, trained with WPL-Trainer.

Original: Currently this approach is tabular but we are investigating ways to adapt it to use neural net approximators.
Revised: Currently this approach is tabular but we are investigating ways to adapt it to use neural net approximators.

Original: We have also tried routing networks in an online setting, training over a sequence of tasks for few shot learning.
Revised: We have also tried routing networks in an online setting, training over a sequence of tasks for few shot learning.

Original: To handle the iterative addition of new tasks we add a new routing agent for each and overﬁt it on the few shot examples while training the function modules with a very slow learning rate.
Revised: To handle the iterative addition of new tasks we add a new routing agent for each and overﬁt it on the few shot examples while training the function modules with a very slow learning rate.

Original: Our results so far have been mixed but this is a very useful setting and we plan to return to this problem.
Revised: Our results so far have been mixed but this is a very useful setting and we plan to return to this problem.

Original: A PPENDIX T ABLE OF R ESULTS
Revised: A PPENDIX T ABLE OF R ESULTS

Original: Epoch
Revised: Epoch

Original: RL (Figure 4)
Revised: RL (Figure 4)

Original: REINFORCE: per layer approx
Revised: REINFORCE: per layer approx

Original: REINFORCE: single approx
Revised: REINFORCE: single approx

Original: Qlearning: per layer approx
Revised: Qlearning: per layer approx

Original: Qlearning: per layer table*
Revised: Qlearning: per layer table*

Original: Qlearning: combinatorial table*
Revised: Qlearning: combinatorial table*

Original: MARL-WPL: per layer table*      arch (Figure 5) routing-all-fc
Revised: MARL-WPL: per layer table*      arch (Figure 5) routing-all-fc

Original:      routing-all-fc recursive      routing-all-fc dispatched      soft mixture-all-fc    routing-all-fc dispatched      task speciﬁc-all-fc
Revised:      routing-all-fc recursive      routing-all-fc dispatched      soft mixture-all-fc    routing-all-fc dispatched      task speciﬁc-all-fc

Original: CIFAR (Figure 6) routing-all-fc      task speciﬁc-all-fc      task speciﬁc-1-fc
Revised: CIFAR (Figure 6) routing-all-fc      task speciﬁc-all-fc      task speciﬁc-1-fc

Original: MIN (Figure 7) routing-all-fc      task speciﬁc-all-fc      task speciﬁc-1fc
Revised: MIN (Figure 7) routing-all-fc      task speciﬁc-all-fc      task speciﬁc-1fc

Original: MNIST (Figure 8) routing-all-fc      task speciﬁc-all-fc      task speciﬁc-1fc      soft mixture-all-fc
Revised: MNIST (Figure 8) routing-all-fc      task speciﬁc-all-fc      task speciﬁc-1fc      soft mixture-all-fc