Original: K NOWLEDGE F LOW : I MPROVE U PON Y OUR T EACHERS
Revised: K NOWLEDGE F LOW : I MPROVE U PON Y OUR T EACHERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for ﬁne-tuning a new model.
Revised: A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for ﬁne-tuning a new model.

Original: To address this issue, in this paper, we develop knowledge ﬂow which moves ‘knowledge’ from multiple deep nets, referred to as teachers, to a new deep net model, called the student.
Revised: To address this issue, in this paper, we develop knowledge ﬂow which moves ‘knowledge’ from multiple deep nets, referred to as teachers, to a new deep net model, called the student.

Original: The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too.
Revised: The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too.

Original: Upon training with knowledge ﬂow the student is independent of the teachers.
Revised: Upon training with knowledge ﬂow the student is independent of the teachers.

Original: We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming ﬁne-tuning and other ‘knowledge exchange’ methods.
Revised: We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming ﬁne-tuning and other ‘knowledge exchange’ methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Research communities have amassed a sizable number of deep net architectures for different tasks, and new ones are added almost daily.
Revised: Research communities have amassed a sizable number of deep net architectures for different tasks, and new ones are added almost daily.

Original: Some of those architectures are trained from scratch while others are ﬁne-tuned, i.e ., before training, their weights are initialized using a structurally similar deep net which was trained on different data.
Revised: Some of those architectures are trained from scratch while others are ﬁne-tuned, i.e ., before training, their weights are initialized using a structurally similar deep net which was trained on different data.

Original: Beyond ﬁne-tuning, particularly in reinforcement learning, teachers have also been considered in one way or another by Rusu et al.
Revised: Beyond ﬁne-tuning, particularly in reinforcement learning, teachers have also been considered in one way or another by Rusu et al.

Original: (2016b)
Revised: (2016b)

Original: ; Fernando et al.
Revised: ; Fernando et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Li & Hoiem (2016); Bengio et al.
Revised: ; Li & Hoiem (2016); Bengio et al.

Original: ; Patel et al.
Revised: ; Patel et al.

Original: ; Chen & Liu (2016); Teh et al.
Revised: ; Chen & Liu (2016); Teh et al.

Original: ; Parisotto et al.
Revised: ; Parisotto et al.

Original: For instance, progressive neural net (Rusu et al., 2016b) keeps multiple teachers during both training and inference, and learns to extract useful features from the teachers for a new target task.
Revised: For instance, progressive neural net (Rusu et al., 2016b) keeps multiple teachers during both training and inference, and learns to extract useful features from the teachers for a new target task.

Original: PathNet (Fernando et al., 2017) uses genetic algorithms to choose pathways from a giant network for learning new tasks.
Revised: PathNet (Fernando et al., 2017) uses genetic algorithms to choose pathways from a giant network for learning new tasks.

Original: ‘Growing a Brain’ (Wang et al., 2017) ﬁne-tunes a neural network while growing the network’s capacity (wider or deeper layers).
Revised: ‘Growing a Brain’ (Wang et al., 2017) ﬁne-tunes a neural network while growing the network’s capacity (wider or deeper layers).

Original: Actor-mimic (Parisotto et al., 2016) pre-trains a big model on multiple source tasks, then the big model is used as a weight initialization for a new model which will be trained on a new target task.
Revised: Actor-mimic (Parisotto et al., 2016) pre-trains a big model on multiple source tasks, then the big model is used as a weight initialization for a new model which will be trained on a new target task.

Original: Knowledge distillation (Hinton et al., 2015) distills knowledge from a large ensemble of models to a smaller student model.
Revised: Knowledge distillation (Hinton et al., 2015) distills knowledge from a large ensemble of models to a smaller student model.

Original: However, all the aforementioned techniques have limitations.
Revised: However, all the aforementioned techniques have limitations.

Original: For example, progressive neural net models (Rusu et al., 2016b) grow with the number of teachers.
Revised: For example, progressive neural net models (Rusu et al., 2016b) grow with the number of teachers.

Original: This large number of parameters limits the number of teachers a progressive neural net can handle, and largely increases the training and testing time.
Revised: This large number of parameters limits the number of teachers a progressive neural net can handle, and largely increases the training and testing time.

Original: In PathNet (Fernando et al., 2017), searching over a big network for pathways is computationally intensive.
Revised: In PathNet (Fernando et al., 2017), searching over a big network for pathways is computationally intensive.

Original: For ﬁne-tuning based methods such as ‘Growing a Brain’ (Wang et al., 2017) and actor-mimic (Parisotto et al., 2016), only one pretrained model can be used at a time.
Revised: For ﬁne-tuning based methods such as ‘Growing a Brain’ (Wang et al., 2017) and actor-mimic (Parisotto et al., 2016), only one pretrained model can be used at a time.

Original: Hence, their performance heavily relies on the chosen pretrained model.
Revised: Hence, their performance heavily relies on the chosen pretrained model.

Original: To address these shortcomings, we develop knowledge ﬂow which moves ‘knowledge’ of multiple teachers when training a student.
Revised: To address these shortcomings, we develop knowledge ﬂow which moves ‘knowledge’ of multiple teachers when training a student.

Original: Irrespective of how many teachers we use, the student is guaranteed to become independent at the ﬁnal stage of training and the size of the resulting student net remains constant.
Revised: Irrespective of how many teachers we use, the student is guaranteed to become independent at the ﬁnal stage of training and the size of the resulting student net remains constant.

Original: In addition, our framework makes no restrictions on the deep net size of the teacher and student, which provides ﬂexibility in choosing teacher models.
Revised: In addition, our framework makes no restrictions on the deep net size of the teacher and student, which provides ﬂexibility in choosing teacher models.

Original: Importantly, our approach is applicable to a variety of tasks from reinforcement learning to fully-supervised training.
Revised: Importantly, our approach is applicable to a variety of tasks from reinforcement learning to fully-supervised training.

Original: We evaluate knowledge ﬂow on a variety of tasks from reinforcement learning to fully-supervised learning.
Revised: We evaluate knowledge ﬂow on a variety of tasks from reinforcement learning to fully-supervised learning.

Original: In particular, we follow Rusu et al.
Revised: In particular, we follow Rusu et al.

Original: (2016b)
Revised: (2016b)

Original: ; Fernando et al.
Revised: ; Fernando et al.

Original: and compare on the same
Revised: and compare on the same

Original: Atari games.
Revised: Atari games.

Original: In addition, we also observed signiﬁcant top-1 error rate improvements on supervised learning datasets, i.e ., CIFAR-10, and CIFAR-100.
Revised: In addition, we also observed signiﬁcant top-1 error rate improvements on supervised learning datasets, i.e ., CIFAR-10, and CIFAR-100.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Knowledge ﬂow is applicable to a variety of settings from supervised learning to reinforcement learning, which we brieﬂy review to introduce notation.
Revised: Knowledge ﬂow is applicable to a variety of settings from supervised learning to reinforcement learning, which we brieﬂy review to introduce notation.

Original: Supervised Learning recovers the parameters θ of a mapping f θ : X → Y from data space X to output space Y .
Revised: Supervised Learning recovers the parameters θ of a mapping f θ : X → Y from data space X to output space Y .

Original: To this end, a dataset D = { ( x i , y i ) } ni
Revised: To this end, a dataset D = { ( x i , y i ) } ni

Original: =1 containing n pairs ( x i , y i ) (assumed to be sampled i.i.d.) is used, where x i ∈ X and y i ∈ Y .
Revised: =1 containing n pairs ( x i , y i ) (assumed to be sampled i.i.d.) is used, where x i ∈ X and y i ∈ Y .

Original: Given this dataset, the parameters θ of the mapping f θ are learned by minimizing a loss function (cid:96) ( x,y ) ( θ ) composed of a regularization term
Revised: Given this dataset, the parameters θ of the mapping f θ are learned by minimizing a loss function (cid:96) ( x,y ) ( θ ) composed of a regularization term

Original: R ( θ ) and an empirical risk (cid:96) ( y, f θ ( x )) which compares groundtruth label y and prediction f θ ( x ) .
Revised: R ( θ ) and an empirical risk (cid:96) ( y, f θ ( x )) which compares groundtruth label y and prediction f θ ( x ) .

Original: The parameters θ are obtained by optimizing the following program:
Revised: The parameters θ are obtained by optimizing the following program:

Original: [Equation]
Revised: [Equation]

Original: The mapping f θ is obtained by maximizing the logits or a corresponding probability distribution ˆ f θ ( y | x ) , i.e ., f θ = arg max y ∈Y ˆ f θ ( y | x ) .
Revised: The mapping f θ is obtained by maximizing the logits or a corresponding probability distribution ˆ f θ ( y | x ) , i.e ., f θ = arg max y ∈Y ˆ f θ ( y | x ) .

Original: Here and below let the hat (‘ ˆ · ’) indicate probability distributions over appropriate domains.
Revised: Here and below let the hat (‘ ˆ · ’) indicate probability distributions over appropriate domains.

Original: Reinforcement Learning considers an agent interacting with an environment according to a policy π θ π : X
Revised: Reinforcement Learning considers an agent interacting with an environment according to a policy π θ π : X

Original: → A which maps a state x t ∈ X to an action a t ∈ A at time t .
Revised: → A which maps a state x t ∈ X to an action a t ∈ A at time t .

Original: The policy depends on the parameters θ π .
Revised: The policy depends on the parameters θ π .

Original: After performing action a t , the agent observes the next state x t +1 and receives a scalar reward r t .
Revised: After performing action a t , the agent observes the next state x t +1 and receives a scalar reward r t .

Original: The discounted return at time t is deﬁned as R t = (cid:80) ∞
Revised: The discounted return at time t is deﬁned as R t = (cid:80) ∞

Original: k =0 γ k r t + k , where γ is the discount factor.
Revised: k =0 γ k r t + k , where γ is the discount factor.

Original: The expected future reward when observing state x and when following policy π θ π is deﬁned as V π θπ ( x ) =
Revised: The expected future reward when observing state x and when following policy π θ π is deﬁned as V π θπ ( x ) =

Original: E t [ R t | x ] .
Revised: E t [ R t | x ] .

Original: The goal of reinforcement learning is to ﬁnd a policy that maximizes the expected future reward from each state x t .
Revised: The goal of reinforcement learning is to ﬁnd a policy that maximizes the expected future reward from each state x t .

Original: Without loss of generality, in this paper, we follow the asynchronous advantage actor-critic (A3C) formulation (Mnih et al., 2016).
Revised: Without loss of generality, in this paper, we follow the asynchronous advantage actor-critic (A3C) formulation (Mnih et al., 2016).

Original: In A3C, the policy mapping π θ π ( x ) =
Revised: In A3C, the policy mapping π θ π ( x ) =

Original: arg max a ∈A ˆ π θ π ( a | x ) is obtained from a probability distribution over states, where ˆ π θ π ( a | x ) is modeled by a deep net with parameters θ π .
Revised: arg max a ∈A ˆ π θ π ( a | x ) is obtained from a probability distribution over states, where ˆ π θ π ( a | x ) is modeled by a deep net with parameters θ π .

Original: The value function is also approximated by a deep net V θ v ( x ) , having parameters θ v .
Revised: The value function is also approximated by a deep net V θ v ( x ) , having parameters θ v .

Original: To optimize the policy parameters θ π given a state x t , a loss function based on a scaled negative log-likelihood and a negative entropy regularizer is common:
Revised: To optimize the policy parameters θ π given a state x t , a loss function based on a scaled negative log-likelihood and a negative entropy regularizer is common:

Original: [Equation]
Revised: [Equation]

Original: Hereby, R t = (cid:80) k − 1 i =

Revised: Hereby, R t = (cid:80) k − 1 i =

Original: γ i r t
Revised: γ i r t

Original: + i + γ k V θ v ( x t + k ) is the empirical return obtained when starting in state x t .
Revised: + i + γ k V θ v ( x t + k ) is the empirical return obtained when starting in state x t .

Original: β is a user-speciﬁed constant, and H (ˆ π θ π ( ·| x t ) is the entropy function, which encourages exploration by favoring a uniform probability distribution
Revised: β is a user-speciﬁed constant, and H (ˆ π θ π ( ·| x t ) is the entropy function, which encourages exploration by favoring a uniform probability distribution

Original: ˆ π θ π ( a | x ) .
Revised: ˆ π θ π ( a | x ) .

Original: To optimize the value function V θ v it is common to use the squared loss (cid:96) tv ( θ v )
Revised: To optimize the value function V θ v it is common to use the squared loss (cid:96) tv ( θ v )

Original: =
Revised: =

Original: 12 ( R t − V θ v ( x t )) 2 .
Revised: 12 ( R t − V θ v ( x t )) 2 .

Original: By minimizing the empirical expectation of (cid:96) tπ ( θ π ) and (cid:96) tv ( θ v ) computed over t , i.e ., by addressing
Revised: By minimizing the empirical expectation of (cid:96) tπ ( θ π ) and (cid:96) tv ( θ v ) computed over t , i.e ., by addressing

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: alternatingly, we learn a policy and a value function that maximize expected return.
Revised: alternatingly, we learn a policy and a value function that maximize expected return.

Original:  K NOWLEDGE F LOW
Revised:  K NOWLEDGE F LOW

Original: Instead of optimizing the programs given in Eq.
Revised: Instead of optimizing the programs given in Eq.

Original: (1) and Eq.
Revised: (1) and Eq.

Original: (2) from scratch, the aforementioned warm-start techniques (see Sec.
Revised: (2) from scratch, the aforementioned warm-start techniques (see Sec.

Original: 5 for more) are applicable.
Revised: 5 for more) are applicable.

Original: To address their mentioned shortcomings, we propose knowledge ﬂow , a framework that moves ‘knowledge’ from an arbitrary number of deep nets, henceforth referred to as ‘teachers’ to a deep net under training, called the ‘student.’
Revised: To address their mentioned shortcomings, we propose knowledge ﬂow , a framework that moves ‘knowledge’ from an arbitrary number of deep nets, henceforth referred to as ‘teachers’ to a deep net under training, called the ‘student.’

Original: 3. O VERVIEW
Revised: 3. O VERVIEW

Original: Knowledge ﬂow is outlined on example deep nets in Fig.
Revised: Knowledge ﬂow is outlined on example deep nets in Fig.

Original: 1 (a,b).
Revised: 1 (a,b).

Original: We train the parameters of the student net which are randomly initialized.
Revised: We train the parameters of the student net which are randomly initialized.

Original: To this end we take advantage of teachers, whose parameters are ﬁxed and obtained from pre-trained models on different source tasks by different algorithms.
Revised: To this end we take advantage of teachers, whose parameters are ﬁxed and obtained from pre-trained models on different source tasks by different algorithms.

Original: For example, for reinforcement learning, we may consider teachers trained by A3C (Mnih et al., 2016),
Revised: For example, for reinforcement learning, we may consider teachers trained by A3C (Mnih et al., 2016),

Original: A2C (Dhariwal et al., 2017) or DQN (Mnih et al., 2015).
Revised: A2C (Dhariwal et al., 2017) or DQN (Mnih et al., 2015).

Original: ‘Knowledge’ of multiple teachers is transferred to a student by adding transformed and scaled intermediate representations from the teacher deep nets to the student net.
Revised: ‘Knowledge’ of multiple teachers is transferred to a student by adding transformed and scaled intermediate representations from the teacher deep nets to the student net.

Original: To achieve this, we modify
Revised: To achieve this, we modify

Original: Teacher2( ! (#) )
Revised: Teacher2( ! (#) )

Original: Teacher

Revised: Teacher

Original: Student ( ! (&) ) layer layer layer layer2 layer layer layer layer + + layer layer layer layer + + ' % (( %% )) %
Revised: Student ( ! (&) ) layer layer layer layer2 layer layer layer layer + + layer layer layer layer + + ' % (( %% )) %

Original: (( %% ) *(( %% ) *(( &% ) ℎ(( &% ) *(( %# ) ' %
Revised: (( %% ) *(( %% ) *(( &% ) ℎ(( &% ) *(( %# ) ' %

Original: Teacher1( ! (%) )
Revised: Teacher1( ! (%) )

Original: Student ( ! (&) )
Revised: Student ( ! (&) )

Original: (a) (b) (c) ℎ
Revised: (a) (b) (c) ℎ

Original: (( %% ) - % = {( &% ,( %% ,( ## } - # = {( &# ,( %# ,( #2 } the student net, i.e ., f θ in the supervised setting and π θ π ( a | x ) , V θ v ( x ) in the reinforcement learning case.
Revised: (( %% ) - % = {( &% ,( %% ,( ## } - # = {( &# ,( %# ,( #2 } the student net, i.e ., f θ in the supervised setting and π θ π ( a | x ) , V θ v ( x ) in the reinforcement learning case.

Original: We add teacher representations which are transformed via a trainable multiplication matrix Q and scaled via a weight p w that is normalized to sum to one for each student layer and parameterized via trainable parameters w .
Revised: We add teacher representations which are transformed via a trainable multiplication matrix Q and scaled via a weight p w that is normalized to sum to one for each student layer and parameterized via trainable parameters w .

Original: The normalized weights encode which of the teachers’ or the student’s representation to trust at every layer of the student net.
Revised: The normalized weights encode which of the teachers’ or the student’s representation to trust at every layer of the student net.

Original: Note that a teacher can help the student at different levels of abstraction with input from different levels of its net.
Revised: Note that a teacher can help the student at different levels of abstraction with input from different levels of its net.

Original: Importantly, after training, the student model should perform well on the target task without relying on teachers.
Revised: Importantly, after training, the student model should perform well on the target task without relying on teachers.

Original: To achieve this, as training progresses, we increasingly encourage a high normalized weight on the student representation, which forces the student to eventually capture all the ‘knowledge.’
Revised: To achieve this, as training progresses, we increasingly encourage a high normalized weight on the student representation, which forces the student to eventually capture all the ‘knowledge.’

Original: Due to the trainable scaling, at an early stage of training, we observe the student to rely heavily on the ‘knowledge’ of the teacher to quickly obtain better performance.
Revised: Due to the trainable scaling, at an early stage of training, we observe the student to rely heavily on the ‘knowledge’ of the teacher to quickly obtain better performance.

Original: However, as training proceeds, the student is encouraged to become more and more independent.
Revised: However, as training proceeds, the student is encouraged to become more and more independent.

Original: During ﬁnal stages of training, the student will no longer be able to rely on teachers, which ensures that the student has learned to master the desired task on its own.
Revised: During ﬁnal stages of training, the student will no longer be able to rely on teachers, which ensures that the student has learned to master the desired task on its own.

Original: This is observed in Fig.
Revised: This is observed in Fig.

Original: 1 (c).
Revised: 1 (c).

Original: To formally encourage this successive transfer we introduce two additional loss functions.
Revised: To formally encourage this successive transfer we introduce two additional loss functions.

Original: The ﬁrst, referred to as the dependency loss (cid:96) dep ( w ) , captures how much a student relies on teachers.
Revised: The ﬁrst, referred to as the dependency loss (cid:96) dep ( w ) , captures how much a student relies on teachers.

Original: It depends on the weight vector w which encodes the strength of the coupling.
Revised: It depends on the weight vector w which encodes the strength of the coupling.

Original: The second one ensures that a student’s behavior doesn’t change rapidly when the teachers’ inﬂuence decreases.
Revised: The second one ensures that a student’s behavior doesn’t change rapidly when the teachers’ inﬂuence decreases.

Original: We use loss (cid:96) KL ( · , · ) to capture the change.
Revised: We use loss (cid:96) KL ( · , · ) to capture the change.

Original: By combining student net modiﬁcations and additional loss terms, for the supervised task we obtain
Revised: By combining student net modiﬁcations and additional loss terms, for the supervised task we obtain

Original: [Equation]
Revised: [Equation]

Original: and for reinforcement learning the transformed program reads as follows:
Revised: and for reinforcement learning the transformed program reads as follows:

Original: [Equation]
Revised: [Equation]

Original: Loss ˜ (cid:96) ·· ( θ, w, Q ) originates from the original loss (cid:96) ·· ( θ ) (Eqs.
Revised: Loss ˜ (cid:96) ·· ( θ, w, Q ) originates from the original loss (cid:96) ·· ( θ ) (Eqs.

Original: by transforming the deep net to include cross-connections, hence its dependence on w, Q .
Revised: by transforming the deep net to include cross-connections, hence its dependence on w, Q .

Original: The tilde (‘ ˜ · ’) notes this dependence, also for probability distribution
Revised: The tilde (‘ ˜ · ’) notes this dependence, also for probability distribution

Original: ˜ˆ f and policy distribution
Revised: ˜ˆ f and policy distribution

Original: ˜ˆ π .
Revised: ˜ˆ π .

Original: Parameters from the current and a previous iteration are referred to via θ and θ old respectively.
Revised: Parameters from the current and a previous iteration are referred to via θ and θ old respectively.

Original: For both supervised and reinforcement learning, λ 1 and λ 2 control the strength which is used to decrease the inﬂuence of the teacher.
Revised: For both supervised and reinforcement learning, λ 1 and λ 2 control the strength which is used to decrease the inﬂuence of the teacher.

Original: A low λ 1 allows the student to rely on teachers.
Revised: A low λ 1 allows the student to rely on teachers.

Original: Close to the end of training, the student should be independent.
Revised: Close to the end of training, the student should be independent.

Original: Therefore, we set λ 1 to a small value at the beginning, and gradually increase its value as training progresses.
Revised: Therefore, we set λ 1 to a small value at the beginning, and gradually increase its value as training progresses.

Original: Note that we don’t make any assumptions about teachers and student’s objective.
Revised: Note that we don’t make any assumptions about teachers and student’s objective.

Original: If a teacher’s and student’s objective differ, negative transfer may occur initially.
Revised: If a teacher’s and student’s objective differ, negative transfer may occur initially.

Original: However, our method quickly decreases the weight for teacher layers to reduce this effect.
Revised: However, our method quickly decreases the weight for teacher layers to reduce this effect.

Original: Despite differences, students could potentially still beneﬁt from the low level representation of the teachers.
Revised: Despite differences, students could potentially still beneﬁt from the low level representation of the teachers.

Original: We do observe this low level knowledge transfer in our experiments.
Revised: We do observe this low level knowledge transfer in our experiments.

Original: In the following we ﬁrst describe how to modify the deep nets, before we detail the loss functions (cid:96) dep and (cid:96) KL , which are used to successively decrease the inﬂuence of the teachers.
Revised: In the following we ﬁrst describe how to modify the deep nets, before we detail the loss functions (cid:96) dep and (cid:96) KL , which are used to successively decrease the inﬂuence of the teachers.

Original: 3. D EEP N ET T RANSFORMATION AND L OSS T ERMS
Revised: 3. D EEP N ET T RANSFORMATION AND L OSS T ERMS

Original: Deep Net Transformation: Knowledge ﬂow enhances the student by adding transformed and scaled intermediate representations from teacher models.
Revised: Deep Net Transformation: Knowledge ﬂow enhances the student by adding transformed and scaled intermediate representations from teacher models.

Original: To perform the transformation, intermediate representations from teachers are ﬁrst multiplied by transformation matrices Q .
Revised: To perform the transformation, intermediate representations from teachers are ﬁrst multiplied by transformation matrices Q .

Original: Then the transformed representations from teachers and representations from the student are linearly combined.
Revised: Then the transformed representations from teachers and representations from the student are linearly combined.

Original: The weights for this linear combination are determined by a weight p w which is normalized to sum to one for each student layer.
Revised: The weights for this linear combination are determined by a weight p w which is normalized to sum to one for each student layer.

Original: Let index m =
Revised: Let index m =

Original: denote the student model and let θ (0) refer to its parameters.
Revised: denote the student model and let θ (0) refer to its parameters.

Original: Further, let θ ( m ) , m ∈ { 1 , . . .
Revised: Further, let θ ( m ) , m ∈ { 1 , . . .

Original: , M } denote teacher models.
Revised: , M } denote teacher models.

Original: We use l im to refer to deep net layer i of teacher m , with i ∈ { 1 , . . .
Revised: We use l im to refer to deep net layer i of teacher m , with i ∈ { 1 , . . .

Original: , L m } and L m the number of layers in teacher m .
Revised: , L m } and L m the number of layers in teacher m .

Original: We deﬁne layer j of the student model to be l j 0 , where j ∈ { 1 , . . .
Revised: We deﬁne layer j of the student model to be l j 0 , where j ∈ { 1 , . . .

Original: , L 0 } and L 0 the number of deep net layers in the student model.
Revised: , L 0 } and L 0 the number of deep net layers in the student model.

Original: The output of layer l km right before and after an activation unit is denoted z ( l km ) and h ( l km ) respectively.
Revised: The output of layer l km right before and after an activation unit is denoted z ( l km ) and h ( l km ) respectively.

Original: To align a teacher’s layer l im with a student’s layer l j 0 , we introduce a learnable transformation matrix
Revised: To align a teacher’s layer l im with a student’s layer l j 0 , we introduce a learnable transformation matrix

Original: Q j ( l im ) ∈ R dim( l j 0 ) × dim( l i m ) , where dim( · ) gives the number of elements in the corresponding layer.
Revised: Q j ( l im ) ∈ R dim( l j 0 ) × dim( l i m ) , where dim( · ) gives the number of elements in the corresponding layer.

Original: The matrix multiplication Q j ( l im )
Revised: The matrix multiplication Q j ( l im )

Original: z ( l im ) aligns the representation from layer i of teacher m with the representation of layer j of the student.
Revised: z ( l im ) aligns the representation from layer i of teacher m with the representation of layer j of the student.

Original: For each layer j in the student model, we deﬁne a candidate set L j , which contains l j 0 and all the teachers’ layers to be considered.
Revised: For each layer j in the student model, we deﬁne a candidate set L j , which contains l j 0 and all the teachers’ layers to be considered.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 1 (a), layer one of the student model is combined with layer one of teacher one and layer two of teacher two.
Revised: 1 (a), layer one of the student model is combined with layer one of teacher one and layer two of teacher two.

Original: Therefore, the candidate set of layer one of the student model is given by L 1 = { l 10 , l 11 , l 22 } .
Revised: Therefore, the candidate set of layer one of the student model is given by L 1 = { l 10 , l 11 , l 22 } .

Original: To decide which teachers’ or the student’s representation to trust at every layer of the student net, we introduce a normalized weight p jw ( l ) for all j ∈ { 1 , . . .
Revised: To decide which teachers’ or the student’s representation to trust at every layer of the student net, we introduce a normalized weight p jw ( l ) for all j ∈ { 1 , . . .

Original: , L 0 } , where l ∈ L j , summing to one for each layer j in the student deep net, i.e ., (cid:88) l ∈
Revised: , L 0 } , where l ∈ L j , summing to one for each layer j in the student deep net, i.e ., (cid:88) l ∈

Original: L j
Revised: L j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To obtain the combined intermediate representation of layer j for the student model, we use
Revised: To obtain the combined intermediate representation of layer j for the student model, we use

Original: [Equation]
Revised: [Equation]

Original: where p jw ( l im ) determines how much the student layer j relies on transformed representations of layer i from the m -th teacher.
Revised: where p jw ( l im ) determines how much the student layer j relies on transformed representations of layer i from the m -th teacher.

Original: Intuitively, if the transformed representation of the m -th teacher layer i is helpful, p jw ( l im ) will be close to one.
Revised: Intuitively, if the transformed representation of the m -th teacher layer i is helpful, p jw ( l im ) will be close to one.

Original: We visualize the deep net transformation in Fig.
Revised: We visualize the deep net transformation in Fig.

Original: 1 (b).
Revised: 1 (b).

Original: Note that the intermediate representations of teachers are not changed in our framework.
Revised: Note that the intermediate representations of teachers are not changed in our framework.

Original: To obtain the output of layer l km we apply the original activation unit to the original representation z ( l im ) , i.e ., h ( l im )
Revised: To obtain the output of layer l km we apply the original activation unit to the original representation z ( l im ) , i.e ., h ( l im )

Original: =
Revised: =

Original: σ ( z ( l im )) , ∀ m ∈ { 1 , . .
Revised: σ ( z ( l im )) , ∀ m ∈ { 1 , . .

Original: , M } , j ∈ { 1 , . . .
Revised: , M } , j ∈ { 1 , . . .

Original: L m } .
Revised: L m } .

Original: Also note that while additional trainable parameters Q and w are introduced in our framework, Q and w are not part of the resulting student network since we ensure p jw ( l ) ≡ 0 ∀ l ∈ L j \ l j 0 at the end of training as discussed next.
Revised: Also note that while additional trainable parameters Q and w are introduced in our framework, Q and w are not part of the resulting student network since we ensure p jw ( l ) ≡ 0 ∀ l ∈ L j \ l j 0 at the end of training as discussed next.

Original: Hence, the additional parameters function as auxiliary knobs that help the student learn faster.
Revised: Hence, the additional parameters function as auxiliary knobs that help the student learn faster.

Original: In the ﬁnal stage of training, the student will be independent (see Fig.
Revised: In the ﬁnal stage of training, the student will be independent (see Fig.

Original: 1 (c)) and does no longer rely on Q , w , or any transformed representations from teachers.
Revised: 1 (c)) and does no longer rely on Q , w , or any transformed representations from teachers.

Original: Decreasing Teachers’ Inﬂuence: We successively decrease the inﬂuence of the teachers during training by gradually encouraging the normalized weight p jw ( l j 0 ) to increase to a value of 1 ∀ j ∈ { 1 , 2 , . . .
Revised: Decreasing Teachers’ Inﬂuence: We successively decrease the inﬂuence of the teachers during training by gradually encouraging the normalized weight p jw ( l j 0 ) to increase to a value of 1 ∀ j ∈ { 1 , 2 , . . .

Original: , L 0 } .
Revised: , L 0 } .

Original: To capture how much the student relies on teachers, we introduce the dependence cost as the negative log probability:
Revised: To capture how much the student relies on teachers, we introduce the dependence cost as the negative log probability:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: By minimizing (cid:96) dep ( w )
Revised: By minimizing (cid:96) dep ( w )

Original: [Table]
Revised: [Table]

Original: we encourage weights for the layers of the student to increase.
Revised: we encourage weights for the layers of the student to increase.

Original: Hence we encourage the student to become more and more independent.
Revised: Hence we encourage the student to become more and more independent.

Original: During the ﬁnal stage of training, p jw ( l j 0 ) approaches one for all j ∈ { 1 , . . .
Revised: During the ﬁnal stage of training, p j w ( l j 0 ) approaches one for all j ∈ { 1 , . . .

Original: , L 0 } , making the student independent of the transformed representation obtained from teachers.
Revised: , L 0 } , making the student independent of the transformed representation obtained from teachers.

Original: Empirically, we found that a fast decrease of the inﬂuence of the teacher can degrade the performance.
Revised: Empirically, we found that a fast decrease of the inﬂuence of the teacher can degrade the performance.

Original: This is intuitive as it requires some time to ﬁnd good transformations Q .
Revised: This is intuitive as it requires some time to ﬁnd good transformations Q .

Original: Moreover, decreasing the inﬂuence of a teacher too fast may change the output distribution over labels or actions of the student model too much, and thus lead to performance loss.
Revised: Moreover, decreasing the inﬂuence of a teacher too fast may change the output distribution over labels or actions of the student model too much, and thus lead to performance loss.

Original: To prevent changing a student’s output distribution too fast, we found a Kullback-Leibler (KL) regularizer to yield good results.
Revised: To prevent changing a student’s output distribution too fast, we found a Kullback-Leibler (KL) regularizer to yield good results.

Original: More speciﬁcally, in the case of supervised learning we use
Revised: More speciﬁcally, in the case of supervised learning we use

Original: [Equation]
Revised: [Equation]

Original: Hereby, θ is the set of current parameters, and θ old are the previous ones.
Revised: Hereby, θ is the set of current parameters, and θ old are the previous ones.

Original: In the reinforcement learning case we use D KL
Revised: In the reinforcement learning case we use D KL

Original: [˜ˆ π θ ( ·| x t ) ||
Revised: [˜ˆ π θ ( ·| x t ) ||

Original: ˜ˆ π θ old ( ·| x t )] .
Revised: ˜ˆ π θ old ( ·| x t )] .

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In the following we evaluate knowledge ﬂow on reinforcement and supervised learning tasks.
Revised: In the following we evaluate knowledge ﬂow on reinforcement and supervised learning tasks.

Original: Results are reported by using only the student model to avoid even the smallest inﬂuence from any teacher nets.
Revised: Results are reported by using only the student model to avoid even the smallest inﬂuence from any teacher nets.

Original: 4. R EINFORCEMENT L EARNING
Revised: 4. R EINFORCEMENT L EARNING

Original: We evaluate knowledge ﬂow on reinforcement learning using Atari games that were used by Rusu et al.
Revised: We evaluate knowledge ﬂow on reinforcement learning using Atari games that were used by Rusu et al.

Original: (2016b)
Revised: (2016b)

Original: ; Fernando et al.
Revised: ; Fernando et al.

Original: Following existing work, the input to our agent are raw images from the environment.
Revised: Following existing work, the input to our agent are raw images from the environment.

Original: The agent learns to predict actions only based on the rewards and the input images from the environment.
Revised: The agent learns to predict actions only based on the rewards and the input images from the environment.

Original: The agent chooses an action every four frames, and the last action is repeated on the skipped four frames.
Revised: The agent chooses an action every four frames, and the last action is repeated on the skipped four frames.

Original: For all teacher models and the student model, we use the fully forward architecture of A3C (Mnih et al., 2016).
Revised: For all teacher models and the student model, we use the fully forward architecture of A3C (Mnih et al., 2016).

Original: The model has three hidden layers.
Revised: The model has three hidden layers.

Original: The ﬁrst layer is a convolutional layer with 16 ﬁlters of size 8x8 and stride 4.
Revised: The ﬁrst layer is a convolutional layer with 16 ﬁlters of size 8x8 and stride 4.

Original: The second layer is a convolutional layer with 32 ﬁlters of size 4x4 and stride 2.
Revised: The second layer is a convolutional layer with 32 ﬁlters of size 4x4 and stride 2.

Original: The third layer is a fully connected layer with 256 hidden units.
Revised: The third layer is a fully connected layer with 256 hidden units.

Original: Following the third hidden layer are two sets of output.
Revised: Following the third hidden layer are two sets of output.

Original: One is a softmax output that provides a probability distribution over all valid actions.
Revised: One is a softmax output that provides a probability distribution over all valid actions.

Original: The other one is a scalar output that provides the estimated value function.
Revised: The other one is a scalar output that provides the estimated value function.

Original: We use the same hyper-parameter settings as Mnih et al.
Revised: We use the same hyper-parameter settings as Mnih et al.

Original: (2016) except for the learning rate.
Revised: (2016) except for the learning rate.

Original: Mnih et al.
Revised: Mnih et al.

Original: (2016) use RMSProp with shared statistics while we use Adam with shared statistics, which we found to give better results when training the baselines.
Revised: (2016) use RMSProp with shared statistics while we use Adam with shared statistics, which we found to give better results when training the baselines.

Original: The learning rate is set to 10 − 4 and gradually decreased to zero for all experiments.
Revised: The learning rate is set to 10 − 4 and gradually decreased to zero for all experiments.

Original: To select λand λ 2 in our framework, we follow progressive neural net (Rusu et al., 2016b): randomly sample λ 1 ∈ { 0 . 05 , 0 . 1 , 0 . 5 } and λ 2 ∈ { 0 . 001 , 0 . 01 , 0 . 05 } .
Revised: To select λand λ 2 in our framework, we follow progressive neural net (Rusu et al., 2016b): randomly sample λ 1 ∈ { 0 . 05 , 0 . 1 , 0 . 5 } and λ 2 ∈ { 0 . 001 , 0 . 01 , 0 . 05 } .

Original: Note that λ 1 is set to zero at the beginning of training, and linearly increased to the sampled value at the end of training.
Revised: Note that λ 1 is set to zero at the beginning of training, and linearly increased to the sampled value at the end of training.

Original: Following Rusu et al.
Revised: Following Rusu et al.

Original: (2016b), we repeat each experiment 25 times with different random seeds and randomly sampled λand λ 2 .
Revised: (2016b), we repeat each experiment 25 times with different random seeds and randomly sampled λand λ 2 .

Original: The results of the top three out of 25 runs are reported.
Revised: The results of the top three out of 25 runs are reported.

Original: As A3C, we run 16 agents on
Revised: As A3C, we run 16 agents on

Original: CPU cores in parallel.
Revised: CPU cores in parallel.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Evaluation Metrics: We follow the evaluation procedure of Mnih et al.
Revised: Evaluation Metrics: We follow the evaluation procedure of Mnih et al.

Original: The trained student models are evaluated by playing each game for 30 episodes.
Revised: The trained student models are evaluated by playing each game for 30 episodes.

Original: We also follow the ‘no-op’ procedure: at the beginning of each testing episode, the agents perform up to 30 ‘no-op’ actions.
Revised: We also follow the ‘no-op’ procedure: at the beginning of each testing episode, the agents perform up to 30 ‘no-op’ actions.

Original: Results:
Revised: Results:

Original: We ﬁrst compare our framework with PathNet (Fernando et al., 2017) and progressive neural net (PNN) (Rusu et al., 2016b), which are state-of-the-art transfer reinforcement learning frameworks, using their experimental settings.
Revised: We ﬁrst compare our framework with PathNet (Fernando et al., 2017) and progressive neural net (PNN) (Rusu et al., 2016b), which are state-of-the-art transfer reinforcement learning frameworks, using their experimental settings.

Original: The comparison is summarized in Table 1.
Revised: The comparison is summarized in Table 1.

Original: The state-of-the-art results (Mnih et al., 2016; Schulman et al., 2017; Wu et al., 2017) on Atari games are also included in
Revised: The state-of-the-art results (Mnih et al., 2016; Schulman et al., 2017; Wu et al., 2017) on Atari games are also included in

Original: Table 1 for reference.
Revised: Table 1 for reference.

Original: Compared to PathNet, a student model trained using our transfer framework with one teacher achieves higher scores in 11 out of 14 experiments.
Revised: Compared to PathNet, a student model trained using our transfer framework with one teacher achieves higher scores in 11 out of 14 experiments.

Original: Compared with PNN, for a two-teacher framework, our trained student model has only 0.7M parameters and PNN has 16M parameters.
Revised: Compared with PNN, for a two-teacher framework, our trained student model has only 0.7M parameters and PNN has 16M parameters.

Original: Nonetheless we observe higher scores in ﬁve out of the seven experiments.
Revised: Nonetheless we observe higher scores in ﬁve out of the seven experiments.

Original: The results demonstrate that knowledge ﬂow effectively transfers knowledge from teachers to the student.
Revised: The results demonstrate that knowledge ﬂow effectively transfers knowledge from teachers to the student.

Original: Tablealso indicates that, in our framework, when the number of teachers increases from one to two, the student’s performance improves signiﬁcantly across all experiments.
Revised: Tablealso indicates that, in our framework, when the number of teachers increases from one to two, the student’s performance improves signiﬁcantly across all experiments.

Original: The training curves for the experiments are shown in Fig.
Revised: The training curves for the experiments are shown in Fig.

Original: The curve is the average of the top three out of 25 runs.
Revised: The curve is the average of the top three out of 25 runs.

Original: We observe our approach to generally perform very well.
Revised: We observe our approach to generally perform very well.

Original: To further evaluate knowledge ﬂow , we experiment with different combinations of environment/teacher settings.
Revised: To further evaluate knowledge ﬂow , we experiment with different combinations of environment/teacher settings.

Original: These settings are not used by PathNet and progressive neural network.
Revised: These settings are not used by PathNet and progressive neural network.

Original: The results are summarized in Tab. 2, where “ours w/ expert” represents that one teacher is expert for the target game; “ours w/ non-expert” represents that both teachers are notexperts for the target game; “Fine-tune” represents ﬁne-tuning from a non-expert on a new target game; “A3C baseline” represents our implementation of the A3C baseline; “A3C” represents the scores reported originally (Mnih et al.,
Revised: The results are summarized in Table 2, where “ours w/ expert” represents that one teacher is expert for the target game; “ours w/ non-expert” represents that both teachers are not

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: our implementation of the A3C baseline; “A3C” represents the scores reported originally (Mnih et al., 2016).

Original: Note that our A3C implementation achieves better scores than those reported by Mnih et al.
Revised: Note that our A3C implementation achieves better scores than those reported by Mnih et al.

Original: (2016) for most of the games.
Revised: (2016) for most of the games.

Original: As shown in Tab. 2, knowledge ﬂow with expert teacher performs better than the baseline across all experiments, which we interpret as evidence that knowledge ﬂow successfully transfers ‘knowledge’ from an expert teacher to the student.
Revised: As shown in Table 2, knowledge ﬂow with expert teacher performs better than the baseline across all experiments, which we interpret as evidence that knowledge ﬂow successfully transfers ‘knowledge’ from an expert teacher to the student.

Original: In addition, knowledge ﬂow with non-expert teachers also outperforms ﬁne-tuning on a non-expert teacher.
Revised: In addition, knowledge ﬂow with non-expert teachers also outperforms ﬁne-tuning on a non-expert teacher.

Original: The reasons are twofold: First, a student model in knowledge ﬂow can learn from multiple teachers while the ﬁne-tuning method can only start from one setting.
Revised: The reasons are twofold: First, a student model in knowledge ﬂow can learn from multiple teachers while the ﬁne-tuning method can only start from one setting.

Original: Second, in knowledge ﬂow , the student can avoid the negative impact from insufﬁciently pretrained teachers, while ﬁne-tuning from an insufﬁciently pretrained model slows down the training process and may degrade the overall performance.
Revised: Second, in knowledge ﬂow , the student can avoid the negative impact from insufﬁciently pretrained teachers, while ﬁne-tuning from an insufﬁciently pretrained model slows down the training process and may degrade the overall performance.

Original: The training curves for the experiments are shown in Fig.
Revised: The training curves for the experiments are shown in Fig.

Original: More training curves are in the Appendix (Fig. 6).
Revised: More training curves are in the Appendix (Fig. 6).

Original: Note that in knowledge ﬂow , the student can beneﬁt from the intermediate representations of the teacher, even if input space, output space and objectives differ.
Revised: Note that in knowledge ﬂow , the student can beneﬁt from the intermediate representations of the teacher, even if input space, output space and objectives differ.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 3 (a), the two teachers are Chopper Command and Space Invaders, which are quite different from the target game
Revised: 3 (a), the two teachers are Chopper Command and Space Invaders, which are quite different from the target game

Original: Seaquest.
Revised: Seaquest.

Original: The student model still beneﬁts from learning from the teachers and achieves scores ten times larger than learning without teacher and ﬁne-tuning from a teacher.
Revised: The student model still beneﬁts from learning from the teachers and achieves scores ten times larger than learning without teacher and ﬁne-tuning from a teacher.

Original: 4. S UPERVISED L EARNING
Revised: 4. S UPERVISED L EARNING

Original: For supervised learning, we use a variety of image classiﬁcation benchmarks, including CIFAR10 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), STL-10 (Coates et al., 2011), and EMNIST (Cohen et al., 2017).
Revised: For supervised learning, we use a variety of image classiﬁcation benchmarks, including CIFAR10 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), STL-10 (Coates et al., 2011), and EMNIST (Cohen et al., 2017).

Original: The parameters λ 1 for the dependent cost and λ 2 for the KL cost are determined using the validation set of each dataset.
Revised: The parameters λ 1 for the dependent cost and λ 2 for the KL cost are determined using the validation set of each dataset.

Original: Evaluation Metrics: To evaluate the trained student model we report top-1 error rate on the test set of each dataset.
Revised: Evaluation Metrics: To evaluate the trained student model we report top-1 error rate on the test set of each dataset.

Original: All plots and reported numbers are the average of three runs obtained using different random seeds.
Revised: All plots and reported numbers are the average of three runs obtained using different random seeds.

Original: CIFAR-10/CIFAR-100: CIFAR-10 and CIFAR-100 datasets consist of colored images of size 32 × 32 .
Revised: CIFAR-10/CIFAR-100: CIFAR-10 and CIFAR-100 datasets consist of colored images of size 32 × 32 .

Original: CIFAR-10 (C10) has 10 classes and CIFAR-100 (C100) has 100 classes.
Revised: CIFAR-10 (C10) has 10 classes and CIFAR-100 (C100) has 100 classes.

Original: For both dataset, the training and test sets contain 50,000 and 10,000 images respectively.
Revised: For both dataset, the training and test sets contain 50,000 and 10,000 images respectively.

Original: We perform all experiments on CIFAR-10 and CIFAR-100 with standard data augmentation (Huang et al., 2017).
Revised: We perform all experiments on CIFAR-10 and CIFAR-100 with standard data augmentation (Huang et al., 2017).

Original: We use Densenet (Huang et al., 2017) (depth 100, growth rate 24) as a baseline and follow their hyper-parameter settings to train our baseline, teacher and student models.
Revised: We use Densenet (Huang et al., 2017) (depth 100, growth rate 24) as a baseline and follow their hyper-parameter settings to train our baseline, teacher and student models.

Original: For our approach, we ﬁrst train teachers on CIFAR-10, CIFAR-100, and SVHN (Netzer et al., 2011).
Revised: For our approach, we ﬁrst train teachers on CIFAR-10, CIFAR-100, and SVHN (Netzer et al., 2011).

Original: We then train the student model using a different combination of teachers.
Revised: We then train the student model using a different combination of teachers.

Original: We compare our results to ﬁne-tuning and the baseline model.
Revised: We compare our results to ﬁne-tuning and the baseline model.

Original: As shown in Table 3 (a), for the CIFAR-10 target task, ﬁne-tuning from the
Revised: As shown in Table 3 (a), for the CIFAR-10 target task, ﬁne-tuning from the

Original: CIFAR-100 expert improves 4% over the baseline.
Revised: CIFAR-100 expert improves 4% over the baseline.

Original: Fine-tuning from the SVHN expert performs worse than the baseline model.
Revised: Fine-tuning from the SVHN expert performs worse than the baseline model.

Original: Intuitively, for the CIFAR-10 target task, the CIFAR-100 deep net is a good teacher while a deep net trained with SVHN isn’t.
Revised: Intuitively, for the CIFAR-10 target task, the CIFAR-100 deep net is a good teacher while a deep net trained with SVHN isn’t.

Original: Presented with both good and inadequate teachers, knowledge ﬂow improves by 13% over the baseline.
Revised: Presented with both good and inadequate teachers, knowledge ﬂow improves by 13% over the baseline.

Original: This demonstrates that knowledge ﬂow can not only leverage a good teacher’s ‘knowledge,’ but it can also avoid misleading inﬂuence.
Revised: This demonstrates that knowledge ﬂow can not only leverage a good teacher’s ‘knowledge,’ but it can also avoid misleading inﬂuence.

Original: As detailed in Table 3 (b), the results are similar on the CIFAR-100 dataset.
Revised: As detailed in Table 3 (b), the results are similar on the CIFAR-100 dataset.

Original: To further demonstrate the properties of knowledge ﬂow , additional results are in the appendix.
Revised: To further demonstrate the properties of knowledge ﬂow , additional results are in the appendix.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: As mentioned before, variants of ‘knowledge’ transfer have been considered using a variety of techniques, for instance, ﬁne-tuning, progressive neural nets (Rusu et al., 2016b), PathNet (Fernando et al., 2017), ‘Growing a Brain’ (Wang et al., 2017), actor-mimic (Parisotto et al., 2016), learning without forgetting (Li & Hoiem, 2016).
Revised: As mentioned before, variants of ‘knowledge’ transfer have been considered using a variety of techniques, for instance, ﬁne-tuning, progressive neural nets (Rusu et al., 2016b), PathNet (Fernando et al., 2017), ‘Growing a Brain’ (Wang et al., 2017), actor-mimic (Parisotto et al., 2016), learning without forgetting (Li & Hoiem, 2016).

Original: Also related are techniques on transfer learning and lifelong learning.
Revised: Also related are techniques on transfer learning and lifelong learning.

Original: We discuss those methods and contrast them to our approach in the following.
Revised: We discuss those methods and contrast them to our approach in the following.

Original: PathNet (Fernando et al., 2017) enables multiple agents to train the same giant deep net while reusing parameters and avoiding catastrophic forgetting.
Revised: PathNet (Fernando et al., 2017) enables multiple agents to train the same giant deep net while reusing parameters and avoiding catastrophic forgetting.

Original: To this end, agents embedded in the neural net discover which weights can be reused for new tasks and restrict application of gradients to those parameters.
Revised: To this end, agents embedded in the neural net discover which weights can be reused for new tasks and restrict application of gradients to those parameters.

Original: In contrast to this formulation we consider availability of multiple teacher nets, which are trained.
Revised: In contrast to this formulation we consider availability of multiple teacher nets, which are trained.

Original: Progressive Net (Rusu et al., 2016b) leverages transfer and avoids catastrophic forgetting by introducing lateral connections to previously learned features.
Revised: Progressive Net (Rusu et al., 2016b) leverages transfer and avoids catastrophic forgetting by introducing lateral connections to previously learned features.

Original: Our discussed method uses similar lateral connections.
Revised: Our discussed method uses similar lateral connections.

Original: However, in contrast to Rusu et al. (2016b), we introduce scaling with normalized weights.
Revised: However, in contrast to Rusu et al. (2016b), we introduce scaling with normalized weights.

Original: This ensures independence of the student upon training, addressing a limitation in (Rusu et al., 2016b) where only a fraction of the capacity of the student is eventually utilized.
Revised: This ensures independence of the student upon training, addressing a limitation in (Rusu et al., 2016b) where only a fraction of the capacity of the student is eventually utilized.

Original: Distral a neologism combining ‘distill & transfer learning’ (Teh et al., 2017) considers joint training of multiple tasks.
Revised: Distral a neologism combining ‘distill & transfer learning’ (Teh et al., 2017) considers joint training of multiple tasks.

Original: Multiple tasks share a ‘distilled’ policy which encodes common behavior between different tasks.
Revised: Multiple tasks share a ‘distilled’ policy which encodes common behavior between different tasks.

Original: While each worker addresses its own task, a shared policy encourages consistency between the policies.
Revised: While each worker addresses its own task, a shared policy encourages consistency between the policies.

Original: Different from this approach we aim at a more direct sharing of representations rather than an exchange via a hypothesized global policy.
Revised: Different from this approach we aim at a more direct sharing of representations rather than an exchange via a hypothesized global policy.

Original: Knowledge distillation (Hinton et al., 2015) distills information form a larger deep net into a smaller one.
Revised: Knowledge distillation (Hinton et al., 2015) distills information form a larger deep net into a smaller one.

Original: It assumes both nets are trained on the same dataset.
Revised: It assumes both nets are trained on the same dataset.

Original: In contrast, our technique allows knowledge transfer between different source and target domains.
Revised: In contrast, our technique allows knowledge transfer between different source and target domains.

Original: Actor-mimic (Parisotto et al., 2016) enables an agent to learn how to address multiple tasks simultaneously and generalize the extracted knowledge to new domains.
Revised: Actor-mimic (Parisotto et al., 2016) enables an agent to learn how to address multiple tasks simultaneously and generalize the extracted knowledge to new domains.

Original: A single policy net learns how to act in a set of tasks following the guidance of several expert teachers.
Revised: A single policy net learns how to act in a set of tasks following the guidance of several expert teachers.

Original: A combination of feature regression and cross entropy loss is used to encourage the student to produce similar actions and representations.
Revised: A combination of feature regression and cross entropy loss is used to encourage the student to produce similar actions and representations.

Original: Our proposed technique differs in that we take advantage of a teachers representation at the beginning of training,
Revised: Our proposed technique differs in that we take advantage of a teachers representation at the beginning of training,

Original: Learning without forgetting (Li & Hoiem, 2016) permits to add a new task to a deep net without forgetting the original capabilities.
Revised: Learning without forgetting (Li & Hoiem, 2016) permits to add a new task to a deep net without forgetting the original capabilities.

Original: Importantly, only data from the new task is used and the old capabilities are retained by ﬁrst recording the old networks output on the new data.
Revised: Importantly, only data from the new task is used and the old capabilities are retained by ﬁrst recording the old networks output on the new data.

Original: Similar techniques have been developed by Furlanello et al.
Revised: Similar techniques have been developed by Furlanello et al.

Original: ; Jung et al.
Revised: ; Jung et al.

Original: In contrast, we transfer ‘knowledge’ from teacher networks more explicitly.
Revised: In contrast, we transfer ‘knowledge’ from teacher networks more explicitly.

Original: Growing a Brain (Wang et al., 2017) analyzes the parameters which change during ﬁne-tuning and points out that more natural model adaptation is obtained when increasing the model capacity, by either extending width or depth.
Revised: Growing a Brain (Wang et al., 2017) analyzes the parameters which change during ﬁne-tuning and points out that more natural model adaptation is obtained when increasing the model capacity, by either extending width or depth.

Original: Appropriate normalization is essential to signiﬁcantly outperform classical ﬁne-tuning.
Revised: Appropriate normalization is essential to signiﬁcantly outperform classical ﬁne-tuning.

Original: Since this technique is based on ﬁne-tuning, it differs from our student-teacher based approach.
Revised: Since this technique is based on ﬁne-tuning, it differs from our student-teacher based approach.

Original: Other related work includes policy distillation (Rusu et al., 2016a), domain adaptation (Pan &
Revised: Other related work includes policy distillation (Rusu et al., 2016a), domain adaptation (Pan &

Original: Yang, 2010; Long et al., 2015; Tzeng et al., 2015) or lifelong learning (Chen & Liu, 2016; Thrun, 1998; Mitchell et al., 2015; Ruvolo & Eaton, 2013).
Revised: Yang, 2010; Long et al., 2015; Tzeng et al., 2015) or lifelong learning (Chen & Liu, 2016; Thrun, 1998; Mitchell et al., 2015; Ruvolo & Eaton, 2013).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We developed a general ‘knowledge transfer’ approach that permits to train a deep net from any number of teachers.
Revised: We developed a general ‘knowledge transfer’ approach that permits to train a deep net from any number of teachers.

Original: We showed results for reinforcement learning and supervised learning, demonstrating improvements compared to training from scratch and to ﬁne-tuning.
Revised: We showed results for reinforcement learning and supervised learning, demonstrating improvements compared to training from scratch and to ﬁne-tuning.

Original: In the future we plan to learn when to use which teacher and how to actively swap teachers during training of a student.
Revised: In the future we plan to learn when to use which teacher and how to actively swap teachers during training of a student.

Original:  A PPENDIX 7. S UPERVISED L EARNING
Revised:  A PPENDIX 7. S UPERVISED L EARNING

Original: Comparison with Knowledge Distillation: We follow knowledge Distillation (KD) (Hinton et al., 2015) to distill knowledge from a larger model (teacher) to a smaller model (student).
Revised: Comparison with Knowledge Distillation: We follow knowledge Distillation (KD) (Hinton et al., 2015) to distill knowledge from a larger model (teacher) to a smaller model (student).

Original: The student models have 50% - 5% parameters of the teacher models.
Revised: The student models have 50% - 5% parameters of the teacher models.

Original: Following their setup, we conduct experiments on MNIST, MNIST with digit ‘3’ missing in the training set, CIFAR-100, and ImageNet.
Revised: Following their setup, we conduct experiments on MNIST, MNIST with digit ‘3’ missing in the training set, CIFAR-100, and ImageNet.

Original: For MNIST and MNIST with digit ‘3’ missing, following KD, the teacher model is an MLP with two hidden layers of 1200 hidden units, and the student model is an MLP with two hidden layers ofhidden units.
Revised: For MNIST and MNIST with digit ‘3’ missing, following KD, the teacher model is an MLP with two hidden layers of 1200 hidden units, and the student model is an MLP with two hidden layers ofhidden units.

Original: For CIFAR-100, we use the model from Chen (2017) as teacher model.
Revised: For CIFAR-100, we use the model from Chen (2017) as teacher model.

Original: The student model follows the structure of the teacher, but the number of output channels of each convolutional layer is halved.
Revised: The student model follows the structure of the teacher, but the number of output channels of each convolutional layer is halved.

Original: For ImageNet, the teacher model is a 50-layer ResNet (He et al., 2016), and the student model is a 18-layer ResNet.
Revised: For ImageNet, the teacher model is a 50-layer ResNet (He et al., 2016), and the student model is a 18-layer ResNet.

Original: The test error of the distilled student model are summarize in
Revised: The test error of the distilled student model are summarize in

Original: Table 4.
Revised: Table 4.

Original: Our framework has consistently better performance than KD, because the student model in our framework beneﬁts not only from the output layer behavior of the teacher but also from intermediate layer representations of the teacher.
Revised: Our framework has consistently better performance than KD, because the student model in our framework beneﬁts not only from the output layer behavior of the teacher but also from intermediate layer representations of the teacher.

Original: EMNIST:
Revised: EMNIST:

Original: The ‘EMNIST Letters’ dataset consists of images of size 28 × 28 pixels showing handwritten letters.
Revised: The ‘EMNIST Letters’ dataset consists of images of size 28 × 28 pixels showing handwritten letters.

Original: It has 26 balanced classes.
Revised: It has 26 balanced classes.

Original: Each class contains lower and upper case letters.
Revised: Each class contains lower and upper case letters.

Original: The training and test sets contain 124,800 and 20,800 images respectively.
Revised: The training and test sets contain 124,800 and 20,800 images respectively.

Original: The ‘EMNIST Digits’ dataset consists of images of size 28 × 28 pixels showing handwritten digits.
Revised: The ‘EMNIST Digits’ dataset consists of images of size 28 × 28 pixels showing handwritten digits.

Original: It has 10 balanced classes.
Revised: It has 10 balanced classes.

Original: The training and test sets contain 240,000 and 40,000 images respectively.
Revised: The training and test sets contain 240,000 and 40,000 images respectively.

Original: In this case we use the MNIST model from Chen (2017) as a baseline, teacher and student model.
Revised: In this case we use the MNIST model from Chen (2017) as a baseline, teacher and student model.

Original: We trained teachers on EMNIST Digits, EMNIST Letters, and EMNIST Letters with only 13 classes.
Revised: We trained teachers on EMNIST Digits, EMNIST Letters, and EMNIST Letters with only 13 classes.

Original: Our target task is EMNIST Letters.
Revised: Our target task is EMNIST Letters.

Original: The student model is trained with different teachers and the results are compared to ﬁne-tuning, the baseline model, and the state-of-the-art results on EMNIST.
Revised: The student model is trained with different teachers and the results are compared to ﬁne-tuning, the baseline model, and the state-of-the-art results on EMNIST.

Original: The results are summarized in Table 5.
Revised: The results are summarized in Table 5.

Original: Compared to the baseline and ﬁne-tuning, student learning in our framework with expert teacher (EMNIST Letters), semi-expert teacher (Half EMNIST Letters), and non-expert teacher (EMNIST Digits) all have better performance.
Revised: Compared to the baseline and ﬁne-tuning, student learning in our framework with expert teacher (EMNIST Letters), semi-expert teacher (Half EMNIST Letters), and non-expert teacher (EMNIST Digits) all have better performance.

Original: In Fig.
Revised: In Fig.

Original: 4 we illustrate the accuracy over epochs for training of different models.
Revised: 4 we illustrate the accuracy over epochs for training of different models.

Original: STL-10:
Revised: STL-10:

Revised: The

Original: The STL-10 dataset consist of colored images of size 96 × 96 pixels.
Revised: STL- 10 dataset consist of colored images of size 96 × 96 pixels.

Original: It has 10 balanced classes.
Revised: It has 10 balanced classes The

Original: The training set contains 5,000 labeled images and 100,000 unlabeled images.
Revised: training set contains 5,000 labeled images and 100,000 unlabeled images.

Original: The test set contains 8, images.
Revised: The test set contains 8, images In

Original: In our experiment, we only use the5,000 labeled images for training.
Revised: our experiment, we only use the labeled images for training.

Original: We use the STL-10 model from Chen (2017) as our baseline, teacher and student model.
Revised: We use the STL-10 model from Chen (2017) as our baseline, teacher and student model.

Original: We trained teachers on CIFAR-10 and CIFAR-100.
Revised: We trained teachers on CIFAR-10 and CIFAR-100.

Original: We compare our results to ﬁne-tuning and the baseline in
Revised: We compare our results to ﬁne-tuning and the baseline in

Original: Table 6.
Revised: Table 6.

Original: Note that STL-10 is very similar to CIFAR-10 and CIFAR-100.
Revised: Note that STL-10 is very similar to CIFAR-10 and CIFAR-100.

Original: Therefore, both CIFAR-
Revised: Therefore, both CIFARand CIFAR-100 are very good teachers. As shown in Table 6, compared to the baseline, ﬁne-tuning a

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: and CIFAR-100 are very good teachers. As shown in Table 6, compared to the baseline, ﬁne-tuning a model using weights pretrained on CIFAR-10 and CIFAR-100 reduce test errors by more than 10% .
Revised: model using weights pretrained on CIFAR-10 and CIFAR-100 reduce test errors by more than 10% .

Original: Compared with ﬁne-tuning, student model training in our framework further reduces the test error by 3% .
Revised: Compared with ﬁne-tuning, student model training in our framework further reduces the test error by 3% .

Original: Note that we only train on the labeled data while other approaches use this data for testing of semi-supervised approaches.
Revised: Note that we only train on the labeled data while other approaches use this data for testing of semi-supervised approaches.

Original: Hence our results are obtained using fewer data and may not be directly comparable.
Revised: Hence our results are obtained using fewer data and may not be directly comparable.

Original: We still list their results in Table 6 for reference.
Revised: We still list their results in Table 6 for reference.

Original: In Fig.
Revised: In Fig.

Original: 5 we illustrate the accuracy over the epochs of training.
Revised: 5 we illustrate the accuracy over the epochs of training.

Original: 7. R EINFORCEMENT L EARNING
Revised: 7. R EINFORCEMENT L EARNING

Original: We also compare to Distral (Teh et al., 2017), which is the state-of-the-art multi-task reinforcement learning framework.
Revised: We also compare to Distral (Teh et al., 2017), which is the state-of-the-art multi-task reinforcement learning framework.

Original: We used ‘KL + ent 1 col’, which has a central model ( m 0 ), and a task model ( m i ) for each task.
Revised: We used ‘KL + ent 1 col’, which has a central model ( m 0 ), and a task model ( m i ) for each task.

Original: We perform the experiments on Atari games.
Revised: We perform the experiments on Atari games.

Original: In the experiments, we have three tasks (task 1, task 2, task 3).
Revised: In the experiments, we have three tasks (task 1, task 2, task 3).

Original: The teachers of task 2 ( m 2 ) and task 3 ( m 3 ) are provided for our framework.
Revised: The teachers of task 2 ( m 2 ) and task 3 ( m 3 ) are provided for our framework.

Original: Distral is trained for 120M steps (40M steps/task), and our model is trained for 40M steps.
Revised: Distral is trained for 120M steps (40M steps/task), and our model is trained for 40M steps.

Original: For fair comparison, we report results of Distral’s task 1 model ( m 1 ), which is better than its center model ( m 0 ).
Revised: For fair comparison, we report results of Distral’s task 1 model ( m 1 ), which is better than its center model ( m 0 ).

Original: The results are summarized in Table 7.
Revised: The results are summarized in Table 7.

Original: Distral is suboptimal, because it aims to learn a multi-task agent.
Revised: Distral is suboptimal, because it aims to learn a multi-task agent.

Original: In addition, identical action and state space is assumed.
Revised: In addition, identical action and state space is assumed.

Original: When the target task is very different from the source tasks, Distral cannot decrease the teacher inﬂuence.
Revised: When the target task is very different from the source tasks, Distral cannot decrease the teacher inﬂuence.

Original: In contrast, our framework can decrease a teacher’s inﬂuence, and thus reduce negative transfer.
Revised: In contrast, our framework can decrease a teacher’s inﬂuence, and thus reduce negative transfer.

Revised: 7. V ISUALIZATION OF N ORMALIZED W EIGHTS OF T EACHERS AND S TUDENT

Revised: Following the reviewer’s suggestion, we plot the averaged normalized weight ( p w ) for teachers and the student in the C10 experiment, where C100 and SVHN experts are teachers.

Revised: Intuitively, the C teacher should have a higher p w value than the SVHN teacher, because C100 is more relevant to C10.

Revised: The plot veriﬁes this intuition.

Original: [Figure]
Revised: As shown in Fig. 7, p w of the C100 teacher is higher than that of the SVHN teacher over the entire training. Note, both teachers’ normalized weights approach zero at the end of training.

Revised: 7. A BLATION S TUDIES 7.4. U NTRAINED T EACHER M ODELS

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: To verify that the student really beneﬁts from the knowledge of teachers, we conduct an ablation study suggested by a reviewer.

Revised: We use teacher models that haven’t been trained at all.

Revised: Intuitively, learning with untrained teachers should have worse performance than learning with knowledgeable teachers.

Revised: Our experiments verify this intuition.

Revised: In Fig.

Revised: 8 (a), where the target task is hero, learning with untrained teachers (‘w/ untrained teachers’) achieves an average reward of 15934.

Revised: Learning with knowledgeable teachers (‘Ours with seaquest and riverraid teacher’) achieves an average reward of 30928.

Revised: More results are presented in Figs.

Revised: 8 (b, c).

Revised: The results show that Knowledge Flow achieves higher rewards than training with untrained teachers in different environments and teacher-student settings.

Revised: 7.4. T RAINING W ITHOUT KL T ERM

Revised: The KL term prevents the student’s output distribution over actions or labels from drastic changes when the teachers’ inﬂuence is decreasing.

Revised: To investigate the importance of the KL term, we conduct an ablation study where the KL coefﬁcient ( λ 2 ) is set to zero.

Revised: The result is summarized in Fig.

Revised: Considering Fig.

Revised: 9 (a), where the target task is MsPacman and the teachers are Hero and Seaquest

Original: [Table]
Revised: [Table]

Revised: experts.

Revised: Without the KL term, when a teacher’s inﬂuence decreases, the rewards drop drastically.

Revised: In contrast, with a KL term, we don’t observe performance drops.

Revised: At the end of training, learning with the KL term achieves an average reward of 2907 and learning without the KL term achieves an average reward of 1215.

Revised: More results are presented in Fig.

Revised: 9 (b, c), which shows that training with the KL term achieves higher reward than training without the KL term.

Revised: 7. T EACHERS WITH D IFFERENT A RCHITECTURE THAN S TUDENT

Revised: In additional experiments, following the suggestion of a reviewer, we use architectures for the teacher which differ from the student model.

Revised: More speciﬁcally, we use the model of Mnih et al.

Revised: (2015) as a teacher model.

Revised: The teacher model consists of 3 convolutional layers, which have 32, 64, andﬁlters, followed by a hidden fully connected layer which has 512 ReLUs.

Revised: We use the model of Mnih et al.

Revised: as the student model.

Revised: The student model consists of 2 convolutional layers, which have 16 and 32 ﬁlters respectively, followed by a hidden fully connected layer which has 256 ReLUs.

Revised: Both models’ fully connected layers are followed by two output layers for actions and values.

Revised: In the experiments, we link each teacher’s ﬁrst convolutional layer to the student’s ﬁrst convolutional layer.

Revised: Moreover, we link each teacher’s third convolutional layer to the student’s second convolutional layer, and each teacher’s fully connected layer to the student’s fully connected layer.

Revised: In the experiment, the target task is KungFu Master, and the teachers are experts for Seaquest and Hero.

Revised: The results are summarized in Fig.

Revised: We observed that learning with teachers, whose architecture differs from the student, to have similar performance as learning with teachers which have the same architecture.

Revised: Consider as an example Fig.

Revised: 10 (a), where the target task is KungFu Master, and the teachers are experts for Seaquest and Hero.

Revised: At the end of training, learning with teachers of different architectures achieves an average reward of 37520, and learning with teachers of the same architecture achieves an average reward of 35012.

Revised: More results are shown in Fig.

Revised: 10 (b, c).

Revised: The results show that Knowledge

Revised: Flow can enable higher rewards, even if the teachers and the student architectures differ.

Revised: 7. A VERAGE N ETWORK AS θ old

Revised: For the parameters θ old an average network can be used.

Revised: To investigate how usage of an average network to obtain the parameters θ old affects the performance, we conduct an experiment where θ old is computed using the exponential running average of the model weight.

Revised: More speciﬁcally, θ old is updated as follows:

Original: [Table]
Revised: θ old ← α · θ old + (1 − α ) · θ , where α = 0 . The results are summarized in Fig. We observe that using an exponential average to compute θ old results in very similar performance as using a single model. Consider Fig.

Revised: 11 (a), where the target task is Boxing and the teacher is a Riverraid expert.

Revised: At the end of training, using an average network to obtain θ old achieves an average reward of 96.2 and using a single network to obtain θ old achieves an average reward of 96.0.

Revised: More results on using an average network are shown in Fig.

Revised: 11 (b, c).

Original: [Table]
Revised: [Figure] [Figure] [Figure] [Figure]