Original: DELTA : DE EP L EARNING T RANSFER USING F EATURE M AP WITH A TTENTION FOR C ONVOLUTIONAL N ETWORKS
Revised: DELTA : DE EP L EARNING T RANSFER USING F EATURE M AP WITH A TTENTION FOR C ONVOLUTIONAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transfer learning through ﬁne-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can signiﬁcantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task.
Revised: Transfer learning through ﬁne-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can signiﬁcantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task.

Original: To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied.
Revised: To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied.

Original: In this paper, we propose a novel regularized transfer learning framework DELTA , namely DEep Learning Transfer using Feature Map with Attention .
Revised: In this paper, we propose a novel regularized transfer learning framework DELTA , namely DEep Learning Transfer using Feature Map with Attention .

Original: Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network.
Revised: Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network.

Original: Speciﬁcally, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner.
Revised: Speciﬁcally, in addition to minimizing the empirical loss, DELTA aligns the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner.

Original: We evaluate DELTA with the state-of-the-art algorithms, including L 2 and
Revised: We evaluate DELTA with the state-of-the-art algorithms, including L 2 and

Original: L 2 - SP .
Revised: L 2 - SP .

Original: The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.
Revised: The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In many real-world applications, deep learning practitioners often have limited number of training instances.
Revised: In many real-world applications, deep learning practitioners often have limited number of training instances.

Original: Direct training a deep neural network with a small training data set usually results in the so-called over-ﬁtting problem and the quality of the obtained model is low.
Revised: Direct training a deep neural network with a small training data set usually results in the so-called over-ﬁtting problem and the quality of the obtained model is low.

Original: A simple yet effective approach to obtain high-quality deep learning models is to perform weight ﬁne-tuning.
Revised: A simple yet effective approach to obtain high-quality deep learning models is to perform weight ﬁne-tuning.

Original: In such practices, a deep neural network is ﬁrst trained using a large (and possibily irrelevant) source dataset (e.g. ImageNet).
Revised: In such practices, a deep neural network is ﬁrst trained using a large (and possibly irrelevant) source dataset (e.g. ImageNet).

Original: The weights of such a network are then ﬁne-tuned using the data from the target application domain.
Revised: The weights of such a network are then ﬁne-tuned using the data from the target application domain.

Original: Fine-tuning is a speciﬁc approach to perform transfer learning in deep learning.
Revised: Fine-tuning is a speciﬁc approach to perform transfer learning in deep learning.

Original: The weights pretrained by the source dataset with sufﬁcient instances usually provide a better initialization for the target task than random initializations.
Revised: The weights pretrained by the source dataset with a sufﬁciently large number of instances usually provide a better initialization for the target task than random initializations.

Original: Standard regularization methods such as the L 1 and Lnormalizations, constraining the weights around the origin, are used to prevent the weights  from being too large during ﬁne-tuning.
Revised: In a typical ﬁne-tuning approach, weights in lower convolution layers are ﬁxed and weights in upper layers are re-trained using data from the target domain.

Original: But this is not adequate to have good generalization performance.

Original: Parameters of the target model may be driven far away from initial values, which also causes overﬁtting in transfer learning scenarios.
Revised: In this approach parameters of the target model may be driven far away from initial values, which also causes over-ﬁtting in transfer learning scenarios.

Original: Approaches called regularization using the starting point as the reference (SPAR), were  proposed to solve the above problem.
Revised: Approaches called regularization using the starting point as the reference (SPAR), were recently proposed to solve the over-ﬁtting problem.

Original: For example, Li et al.
Revised: For example, Li et al.

Original: (Li et al., 2018) recently proposed L 2 - SP that incorporates the Euclid distance between the target weights and the starting point (i.e., weights of source network) as part of loss.
Revised: (Li et al., 2018) recently proposed L 2 - SP that incorporates the Euclid distance between the target weights and the starting point (i.e., weights of source network) as part of loss.

Original: Minimizing this loss function, L 2 - SP aims to minimize the empirical loss of deep learning while reducing the distance of weights between source/target networks.
Revised: Minimizing this loss function, L 2 - SP aims to minimize the empirical loss of deep learning while reducing the distance of weights between source and target networks.

Original: They achieved signiﬁcant improvement compared with standard practice of using weight decay ( Lnormalization).
Revised: They achieved signiﬁcant improvement compared with standard practice of using weight decay ( L 2 normalization).

Original: However such regularization method may not deliver optimal solution for transfer learning.
Revised: However such regularization method may not deliver optimal solution for transfer learning.

Original: On one side, if the regularization is not strong, even with ﬁne-turning, the weights may  be driven far away from the initial position, leading to the lose of useful knowledge, i.e. catastrophic memory loss.
Revised: On one side, if the regularization is not strong, even with ﬁne-turning, the weights may still be driven far away from the initial position, leading to the lose of useful knowledge, i.e. catastrophic memory loss.

Original: On the other side, if the regularization is too strong, newly obtained model is constrained to a local neighborhood of the original model, which may be suboptimal to the target data set.
Revised: On the other side, if the regularization is too strong, newly obtained model is constrained to a local neighborhood of the original model, which may be suboptimal to the target data set.

Original: Although aforementioned methods demonstrated the power of regularization in deep transfer learning, we argue that we need to perform research on at least the following two aspects in order to further improve current regularization methods.
Revised: Although aforementioned methods demonstrated the power of regularization in deep transfer learning, we argue that we need to perform research on at least the following two aspects in order to further improve current regularization methods.

Original: Behavior vs. Mechanisms.
Revised: Behavior vs. Mechanisms.

Original: The practice of weight regularization for CNN is motivated by a simple intuition — the network (layers) with similar weights should produce similar outputs.
Revised: The practice of weight regularization for CNN is motivated by a simple intuition — the network (layers) with similar weights should produce similar outputs.

Original: However, due to the complex structures of deep neural network with strong redundancies, regulating the model parameters directly seems an over-killing of the problem.
Revised: However, due to the complex structures of deep neural network with strong redundancies, regulating the model parameters directly seems an over-killing of the problem.

Original: We argue that we should regularize the “Behavior”, or in our case, the outcome of each layer, rather than model parameters.
Revised: We argue that we should regularize the “Behavior”, or in our case, the outer layer outputs (e.g. the feature maps) produced by each layer, rather than model parameters.

Original: With constrained outer layer outputs, the generalization capacity could be improved through aligning the behaviors of the outer layers of  target network to the source one, which has been pre-trained using an extremely large dataset.
Revised: With constrained feature maps, the generalization capacity could be improved through aligning the behaviors of the outer layers of the target network to the source one, which has been pre-trained using an extremely large dataset.

Revised: In Convolutional Neural Networks, which we focus on exclusively in this paper, an outer layer is a convolution layer and the output of an outer layer is its feature map.

Original: Syntax vs Semantics.
Revised: Syntax vs Semantics.

Original: While regularizing the outer layer outputs might improve the transfer of generalization capacity, it is still difﬁcult to design such regularizers.
Revised: While regularizing the feature maps might improve the transfer of generalization capacity, it is still difﬁcult to design such regularizers.

Original: It is challenging to measure the similarity/distance between the outer layer outputs of source/target networks without understanding its semantics or representations.
Revised: It is challenging to measure the similarity/distance between the feature maps without understanding its semantics or representations.

Original: For example for image classiﬁcaiton, some of the convolution kernesls may be corresponding to features that are shared between the two learning tasks and hence should be preserved in transfer learning while others are speciﬁc to the source task and hence could be eliminated in transfer learning.
Revised: For example for image classiﬁcation, some of the convolution kernels may be corresponding to features that are shared between the two learning tasks and hence should be preserved in transfer learning while others are speciﬁc to the source task and hence could be eliminated in transfer learning.

Original: In this paper, we propose a novel regularization approach DELTA to address the two issues.
Revised: In this paper, we propose a novel regularization approach DELTA to address the two issues.

Original: Specifically, DELTAmodels the semantics of outer layer outputs using feature maps, then selects the discriminative features  through re-weighting the feature maps with a novel supervised attention mechanisms.
Revised: Specifically, DELTA selects the discriminative features from outer layer outputs through re-weighting the feature maps with a novel supervised attention mechanism.

Original: Through paying attention to discriminative parts of feature maps, DELTA characterizes the distance between source/target networks using their outer layer outputs, and incorporates such distance as the regularization term of the loss function.
Revised: Through paying attention to discriminative parts of feature maps, DELTA characterizes the distance between source/target networks using their outer layer outputs, and incorporates such distance as the regularization term of the loss function.

Original: With the back-propagation, such regularization ﬁnally affects the optimization for weights of deep neural network and awards the target network generalization capacity inherited from the source network.
Revised: With the back-propagation, such regularization ﬁnally affects the optimization for weights of deep neural network and awards the target network generalization capacity inherited from the source network.

Original: In summary,we name our key innovation of this work as ” unactivated channel re-usage ”.
Revised: In summary, our key insight is what we call ” unactivated channel re-usage ”.

Original: Specifically we ﬁnd an approach to identify those transferable channels and preserve them through regularization and identify those untransferable channels and reuse them, using an attention mechanism with feature map regularization.
Revised: Speciﬁcally our approach identiﬁes those transferable channels and preserves such ﬁlters through regularization and identify those untransferable channels and reuse them, using an attention mechanism with feature map regularization.

Original: We conduct extensive experiments using a wide range of source/target datasets and compare DELTA to the existing deep transfer learning algorithms that are in pursuit of weight similarity.
Revised: We have conducted extensive experiments using a wide range of source/target datasets and compared DELTA to the existing deep transfer learning algorithms that are in pursuit of weight similarity.

Original: The experiment results show that DELTA signiﬁcantly outperformed the state-of-the-art regularization algorithms including L 2 and L 2 - SP with higher accuracy on a wide group of image classiﬁcation data sets.
Revised: The experiment results show that DELTA signiﬁcantly outperformed the state-of-the-art regularization algorithms including L 2 and L 2 - SP with higher accuracy on a wide group of image classiﬁcation data sets.

Original: The rest of the paper is organized as follows: in Section 2 related works are summarized, in Section 3 our feature map based regularization method is introduced, in Section 4 experimental results are presented and discussed, and ﬁnally in Section 5 the paper is concluded.
Revised: The rest of the paper is organized as follows: in Section 2 related works are summarized, in Section 3 our feature map based regularization method is introduced, in Section 4 experimental results are presented and discussed, and ﬁnally in Section 5 the paper is concluded.

Original:  R ELATED W ORK AND B ACKGROUNDS
Revised:  R ELATED W ORK AND B ACKGROUNDS

Original: In this section, we ﬁrst review the preliminarily works to this paper, where we discuss the contributions made by this work beyond previous studies.
Revised: In this section, we ﬁrst review the related works to this paper, where we discuss the contributions made by this work beyond previous studies.

Original: Then, we present the backgrounds of our work.
Revised: Then, we present the backgrounds of our work.

Original: 2. R ELATED W ORK
Revised: 2. R ELATED W ORK

Original: Transfer learning is a type of machine learning paradigm aiming at transferring the knowledge obtained in a source task to a target task (Caruana, 1997; Pan et al., 2010).
Revised: Transfer learning is a type of machine learning paradigm aiming at transferring the knowledge obtained in a source task to a target task (Caruana, 1997; Pan et al., 2010).

Original: Our work primarily focuses on inductive transfer learning for deep neural networks, where the label space of the target task differs from that of the source task.
Revised: Our work primarily focuses on inductive transfer learning for deep neural networks, where the label space of the target task differs from that of the source task.

Original: For example, Donahue et al (Donahue et al., 2014) proposed to train a classiﬁer based on feature extracted from a pre-trained CNN, where a large mount of param- eters, such as ﬁlters, of the source network are reused directly in the target one.
Revised: For example, Donahue et al (Donahue et al., 2014) proposed to train a classiﬁer based on feature extracted from a pre-trained CNN, where a large mount of parameters, such as ﬁlters, of the source network are reused directly in the target one.

Original: This method may overload the target network with tons of unappropriated features (without discrimination power) involved, while the key features of the target task might be ignored.
Revised: This method may overload the target network with tons of irrelevant features (without discrimination power) involved, while the key features of the target task might be ignored.

Original: To understand whether a feature can be transferred to the target network, Yosinki et al.
Revised: To understand whether a feature can be transferred to the target network, Yosinki et al.

Original: (Yosinski et al., 2014) quantiﬁed the transferability of features from each layer considering the performance gain.
Revised: (Yosinski et al., 2014) quantiﬁed the transferability of features from each layer considering the performance gain.

Original: Moreover, to understand the factors that may affect deep transfer learning performance, Huh et al.
Revised: Moreover, to understand the factors that may affect deep transfer learning performance, Huh et al.

Original: (Huh et al., 2016) empirically analyzed the features obtained by the ImageNet pre-trained source network to a wide range of computer vision tasks.
Revised: (Huh et al., 2016) empirically analyzed the features obtained by the ImageNet pre-trained source network to a wide range of computer vision tasks.

Original: Recently, moreand more relevant studies to improve the inductive transfer learning from different aspects have been done, such as  subset selection (Ge & Yu, 2017; Cui et al., 2018), sparse transfer (Liu et al., 2017), ﬁlter distribution constraining (Aygun et al., 2017),  parameter transfer (Zhang et al., 2018).
Revised: Recently, more studies to improve the inductive transfer learning from a diverse set of angles have been proposed, such as ﬁlter subset selection (Ge & Yu, 2017; Cui et al., 2018), sparse transfer (Liu et al., 2017), ﬁlter distribution constraining (Aygun et al., 2017), and parameter transfer (Zhang et al., 2018).

Original: In terms of deep transfer learning problems, the most relevant work to our study is (Li et al., 2018), where authors investigated regularization schemes to accelerate deep transfer learning while preventing ﬁne-tuning from over-ﬁtting.
Revised: For deep transfer learning problems, the most relevant work to our study is (Li et al., 2018), where authors investigated regularization schemes to accelerate deep transfer learning while preventing ﬁne-tuning from over-ﬁtting.

Original: Their work showed that a simple L 2 -norm regularization on top of the “Starting Point as a Reference” optimization can signiﬁcantly outperform a wide range of regularization-based deep transfer learning mechanisms, such as the standard L 2 -norm regularization.
Revised: Their work showed that a simple L 2 -norm regularization on top of the “Starting Point as a Reference” optimization can signiﬁcantly outperform a wide range of regularization-based deep transfer learning mechanisms, such as the standard L 2 -norm regularization.

Original: Compared to above work, the key contributions made in this paper include 1) rather than regularizing the distance between the parameters of source network and target network, DELTA intends to constrain the L 2 -norm of the difference between their behaviors (i.e., the feature maps of outer layer outputs in the source/target networks); and 2) the regularization term used in DELTA incorporates a supervised attention mechanism, which re-weights regularizers according to their performance gain/loss.
Revised: Compared to above work, the key contributions made in this paper include 1) rather than regularizing the distance between the parameters of source network and target network, DELTA constrains the L 2 -norm of the difference between their behaviors (i.e., the feature maps of outer layer outputs in the source/target networks); and 2) the regularization term used in DELTA incorporates a supervised attention mechanism, which re-weights regularizers according to their performance gain/loss.

Original: In terms of methodologies, our work is also related to the knowledge distillation for model compression (Hinton et al., 2015; Romero et al., 2014).
Revised: In terms of methodologies, our work is also related to the knowledge distillation for model compression (Hinton et al., 2015; Romero et al., 2014).

Original: Generally, knowledge distillation focuses on teacher-student network training, where the teacher and student networks are usually based on the same task (Hinton et al., 2015).
Revised: Generally, knowledge distillation focuses on teacher-student network training, where the teacher and student networks are usually based on the same task (Hinton et al., 2015).

Original: These work frequently intends to transfer the knowledge in the teacher network to the student one through aligning their outputs of some layers (Romero et al., 2014).
Revised: These work frequently intends to transfer the knowledge in the teacher network to the student one through aligning their outputs of some layers (Romero et al., 2014).

Original: The most close works to this paper are (Zagoruyko & Komodakis, 2016;
Revised: The most close works to this paper are (Zagoruyko & Komodakis, 2016;

Original: Yim et al., 2017), where knowledge distillation technique has been studied to improve transfer learning.
Revised: Yim et al., 2017), where knowledge distillation technique has been studied to improve transfer learning.

Original: Compared to above work, our work, including other transfer learning studies, intends to transfer knowledge between different source/target tasks (i.e., source and target tasks), though the source/target networks can be viewed as teachers and students respectively.
Revised: Compared to above work, our work, including other transfer learning studies, intends to transfer knowledge between different source/target tasks (i.e., source and target tasks), though the source/target networks can be viewed as teachers and students respectively.

Original: We follow the conceptual ideas of knowledge distillation to regularize the outer layer outputs of the network (i.e., feature maps), yet further extend such regularization to a supervised transfer learning mechanism by incorporating the labels of target task (which is different from the source task/network).
Revised: We follow the conceptual ideas of knowledge distillation to regularize the outer layer outputs of the network (i.e., feature maps), yet further extend such regularization to a supervised transfer learning mechanism by incorporating the labels of target task (which is different from the source task/network).

Original: Moreover, a supervised attention mechanism has been adopted to regularize the feature maps according to the importance of ﬁlters.
Revised: Moreover, a supervised attention mechanism has been adopted to regularize the feature maps according to the importance of ﬁlters.

Original: Other works relevant to our methodology include: continual learning (Kirkpatrick et al., 2017; Li &
Revised: Other works relevant to our methodology include: continual learning (Kirkpatrick et al., 2017; Li &

Original: Hoiem, 2017), attention mechanism for CNN models (Mnih et al., 2014; Xu et al., 2015; Yang et al., 2016; Zagoruyko & Komodakis, 2016) and so on.
Revised: Hoiem, 2017), attention mechanism for CNN models (Mnih et al., 2014; Xu et al., 2015; Yang et al., 2016; Zagoruyko & Komodakis, 2016), among others.

Original: We appreciate the contributions made by these studies.

Original: 2. B ACKGROUNDS
Revised: 2. B ACKGROUNDS

Original: Deep convolutional networks usually consist of a great number of parameters that need ﬁt to the dataset.
Revised: Deep convolutional networks usually consist of a great number of parameters that need ﬁt to the dataset.

Original: For example, ResNet-110 has more than one million free parameters.
Revised: For example, ResNet-110 has more than one million free parameters.

Original: The size of free parameters causes the risk of over-ﬁtting.
Revised: The size of free parameters causes the risk of over-ﬁtting.

Original: Regularization is the technique to reduce this risk by constraining the parameters within a limited space.
Revised: Regularization is the technique to reduce this risk by constraining the parameters within a limited space.

Original: The general regularization problem is usually formulated as follow.
Revised: The general regularization problem is usually formulated as follow.

Original: 2.2. G ENERAL R EGULARIZATION
Revised: 2.2. G ENERAL R EGULARIZATION

Original: Let’s denote the dataset for the desired task as { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ( x 3 , y 3 ) . . .
Revised: Let’s denote the dataset for the desired task as { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ( x 3 , y 3 ) . . .

Original: , ( x n , y n ) } , where totally n tuples are offered and each tuple ( x i , y i ) refers to the input image and its label in the dataset.
Revised: , ( x n , y n ) } , where totally n tuples are offered and each tuple ( x i , y i ) refers to the input image and its label in the dataset.

Original: We further denote ω ∈ R
Revised: We further denote ω ∈ R

Original: d be the d -dimensional parameter vector containing all d parameters of the target model.
Revised: d be the d -dimensional parameter vector containing all d parameters of the target model.

Original: The optimization object with regularization is to obtain
Revised: The optimization object with regularization is to obtain

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term (cid:80) ni =1 L ( z ( x i , ω ) , y i ) refers to the empirical loss of data ﬁtting while the second term is a general form of regularization.
Revised: where the ﬁrst term (cid:80) ni =1 L ( z ( x i , ω ) , y i ) refers to the empirical loss of data ﬁtting while the second term is a general form of regularization.

Original: The tuning parameter λ > 0 balances the trade-off between the empirical loss and the regularization loss.
Revised: The tuning parameter λ > 0 balances the trade-off between the empirical loss and the regularization loss.

Original: Without any explicit information (such as other datasets) given, one can easily use the L 0 /L 1 /L 2 -norm of the parameter vector ω as the regularization to ﬁx the consistency issue of the network.
Revised: Without any explicit information (such as other datasets) given, one can easily use the L 0 /L 1 /L 2 -norm of the parameter vector ω as the regularization to ﬁx the consistency issue of the network.

Original: 2.2. R EGULARIZATION FOR T RANSFER L EARNING
Revised: 2.2. R EGULARIZATION FOR T RANSFER L EARNING

Original: Given a pre-trained network with parameter ω ∗ based on an extremely large dataset as the source, one can estimate the parameter of target network through the transfer learning paradigms.
Revised: Given a pre-trained network with parameter ω ∗ based on an extremely large dataset as the source, one can estimate the parameter of target network through the transfer learning paradigms.

Original: Using the ω ∗ as the initialization to solve the problem in Eq 1 can accelerate the training of target network through knowledge transfer (Hinton et al., 2006; Bengio et al., 2007).
Revised: Using the ω ∗ as the initialization to solve the problem in Eq 1 can accelerate the training of target network through knowledge transfer (Hinton et al., 2006; Bengio et al., 2007).

Original: However, the accuracy of the target network would be bottlenecked in such settings.
Revised: However, the accuracy of the target network would be bottlenecked in such settings.

Original: To further improve the transfer learning, novel regularized transfer learning paradigms that constrain the divergence between target and source networks has been proposed, such that
Revised: To further improve the transfer learning, novel regularized transfer learning paradigms that constrain the divergence between target and source networks has been proposed, such that

Original: [Equation]
Revised: [Equation]

Original: where the regularization term Ω( ω, ω ∗ ) characterizes the differences between the parameters of target and source network.
Revised: where the regularization term Ω( ω, ω ∗ ) characterizes the differences between the parameters of target and source network.

Original: As ω ∗ is frequently used as the initialization of ω during the optimization procedure, this method sometimes refers to Starting Point As the Reference (SPAR) method.
Revised: As ω ∗ is frequently used as the initialization of ω during the optimization procedure, this method sometimes refers to Starting Point As the Reference (SPAR) method.

Original: To regularize weights straightforwardly, one can easily use the geometric distance between ω and ω ∗ as the regularization terms.
Revised: To regularize weights straightforwardly, one can easily use the geometric distance between ω and ω ∗ as the regularization terms.

Original: For example, L 2 - SP algorithm constrains the Euclid distance of the weights of convolution ﬁlters between the source/target networks (Li et al., 2018).
Revised: For example, L 2 - SP algorithm constrains the Euclid distance of the weights of convolution ﬁlters between the source/target networks (Li et al., 2018).

Original: In this way, wecan summarize the existing deep transfer learning approaches as the solution of the regularized learning problem listed in Eq 2, where the regularizer aims at constraining the divergence of parameters of the two networks while ignoring the behavior of the networks with the training data set { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
Revised: In this way, we summarize the existing deep transfer learning approaches as the solution of the regularized learning problem listed in Eq 2, where the regularizer aims at constraining the divergence of parameters of the two networks while ignoring the behavior of the networks with the training data set { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .

Original: , ( x n , y n ) } .
Revised: , ( x n , y n ) } .

Original: More speciﬁc, the regularization terms used by the existing deep transfer learning approaches neither consider how the network with certain parameters would behave with the new datums (images) or leverages the supervision information from the labeled datums (images) to improve the transfer performance.
Revised: More speciﬁc, the regularization terms used by the existing deep transfer learning approaches neither consider how the network with certain parameters would behave with the new data (images) or leverages the supervision information from the labeled data (images) to improve the transfer performance.

Original:  L EARNING F RAMEWORK AND A LGORITHMS
Revised:  L EARNING F RAMEWORK AND A LGORITHMS

Original: In this section, we ﬁrst formulate the problem, then present the overall design of proposed solution and introduce several key algorithms.
Revised: In this section, we ﬁrst formulate the problem, then present the overall design of proposed solution and introduce several key algorithms.

Original: 3. O VERALL F RAMEWORK
Revised: 3. O VERALL F RAMEWORK

Original: In our research, instead of bounding the difference of weights, we intend to regulate the network behaviors and force some layers of the target network to behave similarly to the source ones.
Revised: In our research, instead of bounding the difference of weights, we intend to regulate the network behaviors and force some layers of the target network to behave similarly to the source ones.

Original: Specifically, we deﬁne the “behaviors” of a layer as the parts of it output, which are with semantics-rich and discriminative informations.
Revised: Specifically, we deﬁne the “behaviors” of a layer as its output, which are with semantics-rich and discriminative information.

Original: DELTA intends to incorporate a new regularizer Ω (cid:48) ( ω, ω ∗ , x ) .
Revised: DELTA intends to incorporate a new regularizer Ω (cid:48) ( ω, ω ∗ , x ) .

Original: Given a pre-trained parameter ω ∗ and any input image x , the regularizer Ω (cid:48) ( ω, ω ∗ , x ) measures the distance between the behaviors of target network with parameter ω and the source one based on ω ∗ .
Revised: Given a pre-trained parameter ω ∗ and any input image x , the regularizer Ω (cid:48) ( ω, ω ∗ , x ) measures the distance between the behaviors of target network with parameter ω and the source one based on ω ∗ .

Original: With such regularizer, the transfer learning problem can be reduced to learning problem as follows:
Revised: With such regularizer, the transfer learning problem can be reduced to learning problem as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) ni =1 Ω( ω, ω ∗ , x i , y i , z ) characterizes the aggregated difference between the source and target network over the whole training dataset using the model z .
Revised: where (cid:80) ni =1 Ω( ω, ω ∗ , x i , y i , z ) characterizes the aggregated difference between the source and target network over the whole training dataset using the model z .

Original: Note that, with the input tuples ( x i , y i ) and for 1 ≤ i ≤ n , the proposed regularizer Ω( ω, ω ∗ , x i , y i , z ) is capable of regularizing the behavioral differences of network model z 
Revised: Note that, with the input tuples ( x i , y i ) and for 1 ≤ i ≤ n , the proposed regularizer Ω( ω, ω ∗ , x i , y i , z ) is capable of regularizing the behavioral differences of network model z based on each labeled sample ( x i , y i ) in the dataset, using the parameters ω and ω ∗ respectively.

Original: Further, inspired by SPAR method, DELTA accelerates the optimization procedure of the regularizer through incorporating a parameter-based proximal term, such that
Revised: Further, inspired by SPAR method, DELTA accelerates the optimization procedure of the regularizer through incorporating a parameter-based proximal term, such that

Original: [Equation]
Revised: [Equation]

Original: where α, β are two non-negative tuning parameters to balance two terms.
Revised: where α, β are two non-negative tuning parameters to balance two terms.

Original: On top of the behavioral regularizer Ω (cid:48) ( ω, ω ∗ , x , y, z ) , DELTA includes a term Ω (cid:48)(cid:48)
Revised: On top of the behavioral regularizer Ω (cid:48) ( ω, ω ∗ , x , y, z ) , DELTA includes a term Ω (cid:48)(cid:48)

Original: ( ω \ ω ∗ ) regularizing a subset of parameters that are privately owned by the target network w only but not exist in the source network w ∗ .
Revised: ( ω \ ω ∗ ) regularizing a subset of parameters that are privately owned by the target network w only but not exist in the source network w ∗ .

Original: Speciﬁcally, Ω (cid:48)(cid:48) ( ω \ ω ∗ ) constrains the L 2 -norm of the private parameters in ω , so as to improve the consistency of inner layer parameters estimation.
Revised: Speciﬁcally, Ω (cid:48)(cid:48) ( ω \ ω ∗ ) constrains the L 2 -norm of the private parameters in ω , so as to improve the consistency of inner layer parameters estimation.

Original: Note that, when using w ∗ as the initialization of ω for optimization, DELTA indeed adopts starting point as reference (SPAR) strategy (Li et al., 2018) to accelerate the optimization and gains better generalizability.
Revised: Note that, when using w ∗ as the initialization of ω for optimization, DELTA indeed adopts starting point as reference (SPAR) strategy (Li et al., 2018) to accelerate the optimization and gains better generalizability.

Original: 3. B EHAVIORAL R EGULARIZATION
Revised: 3. B EHAVIORAL R EGULARIZATION

Original: To regularize the behavior of the networks, DELTA considers the distance between the outer layer outputs of the two networks.
Revised: To regularize the behavior of the networks, DELTA considers the distance between the outer layer outputs of the two networks.

Original: Figure 1 illustrates the concepts of proposed method.
Revised: Figure 1 illustrates the concepts of proposed method.

Original: Speciﬁcally, the outer layer of the network consists of a large set of convolutional ﬁlters.
Revised: Speciﬁcally, the outer layer of the network consists of a large set of convolutional ﬁlters.

Original: Given an input x i (for ∀ 1 ≤ i ≤ n in training set), each ﬁlter can generate a feature map.
Revised: Given an input x i (for ∀ 1 ≤ i ≤ n in training set), each ﬁlter generates a feature map.

Original: Thus, DELTA characterizes the outer layer output of the network model z based on input x i and parameter ω using a set of feature maps, such as FM j ( z, ω, x i ) and 1 ≤ j ≤ N for the N ﬁlters in networks.
Revised: Thus, DELTA characterizes the outer layer output of the network model z based on input x i and parameter ω using a set of feature maps, such as FM j ( z, ω, x i ) and 1 ≤ j ≤ N for the N ﬁlters in networks.

Original: In this way, the behavioral regularizer is deﬁned as:
Revised: In this way, the behavioral regularizer is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where W j ( z, ω ∗ , x i , y i ) refers to the weight assigned to the j th ﬁlter and the i th image (for ∀ 1 ≤ i ≤ n and ∀ 1 ≤ j ≤ N ) and the behavioral difference between the two feature maps, i.e., FM j ( z, ω, x i ) and FM j ( z, ω ∗ , x i ) , is measured using their Euclid distance (denoted as (cid:107) · (cid:107) 2 ).
Revised: where W j ( z, ω ∗ , x i , y i ) refers to the weight assigned to the j th ﬁlter and the i th image (for ∀ 1 ≤ i ≤ n and ∀ 1 ≤ j ≤ N ) and the behavioral difference between the two feature maps, i.e., FM j ( z, ω, x i ) and FM j ( z, ω ∗ , x i ) , is measured using their Euclid distance (denoted as (cid:107) · (cid:107) 2 ).

Original: In following sections, we are going to present (1) the design and implementation of feature map extraction FM j ( z, ω, x ) for 1 ≤ j ≤ N , as well as (2) the the attention model that assigns the weight W j ( z, ω ∗ , x i , y i ) to each labeled image and ﬁlter.
Revised: In following sections, we are going to present (1) the design and implementation of feature map extraction FM j ( z, ω, x ) for 1 ≤ j ≤ N , as well as (2) the the attention model that assigns the weight W j ( z, ω ∗ , x i , y i ) to each labeled image and ﬁlter.

Original: 3. F EATURE M AP E XTRACTION FROM O UTER L AYERS
Revised: 3. F EATURE M AP E XTRACTION FROM C ONVOLUTION L AYERS

Original: Given each ﬁlter of the network with parameter ω and the input x i drawn from the target dataset,
Revised: Given each ﬁlter of the network with parameter ω and the input x i drawn from the target dataset,

Original: DELTA ﬁrst uses such ﬁlter to get the corresponding output based on x , then adopts Rectiﬁed Linear
Revised: DELTA ﬁrst uses such ﬁlter to get the corresponding output based on x , then adopts Rectiﬁed Linear

Original: Units (ReLU) to rectify the output as a matrix.
Revised: Units (ReLU) to rectify the output as a matrix.

Original: Further, DELTA formats the output matrices into vectors through concatenation.
Revised: Further, DELTA formats the output matrices into vectors through concatenation.

Original: In this way, DELTA obtains FM j ( z, ω, x i ) for 1 ≤ j ≤ N and 1 ≤ i ≤ n that have been used in Eq 5.
Revised: In this way, DELTA obtains FM j ( z, ω, x i ) for 1 ≤ j ≤ N and 1 ≤ i ≤ n that have been used in Eq 5.

Original: 3. W EIGHTING F EATURE M APS WITH S UPERVISED A TTENTION M ODELS
Revised: 3. W EIGHTING F EATURE M APS WITH S UPERVISED A TTENTION M ODELS

Original: In DELTA , the proposed regularizer measures the distance between the feature maps generated by the two networks, then aggregates the distances using non-negative weights.
Revised: In DELTA , the proposed regularizer measures the distance between the feature maps generated by the two networks, then aggregates the distances using non-negative weights.

Original: Our aim is to pay more attention to those features with greater capacity of discrimination through supervised learning.
Revised: Our aim is to pay more attention to those features with greater capacity of discrimination through supervised learning.

Original: To obtain such weights for feature maps, we propose a supervised attention method derived from the backward variable selection, where the weights of features are characterized by the potential performance loss when removing these features from the network.
Revised: To obtain such weights for feature maps, we propose a supervised attention method derived from the backward variable selection, where the weights of features are characterized by the potential performance loss when removing these features from the network.

Original: For clear description, according to common understandings, we ﬁrst deﬁne a convolution ﬁlter as follow.
Revised: For clear description, following common conventions, we ﬁrst deﬁne a convolution ﬁlter as follow.

Original: The parameter of a conv2d layer is a four-dimensional tensor with the shape of ( c i +1 , c
Revised: The parameter of a conv2d layer is a four-dimensional tensor with the shape of ( c i +1 , c

Original: i , k h , k w ) , where c i and c i +1 represent for the number of channels of the
Revised: i , k h , k w ) , where c i and c i +1 represent for the number of channels of the

Original: i th
Revised: i th

Original: and ( i + 1) th layer respectively.
Revised: and ( i + 1) th layer respectively.

Original: c i +1 ﬁlters are contained in such a convolutional layer, each of which with the kernel size of c i ∗ k h ∗ k w , taking the feature maps with the size of c i ∗ h i ∗ w i of the i-th layer as input, and outputing the feature map with the size of h
Revised: c i +1 ﬁlters are contained in such a convolutional layer, each of which with the kernel size of c i ∗ k h ∗ k w , taking the feature maps with the size of c i ∗ h i ∗ w i of the i-th layer as input, and outputing the feature map with the size of h

Original: i +1 ∗
Revised: i +1 ∗

Original: w i
Revised: w i

Original: +1 .
Revised: +1 .

Original: In particular, we evaluate the weight of a ﬁlter as the performance reduction when the ﬁlter is disabled in the network.
Revised: In particular, we evaluate the weight of a ﬁlter as the performance reduction when the ﬁlter is disabled in the network.

Original: Intuitively, removing a ﬁlter with greater capacity of discrimination usually causes higher performance loss.
Revised: Intuitively, removing a ﬁlter with greater capacity of discrimination usually causes higher performance loss.

Original: In this way, such channels should be constrained more strictly since a useful representation for the target task is already learned by the source task.
Revised: In this way, such channels should be constrained more strictly since a useful representation for the target task is already learned by the source task.

Original: Given the pre-trained parameter ω ∗ and an input image x i , DELTA sets the weight of the j th channel, using the gap between the empirical losses of the networks on the labeled sample ( x i , y i ) with and without the j th channel, as follow,
Revised: Given the pre-trained parameter ω ∗ and an input image x i , DELTA sets the weight of the j th channel, using the gap between the empirical losses of the networks on the labeled sample ( x i , y i ) with and without the j th channel, as follow,

Original: [Equation]
Revised: [Equation]

Original: where ω ∗\ j refers to the modiﬁcation of original parameter ω ∗ with all elements of the j th ﬁlter set to zero (i.e., removing the j th ﬁlter from the network).
Revised: where ω ∗\ j refers to the modiﬁcation of original parameter ω ∗ with all elements of the j th ﬁlter set to zero (i.e., removing the j th ﬁlter from the network).

Original: We use softmax to normalize the result to ensure all weights are non-negative.
Revised: We use softmax to normalize the result to ensure all weights are non-negative.

Original: Above supervised attention mechanism yields a ﬁlter a higher weight for a speciﬁc image if and only if the corresponding feature map in the pre-trained source network is with higher discrimination power — i.e., paying more attention to such ﬁlter on such image might bring higher performance gain.
Revised: The aforementioned supervised attention mechanism yields a ﬁlter a higher weight for a speciﬁc image if and only if the corresponding feature map in the pretrained source network is with higher discrimination power — i.e., paying more attention to such ﬁlter on such image might bring higher performance gain.

Original: Note that, to calculate L ( z ( x i , ω ∗\ j ) , y i ) and L ( z ( x i , ω ∗ ) , y i ) for supervised attention mechanism, we introduce a baseline algorithm L 2 - FE that ﬁxes the feature extractor (with all parameters copied from source networks) and only trains the discriminators using the target task.
Revised: Note that, to calculate L ( z ( x i , ω ∗\ j ) , y i ) and L ( z ( x i , ω ∗ ) , y i ) for supervised attention mechanism, we introduce a baseline algorithm L 2 - FE that ﬁxes the feature extractor (with all parameters copied from source networks) and only trains the discriminators using the target task.

Original: The L 2 - FE model can be viewed as an adaption of the source network (weights) to the target tasks, without further modiﬁcations to the outer layer parameters.
Revised: The L 2 - FE model can be viewed as an adaption of the source network (weights) to the target tasks, without further modiﬁcations to the outer layer parameters.

Original: In our work, we use L 2 - FE to evaluate
Revised: In our work, we use L 2 - FE to evaluate

Original: L ( z ( x i , ω ∗\ j ) , y i ) and L ( z ( x i , ω ∗ ) , y i ) using the target datasets.
Revised: L ( z ( x i , ω ∗\ j ) , y i ) and L ( z ( x i , ω ∗ ) , y i ) using the target datasets.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: We have conducted a comprehensive experimental study of the proposed DELTA method.
Revised: We have conducted a comprehensive experimental study of the proposed DELTA method.

Original: Below we ﬁrst brieﬂy review the used datasets, followed by a description of experimental procedure and ﬁnally our observations.
Revised: Below we ﬁrst brieﬂy review the used datasets, followed by a description of experimental procedure and ﬁnally our observations.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We evaluate the performance of three benchmarks with different tasks:
Revised: We evaluate the performance of three benchmarks with different tasks:

Original: Caltech 256 for general object recognition, Stanford Dogs 120 for ﬁne-grained object recognition, and MIT Indoors 67 for scene classiﬁcation.
Revised: Caltech 256 for general object recognition, Stanford Dogs 120 for ﬁne-grained object recognition, and MIT Indoors 67 for scene classiﬁcation.

Original: For the ﬁrst two benchmarks, we used ImageNet as the source domain and
Revised: For the ﬁrst two benchmarks, we used ImageNet as the source domain and

Original: Places 365 for the last one.
Revised: Places 365 for the last one.

Original: [Figure]

Original: Caltech 256.
Revised: Caltech 256.

Original: Caltech 256 is a challenging set of 256 object categories containing a total ofimages.
Revised: Caltech 256 is a dataset with 256 object categories containing a total of 30607 images.

Original: Different numbers of training examples are used by researchers to validate the generalization of proposed algorithms.
Revised: Different numbers of training examples are used by researchers to validate the generalization of proposed algorithms.

Original: In this paper, we create two conﬁgurations for Caltech 256, which have 30 and 60 random sampled training examples respectively for each category, following the procedure used in (Li et al., 2018).
Revised: In this paper, we create two conﬁgurations for Caltech 256, which have [Figure] and 60 random sampled training examples respectively for each category, following the procedure used in (Li et al., 2018).

Original: Stanford Dogs 120.
Revised: Stanford Dogs 120.

Original: The Stanford Dogs dataset contains images of 120 breeds of dogs from around the world.
Revised: The Stanford Dogs dataset contains images of 120 breeds of dogs from around the world.

Original: There are exactly 100 examples per category in the training set.
Revised: There are exactly 100 examples per category in the training set.

Original: It is used for the task of ﬁne-grained image categorization.
Revised: It is used for the task of ﬁne-grained image categorization.

Original: We do not use the bounding box annotations.
Revised: We do not use the bounding box annotations.

Original: MIT Indoors 67.
Revised: MIT Indoors 67.

Original: MIT Indoors 67 is a scene classiﬁcation task containing 67 indoor scene categories, each of which consists of 80 images for training and 20 for testing.
Revised: MIT Indoors 67 is a scene classiﬁcation task containing 67 indoor scene categories, each of which consists of 80 images for training and 20 for testing.

Original: Indoor scene recognition is challenging because both spatial properties and object characters are expected to be extracted.
Revised: Indoor scene recognition is challenging because both spatial properties and object characters are expected to be extracted.

Original: Caltech-UCSD Birds-200-2011.
Revised: Caltech-UCSD Birds-200-2011.

Original: CUB-200-2011 contains 11,788 images of 200 bird species.
Revised: CUB-200-2011 contains 11,788 images of 200 bird species.

Original: Each species is associated with a wikipedia article and organized by scientiﬁc classiﬁcation.
Revised: Each species is associated with a wikipedia article and organized by scientiﬁc classiﬁcation.

Original: Each image is annotated with bounding box, part location, and attribute labels.
Revised: Each image is annotated with bounding box, part location, and attribute labels.

Original: We use only classiﬁcation labels during training.
Revised: We use only classiﬁcation labels during training.

Original: While part location annotations are used in a quantitative evaluation of show cases, to explain the transferring effect of our algorithm.
Revised: While part location annotations are used in a quantitative evaluation of show cases, to explain the transferring effect of our algorithm.

Original: Food-101.
Revised: Food-101.

Original: Food-101 a large scale data set of 101 food categories, with 101000 images, for the task of ﬁne-grained image categorization.
Revised: Food-101 a large scale data set of 101 food categories, with 101,000 images, for the task of ﬁne-grained image categorization.

Original: 750 training images and 250 test images are provided for each class.
Revised: 750 training images and 250 test images are provided for each class.

Original: This dataset is challenging because the training images contain some amount of noise.
Revised: This dataset is challenging because the training images contain some amount of noise.

Original: 4. E XPERIMENTAL P ROCEDURE
Revised: 4. E XPERIMENTAL P ROCEDURE

Original: We implemented our method with ResNet-101 and Inception-V3 as the base networks.
Revised: We implemented our method with ResNet-101 and Inception-V3 as the base networks.

Original: For experiment set up we followed almost the same procedure in (Li et al., 2018) due to the close relationship between our work and theirs.
Revised: For experiment set up we followed almost the same procedure in (Li et al., 2018) due to the close relationship between our work and theirs.

Original: After training with the source dataset and before ﬁne-tuning the network with the target dataset, we replace the last layer of the base network with random initialization in suit for the target dataset.
Revised: After training with the source dataset and before ﬁne-tuning the network with the target dataset, we replace the last layer of the base network with random initialization in suit for the target dataset.

Original: For ResNet-101, the input images are resized to 256*256 and normalized to zero mean for each channel, following with data augmentation operations of random mirror and random crop to 224*224.
Revised: For ResNet-101, the input images are resized to 256*256 and normalized to zero mean for each channel, following with data augmentation operations of random mirror and random crop to 224*224.

Original: For Inception-V3, images are resized to 320*320 and ﬁnally cropped to 229*229.
Revised: For Inception-V3, images are resized to 320*320 and ﬁnally cropped to 229*229.

Original: We use a batch size of 64.
Revised: We use a batch size of 64.

Original: SGD with the momentum of 0.9 is used for optimizing all models.
Revised: SGD with the momentum of 0.9 is used for optimizing all models.

Original: The learning rate for the base model starts with 0.01 for ResNet-101 and 0.001 for Inception-V3, and is divided byafter 6000 iterations.
Revised: The learning rate for the base model starts with 0.01 for ResNet-101 and 0.001 for Inception-V3, and is divided byafter 6000 iterations.

Original: The training is ﬁnished at 9000 iterations.
Revised: The training is ﬁnished at 9000 iterations.

Original: We use ﬁve-fold cross validation for searching the best conﬁgurations of the hyperparameter α for each experiment.The hyperparameter [Table]
Revised: We use ﬁve-fold cross validation for searching the best conﬁgurations of the hyperparameter α for each experiment.

Original: [Table] [Table] β is ﬁxed to 0.01.
Revised: The hyperparameter β is ﬁxed to 0.01.

Original: As was mentioned, our experiments compared DELTA to several key baseline algorithms including L 2 , L 2 - SP (Li et al., 2018), and L 2 - FE (see also in Section 3.4), all under the same settings.
Revised: As was mentioned, our experiments compared DELTA to several key baseline algorithms including L 2 , L 2 - SP (Li et al., 2018), and L 2 - FE (see also in Section 3.4), all under [Table] [Table] [Table]

Revised: the same settings.

Original: Each experiment is repeated ﬁve times.
Revised: Each experiment is repeated ﬁve times.

Original: The average top-1 classiﬁcation accuracy and standard devision is reported.
Revised: The average top-1 classiﬁcation accuracy and standard division is reported.

Original: 4. R ESULTS AND C OMPARISONS
Revised: 4. R ESULTS AND C OMPARISONS

Original: In Fig 2 we plotted a sample learning curve of training with different regularization techniques.
Revised: In Fig 2 we plotted a sample learning curve of training with different regularization techniques.

Original: Comparing these regularization techniques, we observe that our proposed DELTA shows faster convergence than the simple L 2 - SP regularization with both step decay(StepLR) and exponential decay(ExponentialLR) learning rate scheduler.
Revised: Comparing these regularization techniques, we observe that our proposed DELTA shows faster convergence than the simple L 2 - SP regularization with both step decay (StepLR) and exponential decay (ExponentialLR) learning rate scheduler.

Original: In addition, we ﬁnd that the learning curve of
Revised: In addition, we ﬁnd that the learning curve of

Original: DELTA is smoother than L 2 - SP and it is not sensitive to the learning rate decay happened at the 6000th iteration when using StepLR.
Revised: DELTA is smoother than L 2 - SP and it is not sensitive to the learning rate decay happened at the 6000th iteration when using StepLR.

Original: In Table 1 we show the results of our proposed method DELTA with and without attention, compared to the baseline of L 2 - SP reported in (Li et al., 2018) and also the naive L 2 - FE and Lmethods.
Revised: In Table 1 we show the results of our proposed method DELTA with and without attention, compared to the baseline of L 2 - SP reported in (Li et al., 2018) and also the naive L 2 - FE and Lmethods.

Original: We surprisingly ﬁnd that ﬁne-tuning using L 2 normalization doesn’t perform better than directly using the pre-trained model as a feature extractor( L 2 - FE ). While L 2 - SP outperforms the naive methods without SPAR.
Revised: We surprisingly ﬁnd that ﬁne-tuning using L 2 normalization does not perform better than directly using the pre-trained model as a feature extractor( L 2 - FE ), while L 2 - SP outperforms the naive methods without SPAR.

Original: We observe that greater beneﬁts are gained using our proposed attention mechanism.
Revised: We observe that greater beneﬁts are gained using our proposed attention mechanism.

Original: Data augmentation is a widely used technique to improve image classiﬁcation.
Revised: Data augmentation is a widely used technique to improve image classiﬁcation.

Original: Following (Li et al., 2018), we used a simple data augmentation method and a post-processing trick.
Revised: Following (Li et al., 2018), we used a simple data augmentation method and a post-processing technique.

Original: First, we keep the original aspect ratio of input images by resizing them with the shorter edge being 256, instead of ignoring the aspect ratio and directly resizing them to 256*256.
Revised: First, we keep the original aspect ratio of input images by resizing them with the shorter edge being 256, instead of ignoring the aspect ratio and directly resizing them to 256*256.

Original: Second, we apply 10-crop testing to further improve the performance.
Revised: Second, we apply 10-crop testing to further improve the performance.

Original: In Table 2, we documented the experimental results using these technique with different regularization methods.
Revised: In Table 2, we documented the experimental results using these technique with different regularization methods.

Original: We observe a clear pattern that with additional data augmentation, all the three evaluated methods L 2 , L 2 - SP , DELTA have improved classiﬁcation accuracy while our method still delivers the best one.
Revised: We observe a clear pattern that with additional data augmentation, all the three evaluated methods L 2 , L 2 - SP , DELTA have improved classiﬁcation accuracy while our method still delivers the best one.

Original: 4. A C ASE S TUDY AND D ISCUSSIONS
Revised: 4. A C ASE S TUDY AND D ISCUSSIONS

Original: To better understand the performance gain of DELTA we performed an experiment where we analyzed how parameters of the convolution ﬁlters change after ﬁne-tuning.
Revised: To better understand the performance gain of DELTA we performed an experiment where we analyzed how parameters of the convolution ﬁlters change after ﬁne-tuning.

Original: Towards that purpose we randomly sampled images from the testing set of Stanford Dogs 120.
Revised: Towards that purpose we randomly sampled images from the testing set of Stanford Dogs 120.

Original: For ResNet-101, which we use exclusively in this paper, we grouped ﬁlters into parts as describe in (he et al., 2016).
Revised: For ResNet-101, which we use exclusively in this paper, we grouped ﬁlters into stages as described in (he et al., 2016).

Original: These parts are conv2 x, conv3 x, conv4 x, conv5 x.
Revised: These stages are conv2 x, conv3 x, conv4 x, conv5 x.

Original: Each part contains a few stacked blocks and a block is a basic inception unit having 3 conv2d layers.
Revised: Each stage contains a few stacked blocks and a block is a basic inception unit having 3 conv2d layers.

Original: One conv2d layer consists of a number of output channels, each of which is a convolution ﬁlter.
Revised: One conv2d layer consists of a number of output ﬁlters.

Original: We ﬂatten each ﬁlter into a one dimension parameter vector for convenience.
Revised: We ﬂatten each ﬁlter into a one dimension parameter vector for convenience.

Original: The Euclidian distance between the parameter vectors before and after ﬁne-tuning is calculated.
Revised: The Euclidian distance between the parameter vectors before and after ﬁne-tuning is calculated.

Original: All distances are sorted as shown in Figure 3.
Revised: All distances are sorted as shown in Figure 3.

Original: We observed a sharp difference between the two distance distributions.
Revised: We observed a sharp difference between the two distance distributions.

Original: Our hypothesis of possible cause of the difference is that simply using L 2 - SP regularization all convolution ﬁlters are forced to be similar to the original ones.
Revised: Our hypothesis of possible cause of the difference is that simply using L 2 - SP regularization all convolution ﬁlters are forced to be similar to the original ones.

Original: Using attention, we allow “unactivated” convolution ﬁlters to be reused for better image classiﬁcation.
Revised: Using attention, we allow “unactivated” convolution ﬁlters to be reused for better image classiﬁcation.

Original: About 90% parameter vectors of DELTA have larger distance than L 2 - SP .
Revised: About 90% parameter vectors of DELTA have larger distance than L 2 - SP .

Original: We also observe that a small number of ﬁlters is driven very far away from the initial value (as shown at the left end of the curves in Figure 3).
Revised: We also observe that a small number of ﬁlters is driven very far away from the initial value (as shown at the left end of the curves in Figure 3).

Original: We call such an effect as “unactivated channel re-usage”.
Revised: We call such an effect as “unactivated channel re-usage”.

Original: of the distance of parameters from the starting point.

Original: In ResNet-101, conv2 x, conv3 x, conv4 x, conv5 x represent for four main layers each of which stacked with numbers of blocks.

Original: The blue line represents for the result of L 2 - SP , and the orange line for DELTA .

Original: To further understand the effect of attention and the implication of “unactivated channel re-usage”, we “attributed” the attention to the original  to see the set of pixels having high contribution in the activated feature maps.
Revised: To further understand the effect of attention and the implication of “unactivated channel re-usage”, we “attributed” the attention to the original image to identify the set of pixels having high contributions in the activated feature maps.

Original: We select some convolution ﬁlters on which the source model (the initialization before ﬁne-tuning) has low activation.
Revised: We select some convolution ﬁlters on which the source model (the initialization before ﬁne-tuning) has low activation.

Original: For the convenience of analyzing the effectof
Revised: For the convenience of analyzing the effect

Original: [Table]
Revised: [Table]

Original:  regularization methods, each element a i of the original activation map is normalized with
Revised: of regularization methods , each element a i of the original activation map is normalized with

Original: [Equation]
Revised: [Equation]

Original: where the min and max term in the formula represent for the minimum and maximum value of the whole activation map respectively.
Revised: where the min and max terms in the formula represent for the minimum and maximum value of the whole activation map respectively.

Original: Activation maps of these convolution ﬁlter for various regularization method are presented on each row.
Revised: Activation maps of these convolution ﬁlter for various regularization method are presented on each row.

Original: As shown in Figure 4, our ﬁrst observation is that without attention, the activation maps from
Revised: As shown in Figure 4, our ﬁrst observation is that without attention, the activation maps from

Original: DELTA in different images are more or less the same activation maps from other regularization methods.
Revised: DELTA in different images are more or less the same activation maps from other regularization methods.

Original: This partially explains the fact that we do not observe signiﬁcant improvement of DELTA without attention.
Revised: This partially explains the fact that we do not observe signiﬁcant improvement of DELTA without attention.

Original: Using attention, however, changes the activation map signiﬁcantly.
Revised: Using attention, however, changes the activation map signiﬁcantly.

Original: Regularization of DELTA with attention show obviously improved concentration.
Revised: Regularization of DELTA with attention show obviously improved concentration.

Original: With attention (the right-most column in Figure 4), we observed a large set of pixels that have high activation at important regions around the head of the animals.
Revised: With attention (the right-most column in Figure 4), we observed a large set of pixels that have high activation at important regions around the head of the animals.

Original: We believe this phenomenon provides additional evidence to support our intuition of “unactivated channel re-usage” as discussed inthe previous paragraph.
Revised: We believe this phenomenon provides additional evidence to support our intuition of “unactivated channel re-usage” as discussed in previous paragraphs.

Original: In addition, we added statistical results of activations on part locations of CUB-200-2011 supporting the above qualitative cases.
Revised: In addition, we included new statistical results of activations on part locations of CUB-200supporting the above qualitative cases.

Original: The CUB-200-2011 datasets deﬁned 15 discriminative parts of birds, eg. the forehead, tail, beak and so on.
Revised: The CUB-200-2011 datasets deﬁned 15 discriminative parts of birds, e.g. the forehead, tail, beak and so on.

Original: Each part is annotated with a pixel location representing for its center position if it is visible.
Revised: Each part is annotated with a pixel location representing for its center position if it is visible.

Original: So for each image, we got several key points which are very important to discriminate its category.
Revised: So for each image, we got several key points which are very important to discriminate its category.

Original: Using all testing examples of CUB-200-2011, we calculated normalized activations on these key points of these different regularization methods.
Revised: Using all testing examples of CUB-200-2011, we calculated normalized activations on these key points of these different regularization methods.

Original: As shown in Table 3, DELTA got the highest average activations on those key points, demonstrating that DELTA focused on more discriminate features for bird recognition.
Revised: As shown in Table 3, DELTA got the highest average activations on those key points, demonstrating that DELTA focused on more discriminate features for bird recognition.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we studied the regularization technique that transfers the behaviors and semantics of the source network to the target one through constraining the difference between the feature maps generated by the outer layers of source/target networks with attentions.
Revised: In this paper we studied a regularization technique that transfers the behaviors and semantics of the source network to the target one through constraining the difference between the feature maps generated by the convolution layers of source/target networks with attentions.

Original: Non-trivial contribution has been made compared to the existing methods that aims at constraining the difference between the parameters of networks such as L 2 - SP regularization Li et al.

Original: Speciﬁcally, we designed a regularized learning algorithm DELTA that models the difference of feature maps with attentions between networks, where the attention models are obtained through supervised learning.
Revised: Speciﬁcally, we designed a regularized learning algorithm DELTA that models the difference of feature maps with attentions between networks, where the attention models are obtained through supervised learning.

Original: Moreover, we further accelerate the optimization for regularization using start point as reference (SPAR).
Revised: Moreover, we further accelerate the optimization for regularization using start point as reference (SPAR).

Original: We conduct extensive experiments to evaluate DELTA using several real-world datasets based on typical convolutional neural networks.
Revised: Our extensive experiments evaluated DELTA using several real-world datasets based on commonly used convolutional neural networks.

Original: The experiment resultsand comparisons show that DELTA can signiﬁcantly outperform the state of the arts with higher accuracy
Revised: The experiment results show that DELTA is able to signiﬁcantly outperform the state-of-the-art transfer learning methods

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Examples with different regularization methods from Stanford Dogs and CUB-200-2011.
Revised: Examples with different regularization methods from Stanford Dogs and CUB-200-2011.

Original: [Figure]
Revised: [Figure]