Original: F EATURE I NCAY FOR R EPRESENTATION R EGULARIZATION
Revised: F EATURE I NCAY FOR R EPRESENTATION R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Softmax-based loss is widely used in deep learning for multi-class classiﬁcation, where each class is represented by a weight vector and each sample is represented as a feature vector.
Revised: Softmax-based loss is widely used in deep learning for multi-class classiﬁcation, where each class is represented by a weight vector and each sample is represented as a feature vector.

Original: Different from traditional learning algorithms where features are pre-deﬁned and only weight vectors are tunable through training, feature vectors are also tunable as representation learning in deep learning.
Revised: Different from traditional learning algorithms where features are pre-deﬁned and only weight vectors are tunable through training, feature vectors are also tunable as representation learning in deep learning.

Original: Thus we investigate how to improve the classiﬁcation performance by better adjusting the features.
Revised: Thus we investigate how to improve the classiﬁcation performance by better adjusting the features.

Original: One main observation is that elongating the feature norm of both correctly-classiﬁed and mis-classiﬁed feature vectors improves learning: (1) increasing the feature norm of correctly-classiﬁed examples induce smaller training loss; (2) increasing the feature norm of mis-classiﬁed examples can upweight the contribution from hard examples.
Revised: One main observation is that elongating the feature norm of both correctly-classiﬁed and mis-classiﬁed feature vectors improves learning: (1) increasing the feature norm of correctly-classiﬁed examples induce smaller training loss; (2) increasing the feature norm of mis-classiﬁed examples can upweight the contribution from hard examples.

Original: Accordingly, we propose feature incay to regularize representation learning by encouraging larger feature norm.
Revised: Accordingly, we propose feature incay to regularize representation learning by encouraging larger feature norm.

Original: In contrast to weight decay which shrinks the weight norm, feature incay is proposed to stretch the feature norm.
Revised: In contrast to weight decay which shrinks the weight norm, feature incay is proposed to stretch the feature norm.

Original: Extensive empirical results on MNIST, CIFAR10, CIFAR and LFW demonstrate the effectiveness of feature incay.
Revised: Extensive empirical results on MNIST, CIFAR10, CIFAR and LFW demonstrate the effectiveness of feature incay.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: Deep Neural Networks (DNNs) with softmax-based loss have achieved stateof-the-art performance on numerous multi-class classiﬁcation related tasks.
Revised: Deep Neural Networks (DNNs) with softmax-based loss have achieved stateof-the-art performance on numerous multi-class classiﬁcation related tasks.

Original: In DNNs, both representations and classiﬁers are learned within a uniﬁed network concurrently, where the ﬁnal representation for a sample is the feature vector f outputted from the penultimate layer, while the last layer outputs scores
Revised: In DNNs, both representations and classiﬁers are learned within a uniﬁed network concurrently, where the ﬁnal representation for a sample is the feature vector f outputted from the penultimate layer, while the last layer outputs scores

Original: z i = w i · f for each category i , where w i is the weight vector for category i .
Revised: z i = w i · f for each category i , where w i is the weight vector for category i .

Original: Before deﬁning the loss, the scores for each category are normalized into probability via softmax function, i.e., p i = ezi (cid:80) j ezj .
Revised: Before deﬁning the loss, the scores for each category are normalized into probability via softmax function, i.e., p i = ezi (cid:80) j ezj .

Original: A well-trained DNN should output signiﬁcant larger probability for the correct label than other labels, which requires the score for the correct label is significantly larger than other labels.
Revised: A well-trained DNN should output signiﬁcant larger probability for the correct label than other labels, which requires the score for the correct label is significantly larger than other labels.

Original: Since z i = w i · f = (cid:107) w i (cid:107)(cid:107) f (cid:107) cos( θ ) , where θ is the angle between w i and f , the goal of signiﬁcant larger score for the correct label than other labels can be achieved by tuning (cid:107) w i (cid:107) , (cid:107) f (cid:107) and θ .
Revised: Since z i = w i · f = (cid:107) w i (cid:107)(cid:107) f (cid:107) cos( θ ) , where θ is the angle between w i and f , the goal of signiﬁcant larger score for the correct label than other labels can be achieved by tuning (cid:107) w i (cid:107) , (cid:107) f (cid:107) and θ .

Original: While increasing the weight norm (cid:107) w i (cid:107) is constrained by weight decay for regularization, thus (cid:107) f (cid:107) and θ become the two main factors for optimization.
Revised: While increasing the weight norm (cid:107) w i (cid:107) is constrained by weight decay for regularization, thus (cid:107) f (cid:107) and θ become the two main factors for optimization.

Original: Although softmax loss can tune both of them, there still exist much room to improve either or both factors.
Revised: Although softmax loss can tune both of them, there still exist wide room to improve either or both factors.

Original: [Figure]
Revised: [Figure]

Original: For example, Liu et al.
Revised: For example, Liu et al.

Original: (2016), Wang et al.
Revised: (2016), Wang et al.

Original: (2017) and Liu et al.
Revised: (2017) and Liu et al.

Original: (2017a) propose different approaches to further optimize the factor of angular θ , and all of them have achieved obviously better performance.
Revised: (2017a) propose different approaches to further optimize the factor of angular θ , and all of them have achieved obviously better performance.

Original: To further emphasize the factor of angular, Ranjan et al.
Revised: To further emphasize the factor of angular, Ranjan et al.

Original: (2017) and Liu et al.
Revised: (2017) and Liu et al.

Original: (2017c) propose to use normalized feature vectors for softmax loss where the factor of feature norm is totally ignored.
Revised: (2017c) propose to use normalized feature vectors for softmax loss where the factor of feature norm is totally ignored.

Original: In this work, we make the effort to optimize the feature norm.
Revised: In this work, we make the effort to optimize the feature norm.

Original: Firstly, we analyze the connections between the feature norm and the classiﬁcation accuracy within softmax loss, they are highly correlated as illustrated in Figure 1.
Revised: Firstly, we analyze the connections between the feature norm and the classiﬁcation accuracy within softmax loss, they are highly correlated as illustrated in Figure 1.

Original: Features with larger norm tend to be correctly classiﬁed with higher probability.
Revised: Features with larger norm tend to be correctly classiﬁed with higher probability.

Original: Here we propose to optimize the feature norm by augmenting the softmax loss with feature incay.
Revised: Here we propose to optimize the feature norm by augmenting the softmax loss with feature incay.

Original: In contrast to weight decay that shrinks weight vectors to be of small norm, feature incay tends to stretch out the feature vectors.
Revised: In contrast to weight decay that shrinks weight vectors to be of small norm, feature incay tends to stretch out the feature vectors.

Original: From the computational perspective, larger feature norm results in larger score differences among categories, which can better separate the categories.
Revised: From the computational perspective, larger feature norm results in larger score differences among categories, which can better separate the categories.

Original: From the perspective of pattern detection, larger feature norm encourages model to learn and detect more prominent patterns.
Revised: From the perspective of pattern detection, larger feature norm encourages model to learn and detect more prominent patterns.

Original: Figure 2(a), 2(b) and 2(c) show the results of comparison experiments by adding feature incay to softmax loss, where feature incay achieves larger feature norm, smaller loss value and higher accuracy on test set.
Revised: Figure 2(a), 2(b) and 2(c) show the results of comparison experiments by adding feature incay to softmax loss, where feature incay achieves larger feature norm, smaller loss value and higher accuracy on test set.

Original: The geometric interpretation of feature incay is illustrated in Figure 2(d), 2(e) and 2(f), we can achieve the largest inter-class separability by explicitly optimizing both the (cid:107) f (cid:107) and θ compared with the other methods.
Revised: The geometric interpretation of feature incay is illustrated in Figure 2(d), 2(e) and 2(f), we can achieve the largest inter-class separability by explicitly optimizing both the (cid:107) f (cid:107) and θ compared with the other methods.

Original: Besides, the proposed feature incay (implemented as
Revised: Besides, the proposed feature incay (implemented as

Original: Reciprocal Norm Loss) is designed to increase the feature norm adaptively according to the original feature norm, which can also help reduce the intra-class variances as illustrated in Figure 5.
Revised: Reciprocal Norm Loss) is designed to increase the feature norm adaptively according to the original feature norm, which can also help reduce the intra-class variances as illustrated in Figure 5.

Original: In summary, we analyze the effect of feature norm and prove that: (1) increasing feature norm for correctly-classiﬁed examples induce smaller training loss; (2) increasing feature norm for misclassiﬁed examples can up-weight the contribution of hard examples; (3) the bound on feature norm to ensure the inter-class separability and intra-class compactness.
Revised: In summary, we analyze the effect of feature norm and prove that (1) increasing feature norm for correctly-classiﬁed examples induce smaller training loss. (2) increasing feature norm for misclassiﬁed examples can up-weight the contribution of hard examples. (3) the bound on feature norm to ensure the inter-class separability and intra-class compactness.

Original: The proposed feature incay is veriﬁed on four widely used classiﬁcation datasets(i.e.
Revised: The proposed feature incay is veriﬁed on four widely used classiﬁcation datasets(i.e.

Original: , MNIST, CIFAR10, CIFAR100 and LFW) using various network architecture.
Revised: , MNIST, CIFAR10, CIFAR100 and LFW) using various network architecture.

Original: By considering the feature incay, we achieve comparable performances on all of them.
Revised: By considering the feature incay, we achieve comparable performances on all of them.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Large-margin Softmax Loss.
Revised: Large-margin Softmax Loss.

Original: Liu et al.
Revised: Liu et al.

Original: (2016) proposed to improve softmax loss by incorporating an adjustable margin m multiplying the angle between a feature vector and the corresponding weight vector.
Revised: (2016) proposed to improve softmax loss by incorporating an adjustable margin m multiplying the angle between a feature vector and the corresponding weight vector.

Original: Compared with the softmax loss, it pays more attention to the angular decision margin between classes as illustrated in Figure 2(e).
Revised: Compared with the softmax loss, it pays more attention to the angular decision margin between classes as illustrated in Figure 2(e).

Original: Large-margin softmax loss appends stronger constraint to the angular, while feature incay considers constraint to feature norm.
Revised: Large-margin softmax loss appends stronger constraint to the angular, while feature incay considers constraint to feature norm.

Original: As illustrated in Figure 2(f), feature incay is orthogonal to large-margin softmax loss.
Revised: As illustrated in Figure 2(f), feature incay is orthogonal to large-margin softmax loss.

Original: Center Loss.
Revised: Center Loss.

Original: Wen et al.
Revised: Wen et al.

Original: (2016) presented the center loss to learn centers for deep features of each class and penalize the distances between the deep features and their corresponding class centers.
Revised: (2016) presented the center loss to learn centers for deep features of each class and penalize the distances between the deep features and their corresponding class centers.

Original: The softmax loss tries to align feature vectors close to the weight vectors based on the inner product similarity, while center loss pushes feature vectors towards their class centers according Euclidean distances.
Revised: The softmax loss tries to align feature vectors close to the weight vectors based on the inner product similarity, while center loss pushes feature vectors towards their class centers according Euclidean distances.

Original: Combining softmax loss with center loss actually uses two sets of classiﬁers, where representation is learned based on both the inner product to weight vector and the Euclidean distance to class center.
Revised: Combining softmax loss with center loss actually uses two sets of classiﬁers, where representation is learned based on both the inner product to weight vector and the Euclidean distance to class center.

Original: The added center loss helps minimize the intra-class distances also by inﬂuencing the feature norm, namely, small feature norm will be increased and large feature norm will be decreased during the process of pushing feature vectors to class centers.
Revised: The added center loss helps minimize the intra-class distances also by inﬂuencing the feature norm, namely, small feature norm will be increased and large feature norm will be decreased during the process of pushing feature vectors to class centers.

Original: Different from center loss, feature incay also increases the large feature norm instead of penalizing feature vectors with large norm as center loss.
Revised: Different from center loss, feature incay also increases the large feature norm instead of penalizing feature vectors with large norm as center loss.

Original: Weight/Feature Normalization.
Revised: Weight/Feature Normalization.

Original: Inspired by the fact that feature normalization before calculating the sample distances usually achieves better performance for retrieval tasks, Ranjan et al.
Revised: Inspired by the fact that feature normalization before calculating the sample distances usually achieves better performance for retrieval tasks, Ranjan et al.

Original: (2017) proposed to use normalized feature vectors in softmax loss during training, thus the feature norm has no effect on softmax loss and angle is the main factor to be optimized.
Revised: (2017) proposed to use normalized feature vectors in softmax loss during training, thus the feature norm has no effect on softmax loss and angle is the main factor to be optimized.

Original: Congenerous cosine loss(Liu et al.
Revised: Congenerous cosine loss(Liu et al.

Original: (2017b)),
Revised: (2017b)),

Original: NormFace(Wang et al.
Revised: NormFace(Wang et al.

Original: (2017)), and cosine normalization(Chunjie et al.
Revised: (2017)), and cosine normalization(Chunjie et al.

Original: (2017)) take a step further to normalize the weight vectors which replace inner product with cosine similarity within softmax loss, and only optimize the factor of angle.
Revised: (2017)) take a step further to normalize the weight vectors which replace inner product with cosine similarity within softmax loss, and only optimize the factor of angle.

Original: Although normalization mechanism achieves much lower intra-class angular variability by emphasizing more on the angle during training, they ignore that feature norm is another useful factor worth to optimize.
Revised: Although normalization mechanism achieves much lower intra-class angular variability by emphasizing more on the angle during training, they ignore that feature norm is another useful factor worth to optimize.

Original: Feature Scale.
Revised: Feature Scale.

Original: COCO(Liu et al.
Revised: COCO(Liu et al.

Original: (2017c))
Revised: (2017c))

Original: and L 2 -softmax(Ranjan et al.
Revised: and L 2 -softmax(Ranjan et al.

Original: (2017)) are the most relevant to our work, and especially COCO is a concurrent work.
Revised: (2017)) are the most relevant to our work, and especially COCO is a concurrent work.

Original: Both of the them introduce a single scaling parameter to increase the magnitude of the features.
Revised: Both of the them introduce a single scaling parameter to increase the magnitude of the features.

Original: However, they all enforce the L 2 -norm of the features to be ﬁxed for all samples.
Revised: However, they all enforce the L 2 -norm of the features to be ﬁxed for all samples.

Original: Speciﬁcally, they are optimizing the feature vectors on a hypersphere with ﬁxed radius.
Revised: Speciﬁcally, they are optimizing the feature vectors on a hypersphere with ﬁxed radius.

Original: Different from them, we are trying to investigate whether it is possible to optimize the original feature space instead of the “hypersphere” feature space.
Revised: Different from them, we are trying to investigate whether it is possible to optimize the original feature space instead of the “hypersphere” feature space.

Original:  O UR WORK 3. R EVISITING SOFTMAX LOSS
Revised:  O UR WORK 3. R EVISITING SOFTMAX LOSS

Original: Let X = { ( x i , y i ) } Ni =1 be the training set contains N samples, where x i is the raw input to the DNN, y i ∈ { 1 , 2 , · · · , K } is the class label that supervises the output of the DNN.
Revised: Let X = { ( x i , y i ) } Ni =1 be the training set contains N samples, where x i is the raw input to the DNN, y i ∈ { 1 , 2 , · · · , K } is the class label that supervises the output of the DNN.

Original: Denote f i as the feature vector for x i learned by the DNN, { w j } Kj =1 represent weight vectors for the K categories.
Revised: Denote f i as the feature vector for x i learned by the DNN, { w j } Kj =1 represent weight vectors for the K categories.

Original: Then, softmax loss is deﬁned as,
Revised: Then, softmax loss is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: where bias terms are ignored following the discussions in recent works Liu et al.
Revised: Bias terms are ignored following the discussions in recent works Liu et al.

Original: (2016) and Wang et al.
Revised: (2016) and Wang et al.

Original: Denote the angle between w j and f i as θ w j , f i , the inner product between w j and f i can be rewritten as
Revised: Denote the angle between w j and f i as θ w j , f i , the inner product between w j and f i can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: By combining the above two equations, we get
Revised: By combining the above two equations, we get

Original: [Equation]
Revised: [Equation]

Original: 3. F EATURE NORM MATTERS
Revised: 3. F EATURE NORM MATTERS

Original: Here we will illustrate how feature norm inﬂuences the softmax loss from two aspects: (1) increasing the feature norm of correctly-classiﬁed examples can induce smaller training loss; (2) increasing the feature norm of mis-classiﬁed examples can up-weight the contribution from the hard examples.
Revised: Here we will illustrate how feature norm inﬂuences the softmax loss from two aspects: (1)increasing the feature norm of correctly-classiﬁed examples can induce smaller training loss. (2)increasing the feature norm of mis-classiﬁed examples can up-weight the contribution from the hard examples.

Original: The ﬁrst property is similar to the proposition proved by Wang et al.
Revised: The ﬁrst property is similar to the proposition proved by Wang et al.

Original: (2017), which states that softmax loss always encourages features of the correctly-classiﬁed examples to have larger magnitudes.
Revised: (2017), which states that softmax loss always encourages features of the correctly-classiﬁed examples to have larger magnitudes.

Original: Property 1 [Feature Norm Matters for Correctly-classiﬁed Examples] Suppose weight vectors and directions of the feature vectors are ﬁxed, increasing  feature norm of correctly-classiﬁed examples canfurther decrease the softmax loss.
Revised: Property 1 [Feature Norm Matters for Correctly-classiﬁed Examples] Suppose weight vectors and directions of the feature vectors are ﬁxed, increasing of the feature norm of correctly-classiﬁed examples can decrease the softmax loss.

Original: Proof.
Revised: Proof.

Original: Let L softmax ( f i ) represent the loss of the i -th sample, i = 1 , · · · , N .
Revised: Let L softmax ( f i ) represent the loss of the i -th sample, i = 1 , · · · , N .

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: Recall that when f i is correctly classiﬁed, we have w Ty i f i > w Tj f i for any j (cid:54) =
Revised: Recall that when f i is correctly classiﬁed, we have w Ty i f i > w Tj f i for any j (cid:54) =

Original: y i , and (cid:107) w j (cid:107)(cid:107)
Revised: y i , and (cid:107) w j (cid:107)(cid:107)

Original: f i (cid:107) cos( θ w j , f i ) − (cid:107) w 
Revised: f i (cid:107) cos( θ w j , f i ) − (cid:107) w y i

Original: [Equation]
Revised: [Equation]

Original: cos
Revised: f i

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: cos

Original: [Equation]
Revised: [Equation]

Original:  yi
Revised: θ w yi

Original: [Equation]
Revised: [Equation]

Original: f i ) ≤ 0 always holds.
Revised: f i ) ≤ 0 always holds.

Original: Then, for any t > 0 , we have
Revised: Then, for any t > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: which means increasing the norm of correctly classiﬁed samples can decrease the softmax loss.
Revised: which means increasing the norm of correctly classiﬁed samples can decrease the softmax loss.

Original: To consider all samples including incorrectly classiﬁed ones, we set t i > 0 if i is correctly classiﬁed and t i = 

Revised: To consider all samples including incorrectly classiﬁed ones, we set t i > 0 if i is correctly classiﬁed and t i =

Original: otherwise, then we have
Revised: otherwise, then we have

Original: [Equation]
Revised: [Equation]

Original: So feature norm is an important factor to achieve smaller softmax loss together with the angle.
Revised: So feature norm is an important factor to achieve smaller softmax loss together with the angle.

Original: [Equation]
Revised: [Equation]

Original: Though the decrease in softmax loss is marginal for some samples already with small softmax loss, the increased feature norm enlarges the margin among different categories which ensures better generalization.
Revised: Though the decrease in softmax loss is marginal for some samples already with small softmax loss, the increased feature norm enlarges the margin among different categories which ensures better generalization.

Original: Property 2 [Feature Norm Matters for Mis-classiﬁed Examples] Fora mis-classiﬁed feature vector f i with small L 2 -norm, the softmax loss tend to suppress it.
Revised: Property 2 [Feature Norm Matters for Mis-classiﬁed Examples] For mis-classiﬁed feature vector f i with small L 2 -norm, the softmax loss tend to suppress it.

Original: [Figure]
Revised: [Figure]

Original: Proof.
Revised: Proof.

Original: According to deﬁnition of softmax loss in
Revised: According to deﬁnition of softmax loss in

Original: Eq.(3), the gradient with respect to weight vector w j ( j = 1 , · · · 
Revised: Eq.(3), the gradient with respect to weight vector w j ( j = 1 , · · · , K ):

Original: [Equation]
Revised: [Equation]

Original: K
Revised: N

Original: [Equation]
Revised: [Equation]

Revised: (7) where P ij =

Revised: e w Tj

Original: [Equation]
Revised: [Equation]

Original: where the h ( i ) is an indicator function, and h ( i ) =when y i 
Revised: Kk =1 e w Tk f i , the h ( i ) is an indicator function, and h ( i ) = 1 when y i =

Original: = j otherwise h ( i ) = 0 .
Revised: j otherwise h ( i ) = 0 .

Original: When (cid:107) f
Revised: When (cid:107) f

Original: i (cid:107) is small, the gradient ∂ L softmax ∂ w j contributed by f i also tend to be small.
Revised: i (cid:107) is small, the gradient ∂ L softmax ∂ w j contributed by f i also tend to be small.

Revised: Especially, we investigate the distribution of features’ norm over correctly-classiﬁed and mis-classiﬁed examples on CIFAR10 in the Figure3, which indicates that mis-classiﬁed examples tend to be of small feature norm.

Original: Thus the gradients contributed by the mis-classiﬁed features with small L 2 -norm will be suppressed.
Revised: Thus the gradients contributed by these mis-classiﬁed features with small L 2 -norm will be suppressed.

Original: [Equation]
Revised: (cid:3)

Original: According to Property 2 , it is necessary to increase the feature norm of mis-classiﬁed examples, especially the ones with small feature norm.
Revised: According to Property 2 , it is necessary to increase the feature norm of mis-classiﬁed examples, especially the ones with small feature norm.

Original: By optimizing the feature norm of mis-classiﬁed vectors,the errors can be ﬁxed as illustrated in the ﬁfth column of
Revised: By optimizing the feature norm of mis-classiﬁed vectors,

Original: Table 5.
Revised: W

Original: [Figure]
Revised: [Figure]

Original: W 
Revised: W the errors can be ﬁxed as illustrated in the ﬁfth column of Table 5. (e.g., 336 examples from the test set of CIFAR10 become correctly-classiﬁed by increasing their average feature norm from 169.3 to 195.9)

Original: The feature norm can be optimized by tuning the weight parameters from the previous layers, increasing the feature norm without inﬂuencing the magnitude of weights parameters from the previous layers is our target, which will be discussed in the supplementary details.
Revised: The feature norm can be optimized by tuning the weight parameters from the previous layers, increasing the feature norm without inﬂuencing the magnitude of weights parameters from the previous layers is our target, which will be discussed in the supplementary details.

Original: The proposed
Revised: The proposed

Original: Reciprocal Norm Loss will be discussed in next subsection to further explain how feature incay works.
Revised: Reciprocal Norm Loss will be discussed in next subsection to further explain how feature incay works.

Original: 3. R ECIPROCAL N ORM L OSS
Revised: 3. R ECIPROCAL N ORM L OSS

Original: The superiorities of increasing feature norm have been investigated in the previous subsection.
Revised: The superiorities of increasing feature norm have been investigated in the previous subsection.

Original: Here we explore several methods that increase the feature norm end-to-end by penalizing an additional term, such as (cid:107) f (cid:107) 2 , log( (cid:107) f (cid:107) 2 ) and − 1 (cid:107) f (cid:107) 2 .
Revised: Here we explore several methods that increase the feature norm end-to-end by penalizing an additional term, such as (cid:107) f (cid:107) 2 , log ( (cid:107) f (cid:107) 2 ) and − 1 (cid:107) f (cid:107) 2 .

Original: The comparison analysis is provided in supplementary.
Revised: The comparison analysis is provided in supplementary.

Original: Speciﬁcally, we choose − 1 (cid:107) f (cid:107) 2 and propose the Reciprocal Norm Loss, where the deﬁnition of Reciprocal Norm Loss is,
Revised: Speciﬁcally, we choose − 1 (cid:107) f (cid:107) 2 and propose the Reciprocal Norm Loss, where the deﬁnition of Reciprocal Norm Loss is,

Original: [Equation]
Revised: [Equation]

Original: (cid:107) w k (cid:107)(cid:124) (cid:123)(cid:122) (cid:125) weight decay + λ 1 N
Revised: (8) 𝜃 𝛼 𝛽 𝑑 1 𝑑𝑊𝑊𝑊𝑊𝑊 𝐾

Original: N (cid:88) i =  (cid:107)
Revised: [Figure]

Original: f i (cid:107) 2 + (cid:15) (cid:124) (cid:123)(cid:122) (cid:125) feature incay (8) 𝜃 𝛼 𝛽 𝑑 1 𝑑𝑊𝑊𝑊𝑊𝑊 𝐾 …… where (cid:15) is a small positive value to prevent dividing by value close to zero, λ is a hyper-parameter used to control the inﬂuence of feature incay.
Revised: where (cid:15) is a small positive value to prevent dividing by value close to zero, λ is a hyper-parameter used to control the inﬂuence of feature incay.

Original: The whole loss function consists of three parts: softmax loss, weight decay and feature incay, of which the later two items prefer small weight norm and large feature norm.
Revised: The whole loss function consists of three parts: softmax loss, weight decay and feature incay, of which the later two items prefer small weight norm and large feature norm.

Original: Feature incay can be considered during the whole training procedure or after some training iterations with softmax loss.
Revised: Feature incay can be considered during the whole training procedure or after some training iterations with softmax loss.

Original: Large feature norm brings large inter-class separability under the constraint of small weight norm, otherwise large feature norm can be trivially achieved by increasing the weight norm.
Revised: Large feature norm brings large inter-class separability under the constraint of small weight norm, otherwise large feature norm can be trivially achieved by increasing the weight norm.

Original: The simple reciprocal form of feature norm has an important property that moves feature vectors with small norm faster along weight vectors than feature vectors with large norm, which results intra-class compactness.
Revised: The simple reciprocal form of feature norm has an important property that moves feature vectors with small norm faster along weight vectors than feature vectors with large norm and results intra-class compactness, which is illustrated in Figure 4.

Original: Speciﬁcally, denote feature incay of f i as F ( f i ) = 1 (cid:107)
Revised: Speciﬁcally, denote feature incay of f i as F ( f i ) = 1 (cid:107)

Original: f i (cid:107) 2 + (cid:15) , the gradient is ∂ F ∂ f i = − [-
-]
Revised: f i (cid:107) 2 + (cid:15) , the gradient is ∂ F ∂ f i = − [-
-]2 f [Equation]

Original: f i ( (cid:107) f
Revised: f [Equation]

Original: i (cid:107) 2 + (cid:15) ) 2 , for any two feature vectors f p and f q satisfying (cid:107) f q (cid:107) > (cid:107) f p (cid:107) , we always have (cid:107) ∂ F ∂ f p (cid:107) > (cid:107) ∂ F ∂ f q (cid:107) .
Revised: 2 + [Equation] For any two feature vectors f p and f q satisfying (cid:107) f q (cid:107) > (cid:107) f p (cid:107) , we always have (cid:107) ∂ F ∂ f p (cid:107) > (cid:107) ∂ F ∂ f q (cid:107) .

Revised: Thus vectors with small norm increase fast along their original directions while feature vectors with large norm increase slowly along their original directions.

Original: 3. G EOMETRIC I NTERPRETATION OF R ECIPROCAL N ORM L OSS
Revised: 3. G EOMETRIC I NTERPRETATION OF R ECIPROCAL N ORM L OSS

Original: As stated in  previous subsection, Reciprocal Norm Loss can increase feature norm adaptively to decrease the intra-class variance.
Revised: As stated in the previous subsection, Reciprocal Norm Loss can increase feature norm adaptively to decrease the intra-class variance.

Original: Here we prove that there exists an upper bound for feature norm.
Revised: Here we prove that there exists an upper bound for feature norm.

Original: Property 3 [Feature Norm Bound]Suppose that (a) the angle between any feature vector f i and its corresponding weight vector w y
Revised: Property 3 [Feature Norm Bound]

Original:  i is zero, (b) the angles between any two neighbor weight vectors of different classes are θ , we have (1) the minimal inter-class distance is 2 α sin( θ 2 ) , where r is lower bound of feature norm, (2)  when K < 2 D , the upper boundof feature norm is in the range of[Table] [(1 + √ 2) α, 3 α ] to ensure the maximal intra-class distance is smaller than the minimal inter-class distance.
Revised: Given (a) the angle between any feature vector f i and its corresponding weight vector w y i is zero, (b) the angles between any two neighbor weight vectors of different classes are θ , we have (1) the minimal inter-class distance is 2 α sin( θ 2 ) , where α is lower bound of feature norm, (2) to ensure the maximal intra-class distance is smaller than the minimal inter-class distance, the upper bound of feature norm is 3 α , especially when K < 2 D , the upper bound in the range of [(1 + √ 2) α, 3 α ] .

Original: Proof.
Revised: Proof.

Original: Figure 5 shows the 2-dimensional case satisfying the (a) and (b), where black arrows named with W i represent weight vectors for each class.
Revised: Figure 5 shows the 2-dimensional case satisfying the (a) and (b), where black arrows named with W i represent weight vectors for each class.

Original: As we have assumed that all feature vectors are lying on the directions of their corresponding W i , blue circle and purple circle denote the lower bound and upper bound of feature norm respectively.
Revised: As we have assumed that all feature vectors are lying on the directions of their corresponding W i , blue circle and purple circle denote the lower bound and upper bound of feature norm respectively.

Original: Thus the maximal intra-class distance is d 2 = β − α and the minimal inter-class distance is d 1 = 2 α sin( θ 2 ) .
Revised: Thus the maximal intra-class distance is d 2 = β − α and the minimal inter-class distance is d 1 = 2 α sin( θ 2 ) .

Original: To ensure minimal inter-class distance is larger than intra-class distance, i
Revised: To ensure minimal inter-class distance is larger than intra-class distance, i.e., d 1 = 2 α sin( θ 2 ) >

Original: [Table] e., d 1 = 2 α sin( θ 2 ) > d 2 = β − α , which requires β < 2 α sin( θ 2 ) + α  . 
Revised: d 2 = β − α , which requires β < 2 α sin( θ 2 ) + α (cid:54) 3 α . Thus 3 α is the general upper bound for the feature norm.

Original: When K < 2 D , according to the Lemma (refer to supplementary), we can ensure that θ ≥ 90 ◦ .
Revised: Especially when K < 2 D ( D is the features’ dimension), according to the Lemma (refer to supplementary), we can ensure that θ ≥ 90 ◦ .

Original: Besides, the angle between any two vectors is smaller than 180 ◦ .
Revised: Besides, the angle between any two vectors is smaller than 180 ◦ .

Original: Then θ 2 ∈ [45 ◦ , 90 ◦ ] and sin( θ 2 ) is a monotonously increasing function within the range [45 ◦ , 90 ◦ ] .
Revised: Then θ 2 ∈ [45 ◦ , 90 ◦ ] and sin( θ 2 ) is a monotonously increasing function within the range [45 ◦ , 90 ◦ ] .

Original: Based on sin( θ 2 ) ∈ [ √ 22 , 1] , the upper bound of L 2 -norm of feature vectors is in the range [(1 + √ 2) α, 3 α ] .
Revised: Based on sin( θ 2 ) ∈ [ √ 22 , 1] , the upper bound of L 2 -norm of feature vectors is in the range [(1 + √ 2) α, 3 α ] .

Original: (cid:3)
Revised: [Equation]

Original: According to the Property 3 , We can estimate the upper bound of the feature norm based on the original features’ L 2 -norm.
Revised: According to the Property 3 , We can estimate the upper bound of the feature norm based on the original features’ L 2 -norm.

Original: In our experiments, we choose the average L 2 -norm as the lower bound to avoid the inﬂuence of outliers.
Revised: In our experiments, we choose the average L 2 -norm as the lower bound to avoid the inﬂuence of outliers.

Original: For example, if the average L 2 -norm on CIFAR10 is 200, we will choose 483 ≈
Revised: For example, if the average L 2 -norm on CIFAR10 is 200, we will choose 483 ≈

Original: (1 + √ 2) × 200 as the threshold to control the feature incay.
Revised: (1 + √ 2) × 200 as the threshold to control the feature incay.

Original: The feature incay for features with feature norm exceeding 483 will be set as 0.
Revised: The feature incay for features with feature norm exceeding 483 will be set as 0.

Original: In summary, feature norm matters and softmax loss can beniﬁt from the proposed feature incay.
Revised: In summary, feature norm matters and softmax loss can beniﬁt from the proposed feature incay.

Revised:  E XPERIMENTS

Revised: In this section, we verify the effectiveness of feature incay through empirical experiments.

Original: E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
Revised: 4. E XPERIMENTAL S ETTINGS

Original: We evaluate feature incay on four datasets, i.e., MNIST, CIFAR10, CIFAR100 and LFW.
Revised: We evaluate feature incay on four datasets, i.e., MNIST, CIFAR10, CIFAR100 and LFW.

Original: MNIST consists of 60,000 training images and 10,000 test images from 10 handwritten digits, both CIFAR and CIFAR100 contain 50,000 training images and 10,000 test images from 10 object categories and 100 object categories respectively.
Revised: MNIST consists of 60,000 training images and 10,000 test images from 10 handwritten digits, both CIFAR and CIFAR100 contain 50,000 training images and 10,000 test images from 10 object categories and 100 object categories respectively.

Original: LFW(Huang et al.
Revised: LFW(Huang et al.

Original: dataset contains 13,233 face images from 5749 different identities, 6000 face pairs are used as test set following the standard protocol.
Revised: dataset contains 13,233 face images from 5749 different identities, 6000 face pairs are used as test set following the standard protocol.

Original: Images are subtracted by mean image  and randomly ﬂipped horizontally for data augmentation.
Revised: Images are subtracted by mean image Images and randomly ﬂipped horizontally for data augmentation.

Original: The speciﬁc network architectures are detailed in supplementary.
Revised: The speciﬁc network architectures are detailed in supplementary.

Original: We adopt Caffe framework(Jia et al.
Revised: We adopt Caffe framework(Jia et al.

Original: for training and testing.
Revised: for training and testing.

Original: The weight µ for weight decay isset as 0.0005 in all experiments.
Revised: The weight µ for weight decay is 0.0005 in all experiments.

Original: We choose different weight λ in different experiments, i.e., 1.0, 0.1 or 0.01.
Revised: We choose different weight λ in different experiments, i.e., 1.0, 0.1 or 0.01.

Original: The momentum is 0.9, and the learning rate starts from 0.1 and is divided by a factor of 10 three times when the training error stops decreasing.
Revised: The momentum is 0.9, and the learning rate starts from 0.1 and is divided by a factor of 10 three times when the training error stops decreasing.

Original: 4. C OMPARISON E XPERIMENTS
Revised: 4. C OMPARISON E XPERIMENTS

Revised: Feature incay is added to Sofmax, L-Softmax and A-Softmax to compare with state-of-the-art approaches, which is represented with RN(Reciprocal Norm loss) plus the baseline method.

Revised: e.g., RN +

Original: Feature incayis added to Sofmax, L-Softmax and A-Softmax to compare with state-of-the-art approaches, and the reproduced results by Softmax, L-Softmax and A-Softmax following Liu et al.
Revised: Softmax means combining the feature incay with Softmax loss. The reproduced results by Softmax, L-Softmax and A-Softmax following Liu et al.

Original: (2016; 2017a) are the same as or slightly better than the referred numbers in the original works.
Revised: (2016; 2017a) are the same as or slightly better than the referred numbers in general.

Original: Table 2 reports the error rates of compared approaches and our method on MNIST, CIFAR10 and [Table]
Revised: Table 2 reports the error rates of compared approaches and our method on MNIST, CIFAR10 and CIFAR100.

Original: [Table] CIFAR100. It can be concluded that feature incay can consistently improve over Softmax and LSoftmax on CIFAR10/CIFAR100.
Revised: It can be concluded that feature incay can consistently improve over Softmax and L-Softmax on CIFAR10/CIFAR100.

Original: For example, RN decreases the error rate of Softmax from 8.59% to 7.84% while L-Softmax achieves 7.56% on CIFAR10, which demonstrates both of them are better than Softmax.
Revised: For example, RN decreases the error rate of Softmax from 8.59% to 7.84% while L-Softmax achieves 7.56% on CIFAR10, which demonstrates both of them are better than Softmax.

Original: By combining RN and L-Sofmax, we  achieve better result 7.22%, which means that they are complementary.
Revised: By combining RN and L-Sofmax, we [Table] [Table] achieve better result 7.22%, which means that they are reciprocal.

Original: However, RN + L-Softmax is slightly worse than L-Softmax on MNIST, our hypothesis is that the performance on MNIST is already saturate and difﬁcult to improve further.
Revised: However, RN + L-Softmax is slightly worse than L-Softmax on MNIST, our hypothesis is that the performance on MNIST is already saturate and difﬁcult to improve further.

Original: To further verify our method’s effectiveness on more challenging datasets, we test RN +  L-Softmax and RN + A-Softmax on LFW and achieve competitive performance with SphereNet-20 and SphereNet-64.
Revised: To further verify our method’s effectiveness on more challenging datasets, we test RN + COCO, RN + L-Softmax and RN + A-Softmax on LFW and achieve competitive performance with SphereNet20 or SphereNet-64.

Original: The results are illustrated in Table 1.
Revised: The results are illustrated in Table 1.

Original: The reproduced results with L-Softmax and A-Softmax are comparable.
Revised: The reproduced results with L-Softmax and A-Softmax are comparable while the reproduced COCO result is not as good due to both the training dataset and network structure are set different.

Original: With feature incay, RN + L-Softmax improves the L-Softmax from 99.03% to 99.18% and RN + A-Softmax improves the A-Softmax from 99.42% to 99.47%.
Revised: With feature incay, RN + L-Softmax improves the L-Softmax from 99.03% to 99.18%, RN + A-Softmax improves the A-Softmax from 99.42% to 99.47% and RN + COCO improves the COCO from 98.90% to 99.02%.

Original: Thus feature incay can even promote the A-Softmax(Liu et al.
Revised: Thus feature incay can even promote both A-Softmax(Liu et al.

Original: (2017a))  with normalized features by elongating the features before normalization.
Revised: (2017a)) and COCO with normalized features by elongating the features before normalization.

Original: 4. E FFECTS OF λ .
Revised: 4. E FFECTS OF λ .

Original: Here we conduct experiments on CIFAR10 to investigate the inﬂuence of hyper-parameter λ .
Revised: Here we conduct experiments on CIFAR10 to investigate the inﬂuence of hyper-parameter λ .

Original: Results are illustrated in Table 3.
Revised: Results are illustrated in Table 3.

Original: Both RN + Softmax and RN + L-Softmax achieve consistent improvement for all different λ except on RN + L-Softmax when λ = 1 , which is caused by that the loss item of
Revised: Both RN + Softmax and RN + L-Softmax achieve consistent improvement for all different λ except on RN + L-Softmax when λ = 1 , which is caused by that the loss item of

Original: L-Softmax can be smaller than the loss item of feature incay.
Revised: L-Softmax can be smaller than the loss item of feature incay.

Original: To balance the L-Softmax and feature incay for training, λ should be set to a relatively small weight. 
Revised: To balance the L-Softmax and feature incay for training, λ should be set to a relatively small weight. [Table] [Table] [Table]

Revised: [Table]

Original: 4. S IMPLY S CALE THE F EATURE
Revised: 4. S IMPLY S CALE THE F EATURE

Original: To verify whether it is possible to improve the performance by simply rescaling the features before computing softmax loss, we conduct extensive experiments on CIFAR10 and present the related results in Table 4.
Revised: To verify whether it is possible to improve the performance by simply rescaling the features before computing softmax loss [Table] we conduct extensive experiments on CIFAR10 and present the related results in Table [Table]

Original: Simply scaling the features fails to improve the performance, where too large value can cause the network fails to converge.
Revised: Simply scaling the features fails to improve the performance [Table] where too large value can cause the network fails to converge.

Original: For example, when we scale the features more than 10 times, the softmax loss will explode during training. [Table] [Table]
Revised: For example, when we scale the features more than 10 times, the softmax loss will explode during training

Original: 4. R ESULT A NALYSIS
Revised: 4. R ESULT A NALYSIS

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h o r s e s h i p t r u c k

Original: [Figure]
Revised: [Figure]

Revised: t e s t a cc u r a c y ( % ) 94.3 96.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  By analyzing the features’ L 2 -norm and classiﬁcation accuracy on each class of CIFAR10, we want to investigate where the  improvements come from, and we ﬁnd 336 examples that are mis-classiﬁed by Softmax but correctly classiﬁed by RN + Softmax.
Revised: [Figure] [Figure] By analyzing the features’ L 2 -norm and classiﬁcation accuracy on CIFAR10 for data of each class, we want to investigate where the concrete improvements come from. Table 5 reports the related details and we ﬁnd 336 examples that are mis-classiﬁed by Softmax but correctly classiﬁed by RN + Softmax.

Original: However, 259 mis-classiﬁed examples are furhter introduced by RN + Softmax, which limits the ﬁnal performance improvement.
Revised: However, 259 mis-classiﬁed examples are furhter introduced by RN + Softmax, which limits the ﬁnal performance improvement.

Original: We also plot the histograms of average L 2 -norm and accuracy for Softmax and Softmax + RN.
Revised: We also plot the histograms of average L 2 -norm and accuracy for Softmax and Softmax + RN.

Original: The details are illustrated in Figure 6.
Revised: The details are illustrated in Figure 6.

Original: With feature incay, the feature norm is enlarged and the accuracy is on all ten classes.
Revised: With feature incay, the feature norm is enlarged and the accuracy is on all ten classes.

Original:  C ONCLUSIONS AND FUTURE WORK
Revised:  C ONCLUSIONS AND FUTURE WORK

Original: In this paper, we propose the feature incay implemented as Reciprocal Norm Loss to increase the feature norm.
Revised: In this paper, we propose the feature incay implemented as Reciprocal Norm Loss to increase the feature norm.

Original: Based on the theoretical analysis of the feature norm, the Reciprocal Norm Loss induces smaller training loss and forces the modelto focus on hard examplesby managing the feature norm of both the correctly-classiﬁed and mis-classiﬁed feature vectors. Extensive experiments on
Revised: Based on the theoretical analysis of the feature norm, the Reciprocal Norm Loss induces smaller training loss focuses model examples the feature norm of both the correctly-classiﬁed and mis-classiﬁed feature vectors

Original: MNIST, CIFAR10, CIFAR100 and LFW verify the effectiveness of our method.
Revised: MNIST CIFAR CIFAR100 and LFW verify the effectiveness of our method

Original:  S UPPLEMENTARY 6. LEMMA
Revised:  S UPPLEMENTARY 6. LEMMA

Original: Here is the Lemma proposed by Ranjan et al.
Revised: Here is the Lemma proposed by Ranjan et al

Original: and is used in the proof of Property 3 and
Revised: and is used in the proof of Property 3 and

Original: Property 4 .
Revised: Property 4 .

Original: Lemma When the number of classes K is smaller than twice the feature dimension D, we can distribute the classes on a hypersphere of dimension D such that any two class weight vectors are at least 90 ◦ apart.
Revised: Lemma When the number of classes K is smaller than twice the feature dimension D we can distribute the classes on a hypersphere of dimension D such that any two class weight vectors are at least 90 ◦ apart.

Original: 6. F EATURE NORM WITHIN S OFTMAX LOSS
Revised: 6. F EATURE NORM WITHIN S OFTMAX LOSS

Original: Property 

Revised: Property

Original: [Feature Norm in Softmax Loss]
Revised: [Feature Norm in Softmax Loss]

Original: For any feature vector f
Revised: For any feature vector f

Original: i , if P iy i → 1 and P ij → 0( ∀ j (cid:54) =
Revised: i , if P iy i → 1 and P ij → 0( ∀ j (cid:54) =

Original: y i ) , then ∂ L softmax ∂ f i → 0 .
Revised: y i ) , then ∂ L softmax ∂ f i → 0 .

Original: Proof.
Revised: Proof.

Original: According to deﬁnition of softmax loss in Eq.(3), the gradient of feature vector f i is:
Revised: According to deﬁnition of softmax loss in Eq.(3) the gradient of feature vector f i is:

Original: [Equation]
Revised: [Equation]

Original: (9) where the P ij =
Revised: (9) where the P ij =

Original: e w Tj f i (cid:80)
Revised: e w Tj f i (cid:80)

Original: Kk
Revised: Kk

Original: =1 e w
Revised: =1 e w

Original: Tk
Revised: Tk

Original: f
Revised: f

Original: i .
Revised: i .

Original: When P iy i → 1 and P ij → 0( ∀ j (cid:54) =
Revised: When P iy i → 1 and P ij → 0( ∀ j (cid:54) =

Original: y i ) , ∂ L softmax ∂ f i → 0 .
Revised: y i ) , ∂ L softmax ∂ f i → 0 .

Original: That is after a training sample is conﬁdently classiﬁed correctly, it will have no contribution to its own representation learning.
Revised: That is after a training sample is conﬁdently classiﬁed correctly, it will have no contribution to its own representation learning.

Original: For example, suppose (cid:107) w j (cid:107) =
Revised: For example, suppose (cid:107) w j (cid:107) =

Original: 1 , θ w yi , f i = 0 and θ w j , f i ≥ 90 ◦ ( ∀ j (cid:54) =
Revised: 1 , θ w yi , f i = 0 and θ w j , f i ≥ 90 ◦ ( ∀ j (cid:54) =

Original: y i )
Revised: y i )

Original: according the Lemma , then (cid:107)
Revised: according the Lemma , then (cid:107)

Original: w y
Revised: w y

Original: i (cid:107)(cid:107) f i (cid:107) cos( θ w yi , f i ) = (cid:107) f i (cid:107) , (cid:107) w j (cid:107)(cid:107)f
Revised: i (cid:107)(cid:107) f i (cid:107) cos( θ w y i , f i ) = (cid:107) f i (cid:107) , (cid:107) w j (cid:107)(cid:107)

Original:  i (cid:107) cos( θ w j , f i ) < 0 , ∀ j (cid:54) =
Revised: f i (cid:107) cos( θ w j , f i ) < 0 , ∀ j (cid:54) =

Original: y i .
Revised: y i .

Original: Putting together, we have P iy i ≥
Revised: Putting together, we have P iy i ≥

Original: e (cid:107) f
Revised: [Equation]

Original: i (cid:107) e (cid:107) f
Revised: +

Original: i (cid:107) + K − 1 , for a modest number of categories say K = 10 ,
Revised: [Equation] 1 , for a modest number of categories say K = 10 ,

Original: P iy i > 0 .
Revised: P iy i > 0 .

Original: 999 when (cid:107) f
Revised: 999 when (cid:107) f

Original: i (cid:107) = 10 .
Revised: i (cid:107) = 10 .

Original: (cid:3)
Revised: (cid:3)

Original: 6. F UNCTIONAL FORMAT OF FEATURE INCAY
Revised: 6. F UNCTIONAL FORMAT OF FEATURE INCAY

Original: The choice of the functional format of feature incay is important.
Revised: The choice of the functional format of feature incay is important.

Original: Here we mainly analyze three different choices.
Revised: Here we mainly analyze three different choices.

Original: • Linear.
Revised: • Linear.

Original: F ( (cid:107) f (cid:107) 2 ) =
Revised: F ( (cid:107) f (cid:107) 2 ) =

Original: (cid:107) f(cid:107)
Revised: (cid:107) f

Original: [Equation]
Revised: (cid:107)∂ F ∂ f = 2 f • Log. F ( (cid:107) f (cid:107) 2 ) = log ( (cid:107) f (cid:107) 2 ) ∂ F ∂ f = 2 f (cid:107) f

Original: • Log.
Revised: (cid:107)• Reciprocal.

Original: F ( (cid:107) f (cid:107) 2 ) = log ( (cid:107) f (cid:107) 2 )
Revised: F ( (cid:107) f (cid:107) 2 ) = − 1 (cid:107) f (cid:107)∂ F ∂ f

Original: [Equation]
Revised: [Equation]

Original: • Reciprocal. [Equation]
Revised: 2 f (cid:107) f (cid:107) 2 (cid:107) f

Original: [Equation]
Revised: (cid:107)

Original: Here we mainly analyze the differences of the above three functions by investigating the relationship between the gradients and the original feature norm.
Revised: Here we mainly analyze the differences of the above three functions by investigating the relationship between the gradients and the original feature norm.

Original: Assuming that we have two features f 1 and fsatisfying (cid:107) f 2 (cid:107) > (cid:107) f 1 (cid:107) .
Revised: Assuming that we have two features f 1 and fsatisfying (cid:107) f 2 (cid:107) > (cid:107) f 1 (cid:107) .

Original: For the linear function case, we have (cid:107) ∂ F ∂ f 2 (cid:107) >
Revised: For the linear function case, we have (cid:107) ∂ F ∂ f 2 (cid:107) >

Original: (cid:107) ∂ F ∂ f 1 (cid:107) .
Revised: (cid:107) ∂ F ∂ f 1 (cid:107) .

Original: Then, the intraclass variance will be increased as the distance between f 1 and f 2 increases with each gradient  update.
Revised: Then, the intraclass variance will be increased as the distance between f 1 and f 2 increases with each gradient [Table] [Table] update.

Original: Besides, the gradients of the linear function have the same magnitude with the feature itself, such large gradients update can lead to explosion during the training.
Revised: Besides, the gradients of the linear function have the same magnitude with the feature itself, such large gradients update can lead to explosion during the training.

Original: For the log function case, we have ∂ F ∂ f = 2 f (cid:107) f (cid:107) 2 = 2 u (cid:107) f
Revised: For the log function case, we have ∂ F ∂ f = 2 f (cid:107) f (cid:107) 2 = 2 u (cid:107) f

Original: (cid:107) , where u is the unit vector with feature norm equals 1.
Revised: (cid:107) , where u is the unit vector with feature norm equals 1.

Original: Thus (cid:107) ∂ F ∂ f 1 (cid:107) > (cid:107) ∂ F ∂ f 2 (cid:107) .
Revised: Thus (cid:107) ∂ F ∂ f 1 (cid:107) > (cid:107) ∂ F ∂ f 2 (cid:107) .

Original: The gradients within reciprocal function is also that (cid:107) ∂ F ∂ f 1 (cid:107) > (cid:107) ∂ F ∂ f 2 (cid:107) always holds once (cid:107)
Revised: The gradients within reciprocal function is also that (cid:107) [Table] (cid:107) > (cid:107) [Table]

Revised: 2 (cid:107) always holds once (cid:107)

Original: f 2 (cid:107) > (cid:107) f 1 (cid:107) .
Revised: f 2 (cid:107) > (cid:107) f 1 (cid:107) .

Original: Although both log function and reciprocal function increase the features with small feature norm faster than the features with larger feature norm, we ﬁnd the performance of reciprocal function is better.
Revised: Although both log function and reciprocal function increase the features with small feature norm faster than the features with larger feature norm, we ﬁnd the performance of reciprocal function is better.

Original: In summary, reciprocal function can increase the overall feature norm and increase the intra-class similarity simultaneously, where the intra-class similarity along the direction of the weight vectors can be decreased with the log function.
Revised: In summary, reciprocal function can increase the overall feature norm and increase the intra-class similarity simultaneously, where the intra-class similarity along the direction of the weight vectors can be decreased with the log function.

Original: We choose the reciprocal function in all of our experiments.
Revised: We choose the reciprocal function in all of our experiments.

Original: [Table]

Original: Table 6 reports the classiﬁcation accuracies adopting the three different considered feature incay.
Revised: Table 6 reports the classiﬁcation accuracies adopting the three different considered feature incay.

Original: The superiority of Softmax + RN over Softmax + LN and Softmax + LogN is well illustrated, where
Revised: The superiority of Softmax + RN over Softmax + LN and Softmax + LogN is well illustrated, where

Original: Softmax + RN can achieve better intra-class similarity according to the above analysis.
Revised: Softmax + RN can achieve better intra-class similarity according to the above analysis.

Original: 6. CNN A RCHITECTURES S ETTINGS
Revised: 6. CNN A RCHITECTURES S ETTINGS

Original: For LFW
Revised: For LFW

Original: [Table]
Revised: [Table]

Original: we adopt 20-layer/64-layer SphereNet following the same settings in Liu et al. (2017a).
Revised: we adopt 20-layer/64-layer SphereNet following the same settings in Liu et al. (2017a).

Original: We modify the network settings for MNIST/CIFAR10/CIFAR100 based on the previous work(Liu et al.
Revised: We modify the network settings for MNIST/CIFAR10/CIFAR100 based on the previous work(Liu et al.

Original: and list them in Table 
Revised: and list them in Table 8.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 6. E FFECTS OF W EIGHT D ECAY
Revised: 6. E FFECTS OF W EIGHT D ECAY

Original: Here we also investigate the inﬂuence of the weigth decay on the classiﬁcation accuracy and feature norm, where we conduct experiments considering only Softmax loss for fairness.
Revised: Here we also investigate the inﬂuence of the weigth decay on the classiﬁcation accuracy and feature norm, where we conduct experiments considering only Softmax loss for fairness.

Original: The results are reported in Table 7, where we ﬁnd that it fails to improve neither accuracy nor feature norm by simply increasing the weight decay or decreasing the weight decay.
Revised: The results are reported in Table [Table] where we ﬁnd that it fails to improve neither accuracy nor feature norm by simply increasing the weight decay or decreasing the weight decay.

Original: Thus simply changing the weight decay leads to either underﬁtting or overﬁtting.
Revised: Thus simply changing the weight decay leads to either underﬁtting or overﬁtting.

Original: 6. E FFECTS ON W EIGHTS ’ D ISTRIBUTION
Revised: 6. E FFECTS ON W EIGHTS ’ D ISTRIBUTION

Original: To avoid overﬁtting, we consider weight decay in all experiments.
Revised: To avoid overﬁtting, we consider weight decay in all experiments.

Original: However, one main concern is the side effect of feature incay may increase the magnitude of the shallow layers.
Revised: However, one main concern is the side effect of feature incay may increase the magnitude of the shallow layers.

Original: Here we plot the weights’ distributions of initial state, difference choices of weight decay within Softmax and RN +
Revised: Here we plot the weights’ distributions of initial state, difference choices of weight decay within Softmax and RN +

Original: Softmax.
Revised: Softmax.

Original: We ﬁnd that the inﬂuence of feature incay is limited due to the constraint from weight decay.
Revised: We ﬁnd that the inﬂuence of feature incay is limited due to the constraint from weight decay.

Original: In summary, the feature incay enlarges the feature norm without harming the magnitude of weights from previous layers and will not lead to overﬁtting.
Revised: In summary, the feature incay enlarges the feature norm without harming the magnitude of weights from previous layers and will not lead to overﬁtting.

Original: Besides, we can also observe that different weight decay has big impact on the ﬁnal weight distribution, such as larger weigth decay(e.g., µ = 0 .
Revised: Besides, we can also observe that different weight decay has big impact on the ﬁnal weight distribution, such as larger weigth decay(e.g., µ = 0 .

Original: 005 ) results in more weights are constrained to zero, which may lead to underﬁtting according to their ﬁnal classiﬁcation performances.
Revised: 005 ) results in more weights are constrained to zero, which may lead to underﬁtting according to their ﬁnal classiﬁcation performances.

Original: 6. E XPERIMENTAL A NALYSIS
Revised: 6. E XPERIMENTAL A NALYSIS

Original: Figure 8(a), 8(b) and 8(c) show the accuracy, average feature norm and softmax loss during training by using λ = 1 , 0 . 1 , 0 . 01 respectively on CIFAR10.
Revised: Figure 8(a), 8(b) and 8(c) show the accuracy, average feature norm and softmax loss during training by using λ = 1 , 0 . 1 , 0 . 01 respectively on CIFAR10.

Original: Feature incay achieves better or comparable accuracy compared with softmax loss under a wide range of λ , and results in larger feature norm on both training and test set.
Revised: Feature incay achieves better or comparable accuracy compared with softmax loss under a wide range of λ , and results in larger feature norm on both training and test set.

Original: All methods achieve close to zero softmax loss on training set, while feature incay ensures lower softmax loss on test set.
Revised: All methods achieve close to zero softmax loss on training set, while feature incay ensures lower softmax loss on test set.

Original: Figure 8(d) shows the accuracy vs iteration number on CIFAR100, which is similar to the results on CIFAR10.
Revised: Figure 8(d) shows the accuracy vs iteration number on CIFAR100, which is similar to the results on CIFAR10.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]