Original: F EATURE I NCAY FOR R EPRESENTATION R EGULARIZATION
Revised: F EATURE I NCAY FOR R EPRESENTATION R EGULARIZATION

Original: Anonymous authors Paper under double-blind review
Revised: Yang DeepMotion deepmotion

Original: A BSTRACT
Revised: A BSTRACT

Original: Softmax-based loss is widely used in deep learning for multi-class classiﬁcation, where each class is represented by a weight vector and each sample is represented as a feature vector.
Revised: Softmax-based loss is widely used in deep learning for multi-class classiﬁcation, where each class is represented by a weight vector and each sample is represented as a feature vector.

Original: Different from traditional learning algorithms where features are pre-deﬁned and only weight vectors are tunable through training, feature vectors are also tunableas representation learning in deep learning.Thus we investigate how to improve the classiﬁcation performance by better adjusting the features.
Revised: Inspired by that weight decay is a common practice to regularize the weight vectors, we investigate how to regularize the feature vectors since representation is also tunable in deep learning.

Original: One main observation is that elongating the feature norm of both correctly-classiﬁed and mis-classiﬁed feature vectors improves learning: (1) increasing the feature norm of correctly-classiﬁed examples induce smaller training loss; (2) increasing the feature norm of mis-classiﬁed examples can upweight the contribution from hard examples.
Revised: One main observation is that elongating the feature norm of both correctlyclassiﬁed and mis-classiﬁed feature vectors improves learning: (1) increasing the feature norm of correctly-classiﬁed examples enlarges the probability margin among different classes and ensures better generalization. (2) increasing the feature norm of mis-classiﬁed examples can up-weight the contribution from hard examples.

Original: Accordingly, we propose feature incay to regularize representation learning by encouraging larger feature norm.
Revised: Accordingly, we propose feature incay to regularize feature vectors by encouraging larger feature norm.

Original: In contrast to weight decay which shrinks the weight norm, feature incay is proposed to stretch the feature norm.

Original: Extensive empirical results on MNIST, CIFAR10, CIFAR and LFW demonstrate the effectiveness of feature incay.
Revised: Extensive empirical results on MNIST, CIFAR10, CIFAR100 and LFW demonstrate the effectiveness of feature incay.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]

Original: Deep Neural Networks (DNNs) with softmax-based loss have achieved stateof-the-art performance on numerous multi-class classiﬁcation related tasks.
Revised: Deep Neural Networks (DNNs) with softmax-based loss have achieved state-of-the-art performance on numerous multi-class classiﬁcation related tasks.

Original: In DNNs, both representations and classiﬁers are learned within a uniﬁed network concurrently, where the ﬁnal representation for a sample is the feature vector f outputted from the penultimate layer, while the last layer outputs scores
Revised: In DNNs, both representations and classiﬁers are learned within a uniﬁed network concurrently, where the ﬁnal representation for a sample is the feature vector f outputted from the penultimate layer, while the last layer outputs scores

Original: z i = w i · f for each category i , where w i is the weight vector for category i .
Revised: z i = w i · f for each category i , where w i is the weight vector for category i .

Original: Before deﬁning the loss, the scores for each category are normalized into probability via softmax function, i.e., p i = ezi (cid:80) j ezj .
Revised: Before deﬁning the loss, the scores for each category are normalized into probability via softmax function, i.e., p i = ezi (cid:80) j ezj .

Original: A well-trained DNN should output signiﬁcant larger probability for the correct label than other labels, which requires the score for the correct label is significantly larger than other labels.
Revised: A well-trained DNN should output signiﬁcant larger probability for the correct label than other labels, which requires the score for the correct label is signiﬁcantly larger than other labels.

Original: Since z i = w i · f = (cid:107) w i (cid:107)(cid:107) f (cid:107) cos( θ ) , where θ is the angle between w i and f , the goal of signiﬁcant larger score for the correct label than other labels can be achieved by tuning (cid:107) w i (cid:107) , (cid:107) f (cid:107) and θ .
Revised: Since z i = w i · f = (cid:107) w i (cid:107)(cid:107) f (cid:107) cos( θ ) , where θ is the angle between w i and f , the goal of signiﬁcant larger score for the correct label than other labels can be achieved by tuning (cid:107) w i (cid:107) , (cid:107) f (cid:107) and θ .

Original: While increasing the weight norm (cid:107) w i (cid:107) is constrained by weight decay for regularization, thus (cid:107) f (cid:107) and θ become the two main factors for optimization.
Revised: While increasing the weight norm (cid:107) w i (cid:107) is constrained by weight decay for regularization, thus (cid:107) f (cid:107) and θ become the two main factors for optimization.

Original: Although softmax loss can tune both of them, there  still exist wide room to improve either or both factors.[Figure]
Revised: Although softmax loss can tune both of them, there is still much room to improve either or both factors.

Original: For example, Liu et al.
Revised: Recently, Liu et al.

Original: (2016), Wang et al.
Revised: (2016), Wang et al.

Original: (2017) and Liu et al.
Revised: (2017) and Liu et al.

Original: (2017a) propose different approaches to further optimize the factor of angular θ , and all of them have achieved obviously better performance.
Revised: (2017a) propose different approaches to further optimize the factor of angular θ , and all of them have achieved obviously better performance.

Original: To further emphasize the factor of angular, Ranjan et al.
Revised: To further emphasize the factor of angular, Ranjan et al.

Original: (2017) and Liu et al.
Revised: (2017) and Liu et al.

Original: (2017c) propose to use normalized feature vectors for softmax loss where the factor of feature norm is totally ignored.
Revised: (2017c) propose to use normalized feature vectors for softmax loss where the factor of feature norm is totally ignored.

Original: In this work, we make the effort to optimize the feature norm.
Revised: In this work, we make the effort to optimize the feature norm.

Original: Firstly, we analyze the connections between the feature norm and the classiﬁcation accuracy within softmax loss, they are highly correlated as illustrated in Figure 1.
Revised: Firstly, we analyze the connections between the feature norm and the classiﬁcation accuracy within softmax loss, they are highly correlated as illustrated in Figure 1.

Original: Features with larger norm tend to be correctly classiﬁed with higher probability.
Revised: Features with larger norm tend to be correctly classiﬁed with higher probability.

Revised: [Figure]

Revised: [Figure]

Original: Here we propose to optimize the feature norm by augmenting the softmax loss with feature incay.
Revised: Here we propose to optimize the feature norm by augmenting the softmax loss with feature incay.

Original: In contrast to weight decay that shrinks weight vectors to be of small norm, feature incay tends to stretch out the feature vectors.
Revised: In contrast to weight decay that shrinks weight vectors to be of small norm, feature incay tends to stretch out the feature vectors.

Original: From the computational perspective, larger feature norm results in larger score differences among categories, which can better separate the categories.
Revised: From the computational perspective, larger feature norm results in larger score differences among categories, which can better separate the categories.

Original: From the perspective of pattern detection, larger feature norm encourages model to learn and detect more prominent patterns.
Revised: From the perspective of pattern detection, larger feature norm encourages model to learn and detect more prominent patterns.

Original: Figure 2(a), 2(b) and 2(c) show the results of comparison experiments by adding feature incay to softmax loss, where feature incay achieves larger feature norm, smaller loss value and higher accuracy on test set.
Revised: Figure 2(a), 2(b) and 2(c) show the results of comparison experiments by adding feature incay to softmax loss, where feature incay achieves larger feature norm, smaller loss value and higher accuracy on test set.

Original: The geometric interpretation of feature incay is illustrated in Figure 2(d), 2(e) and 2(f), we can achieve the largest inter-class separability by explicitly optimizing both the (cid:107) f (cid:107) and θ compared with the other methods.
Revised: The geometric interpretation of feature incay is illustrated in Figure 2(d), 2(e) and 2(f), we can achieve the larger inter-class separability by explicitly optimizing both the (cid:107) f (cid:107) and θ compared with the other methods.

Original: Besides, the proposed feature incay (implemented as
Revised: Besides, the proposed feature incay (implemented as

Original: Reciprocal Norm Loss) is designed to increase the feature norm adaptively according to the original feature norm, which can also help reduce the intra-class variances as illustrated in Figure 5.
Revised: Reciprocal Norm Loss) is designed to increase the feature norm adaptively according to the original feature norm, which can also help reduce the intra-class variances as discussed in section 3.4.

Original: In summary, we analyze the effect of feature norm and prove that (1) increasing feature norm for correctly-classiﬁed examples induce smaller training loss.
Revised: Our main contributions are summarized as follows:

Original: (2) increasing  feature norm for misclassiﬁed examplescan up-weight the contribution of hard examples. (3) the bound on feature norm to ensure the inter-class separability and intra-class compactness.
Revised: • The beneﬁts of increasing the feature norm for both correctly-classiﬁed examples and misclassiﬁed examples are analyzed theoretically.

Original: The proposed feature incay is veriﬁed on four widely used classiﬁcation datasets(i.e.
Revised: • We verify the effectiveness of feature incay empirically on four widely used classiﬁcation datasets(i

Original: , MNIST, CIFAR10, CIFAR100 and LFW) using various network architecture.
Revised: e., MNIST, CIFAR10, CIFAR100 and LFW) using various network architectures.

Original: By considering the feature incay, we achieve comparable performances on all of them.
Revised: With feature incay, we achieve consistent performance improvements on all of them.

Revised: • We also conduct experiments to study the weights’ distribution w/ feature incay and w/o feature incay and investigate where the improvements come from.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Large-margin Softmax Loss.
Revised: Large-margin Softmax Loss.

Original: Liu et al.
Revised: Liu et al.

Original: (2016) proposed to improve softmax loss by incorporating an adjustable margin m multiplying the angle between a feature vector and the corresponding weight vector.
Revised: (2016) proposed to improve softmax loss by incorporating an adjustable margin m multiplying the angle between a feature vector and the corresponding weight vector.

Original: Compared with the softmax loss, it pays more attention to the angular decision margin between classes as illustrated in Figure 2(e).
Revised: Compared with the softmax loss, it pays more attention to the angular decision margin between classes as illustrated in Figure 2(e).

Original: Large-margin softmax loss appends stronger constraint to the angular, while feature incay considers constraint to feature norm.
Revised: Large-margin softmax loss appends stronger constraint to the angular, while feature incay considers constraint to feature norm.

Original: As illustrated in Figure 2(f), feature incay is orthogonal to large-margin softmax loss.
Revised: As illustrated in Figure 2(f), feature incay is orthogonal to large-margin softmax loss.

Original: Center Loss.
Revised: Center Loss.

Original: Wen et al.
Revised: Wen et al.

Original: (2016) presented the center loss to learn centers for deep features of each class and penalize the distances between the deep features and their corresponding class centers.
Revised: (2016) presented the center loss to learn centers for deep features of each class and penalize the distances between the deep features and their corresponding class centers.

Original: The softmax loss tries to align feature vectors close to the weight vectors based on the inner product similarity, while center loss pushes feature vectors towards their class centers according Euclidean distances.

Original: Combining softmax loss with center loss actually uses two sets of classiﬁers, where representation is learned based on both the inner product to weight vector and the Euclidean distance to class center.
Revised: Combining softmax loss with center loss actually uses two sets of classiﬁers, where representation is learned based on both the inner product to weight vector and the Euclidean distance to class center.

Original: The added center loss helps minimize the intra-class distances also by inﬂuencing the feature norm, namely, small feature norm will be increased and large feature norm will be decreased during the process of pushing feature vectors to class centers.
Revised: The added center loss helps minimize the intra-class distances also by inﬂuencing the feature norm, namely, small feature norm will be increased and large feature norm will be decreased during the process of pushing feature vectors to class centers.

Original: Different from center loss, feature incay also increases the large feature norm instead of penalizing feature vectors with large norm as center loss.
Revised: Different from center loss, feature incay also increases the large feature norm instead of penalizing feature vectors with large norm as center loss.

Original: Weight/Feature Normalization.
Revised: Weight/Feature Normalization.

Original: Inspired by the fact that feature normalization before calculating the sample distances usually achieves better performance for retrieval tasks, Ranjan et al.
Revised: Inspired by the fact that feature normalization before calculating the sample distances usually achieves better performance for retrieval tasks, Ranjan et al.

Original: (2017) proposed to use normalized feature vectors in softmax loss during training, thus the feature norm has no effect on softmax loss and angle is the main factor to be optimized.
Revised: (2017) proposed to use normalized feature vectors in softmax loss during training, thus the feature norm has no effect on softmax loss and angle is the main factor to be optimized.

Original: Congenerous cosine loss(Liu et al.
Revised: Congenerous cosine loss(Liu et al.

Original: (2017b)),
Revised: (2017b)),

Original: NormFace(Wang et al.
Revised: NormFace(Wang et al.

Original: (2017)), and cosine normalization(Chunjie et al.
Revised: (2017)), and cosine normalization(Chunjie et al.

Original: (2017)) take a step further to normalize the weight vectors which replace inner product with cosine similarity within softmax loss, and only optimize the factor of angle. 
Revised: (2017)) take a step further to normalize the weight vectors which replace inner product with cosine similarity within softmax loss, and only optimize the factor of angle. Recently, Wang et al. (2018b), Wang et al.

Revised: (2018a)

Original:  Although normalization mechanism achieves much lower intra-class angular variability by emphasizing more on the angle during training, they ignore that feature norm is another useful factor worth to optimize.
Revised: and Deng et al. (2018) all proposed to add the angle margin after normalizing both the weight vectors and feature vectors during nearly the same period. Although normalization mechanism achieves much lower intra-class angular variability by emphasizing more on the angle during training, they ignore that feature norm is another useful factor worth to optimize.

Original: Feature Scale.
Revised: Feature Scale.

Original: COCO(Liu et al.
Revised: COCO(Liu et al.

Original: (2017c))
Revised: (2017c))

Original: and L 2 -softmax(Ranjan et al.
Revised: and L 2 -softmax(Ranjan et al.

Original: (2017)) are the most relevant to our work, and especially COCO is a concurrent work.
Revised: (2017)) are the most relevant to our work, and especially COCO is a concurrent work.

Original:  Both of the them introduce a single scaling parameter to increase the magnitude of the features. However, they all enforce the L 2 -norm of the features to be ﬁxed for all samples.
Revised: Here we compare our method with COCO theoretically, Relations : Both COCO and feature incay increase the feature norm, which is the common reason for the performance improvement.

Original: Speciﬁcally, they are optimizing the feature vectors on a hypersphere  with ﬁxed radius.
Revised: Differences : COCO is optimizing the feature embedding on a hypersphere while feature incay optimizes feature embedding located between two hyperspheres with different radiuses(Property 3 in section 3.4).

Original: Different from them, we are trying to investigate whether it is possible to optimize the original feature space instead of the “hypersphere” feature space.
Revised: COCO proposes a novel con- generous cosine loss while feature incay uses the original softmax loss. Feature incay is simpler than COCO and it can be easily plugged into almost all the related work that use softmax loss. Empirically comparision is provided in section 4.2.

Original:  O UR WORK 3. R EVISITING SOFTMAX LOSS
Revised:  O UR WORK 3. R EVISITING SOFTMAX LOSS

Original: Let X = { ( x i , y i ) } Ni =1 be the training set contains N samples, where x i is the raw input to the DNN, y i ∈ { 1 , 2 , · · · , K } is the class label that supervises the output of the DNN.
Revised: Let X = { ( x i , y i ) } Ni =1 be the training set contains N samples, where x i is the raw input to the DNN, y i ∈ { 1 , 2 , · · · , K } is the class label that supervises the output of the DNN.

Original: Denote f i as the feature vector for x i learned by the DNN, { w j } Kj =1 represent weight vectors for the K categories.
Revised: Denote f i as the feature vector for x i learned by the DNN, { w j } Kj =1 represent weight vectors for the K categories.

Original: Then, softmax loss is deﬁned as,
Revised: Then, softmax loss is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: Bias terms are ignored following the discussions in recent works Liu et al.
Revised: Bias terms are ignored following the discussions in recent works Liu et al.

Original: (2016) and Wang et al.
Revised: (2016) and Wang et al.

Original: Denote the angle between w j and f i as θ w j , f i , the inner product between w j and f i can be rewritten as
Revised: Denote the angle between w j and f i as θ w j , f i , the inner product between w j and f i can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: By combining the above two equations, we get
Revised: By combining the above two equations, we get

Original: [Equation]
Revised: [Equation]

Original: 3. F EATURE NORM MATTERS
Revised: 3. F EATURE NORM MATTERS

Original: Here we will illustrate how feature norm inﬂuences the softmax loss from two aspects: (1)increasing the feature norm of correctly-classiﬁed examples can induce smaller training loss.
Revised: Here we will illustrate how feature norm inﬂuences the softmax loss from two aspects: (1) increasing the feature norm of correctly-classiﬁed examples enlarges the probability margin among different categories.

Original: (2)increasing the feature norm of mis-classiﬁed examples can up-weight the contribution from the hard examples.
Revised: (2) increasing the feature norm of mis-classiﬁed examples can up-weight the gradients of weight vectors, especially the weight vectors of the correct category and mis-classiﬁed category.

Original: The ﬁrst property is similar to the proposition proved by Wang et al.
Revised: The ﬁrst property is similar to the proposition proved by Wang et al.

Original: (2017), which states that softmax loss always encourages features of the correctly-classiﬁed examples to have larger magnitudes.
Revised: (2017), which states that softmax loss always encourages features of the correctly-classiﬁed examples to have larger magnitudes.

Original: Property 1 [Feature Norm Matters for Correctly-classiﬁed Examples] Suppose weight vectors and directions of the feature vectors are ﬁxed, increasing of the feature norm of correctly-classiﬁed examples can decrease the softmax loss.
Revised: Property 1 [Feature Norm Matters for Correctly-classiﬁed Examples] Suppose weight vectors and directions of the feature vectors are ﬁxed, increasing of the feature norm of correctly-classiﬁed examples can enlarge the probability margin among different categories.

Original: Proof.
Revised: Proof.

Original: Let L softmax ( f i ) represent the loss of the i -th sample, i = 1 , · · · , N . Speciﬁcally,
Revised: Assuming f i is correctly-classiﬁed into category

Original: [Equation] Recall that when f i is correctly classiﬁed, we have w Ty i f i > w Tj f i for any  j (cid:54) =
Revised: y i , here we deﬁne the probability margin between category y i and any other category j ( j (cid:54) =

Original: y i, and (cid:107) w j (cid:107)(cid:107) f i (cid:107) cos( θ w j , f i ) − (cid:107) w y i
Revised: y i ) as:

Original: [Equation]
Revised: [Equation]

Original:  f i [Equation]
Revised: Recall that when f i is correctly classiﬁed, we have w Ty i f i >

Original: cos [Equation] θ w yi
Revised: w Tj f i for j (cid:54) =

Original: [Equation] f i ) ≤ 0 always holds.
Revised: y i .

Original: Then, for any t > 0 , we have
Revised: Then, for any t > 0 , we have,

Original: [Equation]
Revised: [Equation]

Original: [-which means increasing the norm of correctly classiﬁed samples can decrease the softmax loss. To consider all samples including incorrectly classiﬁed ones, we set t i > 0 if i is correctly classiﬁed and t i = 
 otherwise, then we have-]
Revised: [-which means increasing the norm of correctly classiﬁed samples can decrease the softmax loss. To consider all samples including incorrectly classiﬁed ones, we set t i > 0 if i is correctly classiﬁed and t i = 
 otherwise, then we have-][Equation]

Original: [Equation]
Revised: [Figure]

Original: So feature norm is an important factor toachieve smaller softmax loss together with the angle.
Revised: According to the above inequations, we have

Original: [Equation]
Revised: [Equation]

Original: Though the decrease in softmax loss is marginal for some samples already with small softmax loss, theincreased feature norm enlarges the marginamong different categories which ensures better generalization.
Revised: In summary, the probability margin increases by increasing the feature norm and larger probability margin ensures better generalization.

Original: Property 2 [Feature Norm Matters for Mis-classiﬁed Examples] For mis-classiﬁed feature vector f i with small L 2 -norm, the softmax loss tend to suppress it.
Revised: Property 2 [Feature Norm Matters for Mis-classiﬁed Examples] Assuming the feature vector f i is mis-classiﬁed into category k

Original: [Figure]
Revised: ( k (cid:54) = y i ), increasing the feature norm of f i can up-weight the gradients of the weight vector w k .

Original: Proof.
Revised: Proof.

Original: According to deﬁnition of softmax loss in Eq.(3), the gradient with respect to weight vector w j ( j = 1 , · · · , K ):
Revised: According to deﬁnition of softmax loss in Eq.(3), the gradient with respect to weight vector w j ( j = 1 , · · · , K ) is:

Original: [Equation]
Revised: [Equation]

Original: N [Equation] (7) where P ij = e w Tj 
Revised: where P ij = e w Tj f i (cid:80)

Original: [Equation]

Original: Kk =1 e w
Revised: Kk =1 e w

Original: Tk f
Revised: Tk f

Original: i , the h ( i ) is an indicator function,and h ( i ) = 1 when{+
 if+} y i =
Revised: i , and h ( i ) is an indicator function, h ( i ) = {+
 if+} y i =

Original: j otherwise h ( i ) = 0 .
Revised: j otherwise h ( i ) = 0 .

Revised: Speciﬁcally, we analyze how the value of ∂ L softmax ∂ w k changes, which contains two factors: (1) ( P ik − h

Original: When (cid:107) f
Revised: ( i )) increases with larger (cid:107) f i (cid:107) when f i is mis-classiﬁed as category k . By increasing (cid:107) f

Original: i (cid:107) is small, the gradient ∂ L softmax ∂ w j contributed by f i also tend to be small.
Revised: i (cid:107) we will always have larger P ik and h ( i ) = 0 when computing w k . So the absolute value of ( P ik − h ( i )) is increased. (2) (cid:107) f

Revised: i (cid:107) is increased.

Original: Especially, we investigate the distribution of features’ norm over correctly-classiﬁed and mis-classiﬁed examples on CIFAR10 in the Figure3, which indicates that mis-classiﬁed examples tend to be of small feature norm.
Revised: So increasing the feature norm (cid:107) f i (cid:107) can up-weight the gradients for w k . Especially for hard examples f i with P ik close to 1 and P iy

Original: Thus the gradients contributed by these mis-classiﬁed features with small L 2 -norm will be suppressed.
Revised: i close to 0, ( P iy i − h ( i )) is closer to -1 by increasing (cid:107) f i (cid:107) , thus increasing (cid:107) f i (cid:107) can also increase the gradients for weight vector w y i .

Original: (cid:3)
Revised: (cid:3)

Original: According to Property 2 , it is necessary to increase the  feature norm of mis-classiﬁed examples, especially the ones with small feature norm. By optimizing the feature norm of  mis-classiﬁed vectors,
Revised: We investigate the distribution of feature norm over correctly-classiﬁed and mis-classiﬁed examples on CIFAR10 in Figure3, and the gradients contributed by the mis-classiﬁed features is suppressed due to most of the mis-classiﬁed examples are of small feature norm.

Original: W
Revised: Thus it is necessary to increase the feature norm of mis-classiﬁed examples.

Original: [Figure] W the errors can be ﬁxed as illustrated in the ﬁfth column of Table 5.
Revised: By optimizing the feature norm of mis-classiﬁed vectors, the errors can be ﬁxed as illustrated in the ﬁfth column of Table 6.

Original: (e.g., 336 examples from the test set of CIFAR10 become correctly-classiﬁed by increasing their average feature norm from 169.3 to 195.9)
Revised: (e.g., 336 examples from the test set of CIFAR10 become correctly-classiﬁed by increasing their average feature norm from 169.3 to 195.9)

Original: The feature norm can be optimized by tuning the weight parameters from the previous layers, increasing the feature normwithout inﬂuencing the magnitude ofweights parameters from the previous layers is our target, which will be discussed in thesupplementary details. The proposed Reciprocal Norm Loss will be discussed in next subsection to further explain how feature incay works.
Revised: Besides, the proposed feature incay increases the originally small feature norm faster than the originally large feature norm. Thus the feature norm of the mis-classiﬁed examples will increase faster than the feature norm of the correctly-classiﬁed examples, which can further up-weight the contribution of the hard examples.

Original: 3. R ECIPROCAL N ORM L OSS
Revised: 3. R ECIPROCAL N ORM L OSS

Original: The superiorities of increasing feature norm have been investigated in the previous subsection.
Revised: The superiorities of increasing feature norm have been investigated in the previous subsection.

Original: Here we explore several methods that increase the feature norm end-to-end by penalizing an additional term, such as (cid:107) f (cid:107) 2 ,  log ( (cid:107) f (cid:107) 2 ) and− 1 (cid:107) f (cid:107) 2 .
Revised: Here we explore several methods that increase the feature norm end-to-end by penalizing an additional term, such as −(cid:107) f (cid:107) 2 , − log ( (cid:107) f (cid:107) 2 ) and 1 (cid:107) f (cid:107) 2 .

Original: The comparison analysis is provided in supplementary.
Revised: The comparison analysis is provided in supplementary.

Original: Speciﬁcally, we choose− 1 (cid:107) f (cid:107) 2 and propose the Reciprocal Norm Loss, where the deﬁnition of Reciprocal Norm Loss is,
Revised: Speciﬁcally, we choose 1 (cid:107) f (cid:107) 2 and propose the Reciprocal Norm Loss, where the deﬁnition of [Figure]

Original: [Equation]
Revised: [Figure]

Original: (8) 𝜃 𝛼 𝛽 𝑑 1 𝑑𝑊𝑊𝑊𝑊𝑊 𝐾
Revised: Reciprocal Norm Loss is,

Original: [Figure]
Revised: [Equation]

Original: where (cid:15) is a small positive value to prevent dividing by value close to zero, λ is a hyper-parameter used to control the inﬂuence of feature incay.
Revised: where (cid:15) is a small positive value to prevent dividing by value close to zero, λ is a hyper-parameter used to control the inﬂuence of feature incay.

Original: The whole loss function consists of three parts: softmax loss, weight decay and feature incay, of which the later two items prefer small weight norm and large feature norm.
Revised: The whole loss function consists of three parts: softmax loss, weight decay and feature incay, of which the later two items prefer small weight norm and large feature norm.

Original: Feature incay can be considered during the whole training procedure or after some training iterations with softmax loss.
Revised: Feature incay can be considered during the whole training procedure or after some training iterations with softmax loss.

Original: Large feature norm brings large inter-class separability under the constraint of small weight norm, otherwise large feature norm can be trivially achieved by increasing the weight norm.
Revised: Large feature norm brings large inter-class separability under the constraint of small weight norm, otherwise large feature norm can be trivially achieved by increasing the weight norm.

Original: The simple reciprocal form of feature norm has an important property that moves feature vectors with small norm faster along weight vectors than feature vectors with large norm and results intra-class compactness, which is illustrated in Figure 4.
Revised: The simple reciprocal form of feature norm has an important property that moves feature vectors with small norm faster along weight vectors than feature vectors with large norm and results intra-class compactness, which is illustrated in Figure 4.

Original: Speciﬁcally, denote feature incay of f i as F ( f i ) = 1 (cid:107)
Revised: Speciﬁcally, denote feature incay of f i as F ( f i ) = [Equation]

Original: f i (cid:107) 2 + (cid:15) , the gradient is ∂ F ∂ f i = − 2 f
Revised: , the gradient is [Equation] = − [Equation]

Original: [Equation]
Revised: [Equation]

Original:  f 
Revised: For any two feature vectors f p and f q satisfying (cid:107) f q (cid:107) > (cid:107) f p (cid:107) , we always have (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 2 +
Revised: F

Original: [Equation]
Revised: [Equation]

Original: For any two feature vectors f p and f q satisfying (cid:107) f q (cid:107) > (cid:107) f p (cid:107) , we always have (cid:107) ∂ F ∂ f p (cid:107) > (cid:107) ∂ F ∂ f q (cid:107) .
Revised: f p (cid:107) > (cid:107) ∂ F ∂ f q (cid:107) .

Original: Thus vectors with small norm increase fast along their original directions while feature vectors with large norm increase slowly along their original directions. 
Revised: Thus vectors with small norm increase fast along their original directions while feature vectors with large norm increase slowly along their original directions. [Table]

Original: 3. G EOMETRIC I NTERPRETATION OF R ECIPROCAL N ORM L OSS
Revised: 3. G EOMETRIC I NTERPRETATION OF R ECIPROCAL N ORM L OSS

Original: As stated in the previous subsection, Reciprocal Norm Loss can increase feature norm adaptively to decrease the intra-class variance.
Revised: As stated in the previous subsection [Table] Reciprocal Norm Loss can increase feature norm adaptively to decrease the intra-class variance.

Original: Here we prove that there exists an upper bound for  feature norm.
Revised: Here we prove that there exists an upper bound for the ratio between the feature norm’s upper bound and the feature norm’s lower bound.

Original: Property 3 [Feature Norm Bound]
Revised: Property 3 [Feature Norm Bound]

Original: Given (a) the angle between any feature vector f i and its corresponding weight vector w y i is zero, (b) the angles between any two neighbor weight vectors of different classes are θ , we have (1) the minimal inter-class distance is 2 α sin( θ 2 ) , where α is lower bound of feature norm, (2) to ensure the maximal intra-class distance is smaller than the minimal inter-class distance, the upper bound of feature norm is 3 α , especially when K < 2 D , the upper bound in the range of [(1 + √ 2) α, 3 α ] .
Revised: Given (a) the angle between any feature vector f i and its corresponding weight vector w y i is zero, (b) the angles between any two neighbor weight vectors of different classes are θ , we have (1) the minimal inter-class distance is 2 α sin( θ 2 ) , where α is lower bound of feature norm, (2) to ensure the maximal intra-class distance is smaller than the minimal inter-class distance, the upper bound of feature norm is 3 α , especially when K < 2 D ( K is the number of classes and D is the features’ dimension), the upper bound in the range of [(1 + √ 2) α, 3 α ] .

Original: Proof.
Revised: Proof.

Original: Figure 5 shows the 2-dimensional case satisfying the (a) and (b), where black arrows named with W i represent weight vectors for each class.
Revised: Figure 5 shows the 2-dimensional case satisfying conditions (a) and (b), where black arrows named with W i represent weight vectors for each class.

Original: As we have assumed that all feature vectors are lying on the directions of their corresponding W i , blue circle and purple circle denote the lower bound and upper bound of feature norm respectively.
Revised: As we have assumed that all feature vectors are lying on the directions of their corresponding W i , blue circle and purple circle denote the lower bound and upper bound of feature norm respectively.

Original: Thus the maximal intra-class distance is d 2 = β − α and the minimal inter-class distance is d 1 = 2 α sin( θ 2 ) .
Revised: Thus the maximal intra-class distance is d 2 = β − α and the minimal inter-class distance is d 1 = 2 α sin( θ 2 ) .

Original: To ensure minimal inter-class distance is larger than intra-class distance, i.e., d 1 = 2 α sin( θ 2 ) >
Revised: To ensure minimal inter-class distance is larger than intra-class distance, i.e., d 1 = 2 α sin( θ 2 ) >

Original: d 2 = β − α , which requires β < 2 α sin( θ 2 ) + α (cid:54) 3 α .
Revised: d 2 = β − α , which requires β < 2 α sin( θ 2 ) + α (cid:54) 3 α .

Original: Thus 3 α is the general upper bound for the feature norm.
Revised: Thus 3 α is the general upper bound for the feature norm.

Original: Especially when K < 2 D ( D is the features’ dimension), according to the Lemma (refer to supplementary), we can ensure that θ ≥ 90 ◦ .
Revised: Especially when K < 2 D , according to the Lemma (refer to supplementary), we can ensure that θ ≥ 90 ◦ .

Original: Besides, the angle between any two vectors is smaller than 180 ◦ .
Revised: Besides, the angle between any two vectors is smaller than 180 ◦ .

Original: Then θ 2 ∈ [45 ◦ , 90 ◦ ] and sin( θ 2 ) is a monotonously increasing function within the range [45 ◦ , 90 ◦ ] .
Revised: Then θ 2 ∈ [45 ◦ , 90 ◦ ] and sin( θ 2 ) is a monotonously increasing function within the range [45 ◦ , 90 ◦ ] .

Original: Based on sin( θ 2 ) ∈ [ √ 22 , 1] , the upper bound of L 2 -norm of feature vectors is in the range [(1 + √ 2) α, 3 α ] .
Revised: Based on sin( θ 2 ) ∈ [ √ 2 2 , 1] , the upper bound of L 2 -norm of feature vectors is in the range [(1 + √ 2) α, 3 α ] .

Original: [Equation]
Revised: (cid:3)

Original: According to the Property 3 ,  We can estimate the upper bound of the feature norm based on the original features’ L 2 -norm.
Revised: According to the Property 3 , the ratio of the upper bound β and the lower bound α is bounded( βα ≤ 3 ). We can estimate the upper bound of the feature norm based on the original features’ L 2 -norm.

Original: In our experiments, we choose the average L 2 -norm as the lower bound to avoid the inﬂuence of outliers.
Revised: In our experiments, we choose the average L 2 -norm as the lower bound to avoid the inﬂuence of outliers.

Original: For example, if the average L 2 -norm on CIFAR10 is 200, we will choose 483 ≈
Revised: For example, if the average L 2 -norm on CIFAR10 is 200, we will choose 483 ≈

Original: (1 + √ 2) × 200 as the threshold to control the feature incay.
Revised: (1 + √ 2) × 200 as the threshold to control the feature incay.

Original: The feature incay for features with feature norm exceeding 483 will be set as 0.
Revised: The feature incay for features with feature norm exceeding 483 will be set as 0.

Original: In summary, feature norm matters and softmax loss can beniﬁt from the proposed feature incay.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we verify the effectiveness of feature incay through empirical experiments.
Revised: In this section, we verify the effectiveness of feature incay through empirical experiments.

Original: 4. E XPERIMENTAL S ETTINGS
Revised: 4. E XPERIMENTAL S ETTINGS

Original: We evaluate feature incay on four datasets, i.e., MNIST, CIFAR10, CIFAR100 and LFW.
Revised: We evaluate feature incay on four datasets, i.e., MNIST, CIFAR10, CIFAR100 and LFW.

Original: MNIST consists of 60,000 training images and 10,000 test images from 10 handwritten digits, both CIFAR  and CIFAR100 contain 50,000 training images and 10,000 test images from 10 object categories and 100 object categories respectively.
Revised: MNIST consists of 60,000 training images and 10,000 test images from 10 handwritten digits, both CIFAR [Table] [Table] and CIFAR100 contain 50,000 training images and 10,000 test images from 10 object categories and 100 object categories respectively.

Original: LFW(Huang et al.
Revised: LFW(Huang et al.

Original: dataset contains 13,233 face images from 5749 different identities, 6000 face pairs are used as test set following the standard protocol.
Revised: dataset contains 13,233 face images from 5749 different identities, 6000 face pairs are used as test set following the standard protocol.

Original: Images are subtracted by mean image Images and randomly ﬂipped horizontally for data augmentation.
Revised: Images are subtracted by mean image Images and randomly ﬂipped horizontally for data augmentation.

Original: The speciﬁc network architectures are detailed in supplementary.
Revised: The speciﬁc network architectures are detailed in supplementary.

Original: We adopt Caffe framework(Jia et al.
Revised: We adopt Caffe framework(Jia et al.

Original: for training and testing.
Revised: for training and testing.

Original: The weight µ for weight decay is 0.0005 in all experiments.
Revised: The weight µ for weight decay is 0.0005 in all experiments.

Original: We choose different weight λ in different experiments, i.e., 1.0, 0.1 or 0.01.
Revised: We choose different weight λ in different experiments, i.e., 1.0, 0.1 or 0.01.

Original: The momentum is 0.9, and the learning rate starts from 0.1 and is divided by a factor of 10 three times when the training error stops decreasing.
Revised: The momentum is 0.9, and the learning rate starts from 0.1 and is divided by a factor of 10 three times when the training error stops decreasing.

Original: 4. C OMPARISON E XPERIMENTS
Revised: 4. C OMPARISON E XPERIMENTS

Original: Feature incay is added to Sofmax, L-Softmax and A-Softmax to compare with state-of-the-art approaches, which is represented with RN(Reciprocal Norm loss) plus the baseline method.
Revised: Feature incay is added to Sofmax, L-Softmax and A-Softmax to compare with state-of-the-art approaches, which is represented with RN(Reciprocal Norm loss) plus the baseline method.

Original: e.g., RN +
Revised: e.g., RN +

Original: Softmax means combining the feature incay with Softmax loss.
Revised: Softmax means combining the feature incay with Softmax loss.

Original: The reproduced results by Softmax,
Revised: The reproduced results by Softmax,

Original: L-Softmax and A-Softmax following Liu et al.
Revised: L-Softmax and A-Softmax following Liu et al.

Original: (2016; 2017a) are the same as or slightly better than the referred numbers in general.
Revised: (2016; 2017a) are the same as or slightly better than the referred numbers in general.

Original: Table 2 reports the error rates of compared approaches and our method on MNIST, CIFAR10 and CIFAR100.
Revised: Table 1 reports the error rates of compared approaches and our method on MNIST, CIFAR10 and CIFAR100.

Original: It can be concluded that feature incay can consistently improve over Softmax and L-Softmax on CIFAR10/CIFAR100.
Revised: It can be concluded that feature incay can consistently improve over Softmax and L-Softmax on CIFAR10/CIFAR100.

Original: For example, RN decreases the error rate of Softmax from 8.59% to 7.84% while L-Softmax achieves 7.56% on CIFAR10, which demonstrates both of them are better than Softmax.
Revised: For example, RN decreases the error rate of Softmax from 8.59% to 7.84% while L-Softmax achieves 7.56% on CIFAR10, which demonstrates both of them are better than Softmax.

Original: By combining RN and L-Sofmax, we[Table] [Table] achieve better result 7.22%, which means that they are reciprocal.
Revised: By combining RN and L-Sofmax, we achieve better result 7.22%, which means that they are reciprocal.

Original: However, RN + L-Softmax is slightly worse than L-Softmax on MNIST, our hypothesis is that the performance on MNIST is already saturate and difﬁcult to improve further.
Revised: However, RN + L-Softmax is slightly worse than L-Softmax on MNIST, our hypothesis is that the performance on MNIST is already saturate and difﬁcult to improve further.

Original: To further verify our method’s effectiveness on more challenging datasets, we test RN + COCO, RN + L-Softmax and RN + A-Softmax on LFW and achieve competitive performance with SphereNet20 or SphereNet-64.
Revised: To further verify our method’s effectiveness on more challenging datasets, we test RN + COCO, RN + L-Softmax and RN + A-Softmax on LFW and achieve competitive performance with SphereNet20 or SphereNet-64.

Original: The results are illustrated in Table 1.
Revised: The results are illustrated in Table 2.

Original: The reproduced results with L-Softmax and A-Softmax are comparable while the reproduced COCO result is not as good due to both the training dataset and network structure are set different.
Revised: The reproduced results with L-Softmax and A-Softmax are comparable while the reproduced COCO result is not as good due to both the training dataset and network structure are set different.

Original: With feature incay, RN + L-Softmax improves the L-Softmax from 99.03% to 99.18%, RN + A-Softmax improves the A-Softmax from 99.42% to 99.47% and RN + COCO improves the COCO from 98.90% to 99.02%.
Revised: With feature incay, RN + L-Softmax improves the L-Softmax from 99.03% to 99.18%, RN + A-Softmax improves the A-Softmax from 99.42% to 99.47% and RN + COCO improves the COCO from 98.90% to 99.02%.

Original: Thus feature incay can even promote both A-Softmax(Liu et al.
Revised: Thus feature incay can even promote both A-Softmax and COCO with normalized features by elongating the features before normalization.

Original: (2017a))
Revised: [Table]

Original: and COCO with normalized features by elongating the features before normalization.
Revised: [Figure] [Table] [Table]

Original: 4. E FFECTS OF λ.
Revised: E FFECTS OF λ

Original: Here we conduct experiments on CIFAR10 to investigate the inﬂuence of hyper-parameter λ .
Revised: Here we conduct experiments on CIFAR10 to investigate the inﬂuence of hyper-parameter λ .

Original: Results are illustrated in Table 3.
Revised: Results are illustrated in Table 3.

Original: Both RN + Softmax and RN + L-Softmax achieve consistent improvement for all different λ except on RN + L-Softmax when λ = 1 , which is caused by that the loss item of
Revised: Both RN + Softmax and RN + L-Softmax achieve consistent improvement for all different λ except on RN + L-Softmax when λ = 1 , which is caused by that the loss item of

Original: L-Softmax can be smaller than the loss item of feature incay.
Revised: L-Softmax can be smaller than the loss item of feature incay.

Original: To balance the L-Softmax and feature incay for training, λ should be set to a relatively small weight.
Revised: To balance the L-Softmax and feature incay for training, λ should be set to a relatively small weight.

Original: [Table] [Table] [Table]
Revised: To verify whether it is possible to improve the performance by simply rescaling the features before computing softmax loss, we conduct extensive experiments on CIFAR10 and present the related results in Table 4.

Original: [Table]
Revised: Simply scaling the features fails to improve the performance, where too large value can cause the network fails to converge. For example, when we scale the features more than 10 times, the softmax loss will explode during training(NAN represents the softmax loss is exploded during training).

Original: 4. S IMPLY S CALE THE F EATURE
Revised: 4. E FFECTS OF W EIGHT D ECAY

Original: To verify whether it is possible to improve the performance bysimply rescaling the features before computing softmax loss
Revised: Here we also investigate the inﬂuence of the weigth decay on the classiﬁcation accuracy and feature norm, where we conduct experiments considering only Softmax loss for fairness. The results are reported in Table 5, where we ﬁnd that it fails to improve neither accuracy nor feature norm by

Original: [Table]
Revised: [Table]

Original: we conduct extensive experiments on CIFAR10 and present the related results in Table

Original: [Table]
Revised: [Table]

Original: Simply scaling the features fails to improve the performance
Revised: RN +

Original: [Table]
Revised: [Table]

Original: where too large value can cause the network fails to converge.
Revised: simply increasing the weight decay or decreasing the weight decay. Thus simply changing the weight decay leads to either underﬁtting or overﬁtting. 4. E FFECTS ON W EIGHTS ’ D ISTRIBUTION

Original: For example, when we scale the features more than 10 times, the softmax loss will explode during training
Revised: To avoid overﬁtting, we consider weight decay in all experiments. However, one main concern is the side effect of feature incay may increase the magnitude of the shallow layers. We plot the weights’ distributions of initial state, difference choices of weight decay within Softmax and RN + Softmax in Figure 7(refer to supplementary for details of the network architectures settings).

Original: 4. R ESULT A NALYSIS
Revised: It is interesting to ﬁnd that the weights of the ﬁrst convolution layer(Conv0) becomes even sparser with feature incay. Overall, the inﬂuence of feature incay is limited due to the constraint from weight decay.

Original: [Figure] a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h o r s e s h i p t
Revised: So the feature incay enlarges the feature norm without harming the magnitude of weights from previous layers and will not lead to overﬁtting.

Original: r u c k [Figure] t e s t a cc u r a c y ( % ) 94.3 96.
Revised: Besides, we can also observe that different weight decay has big impact on the ﬁnal weight distribution, such as larger weigth decay(e.g., µ = 0 . 005 ) results in more weights are constrained to zero, which may lead to underﬁtting according to their ﬁnal classiﬁcation performances.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: More details are illustrated in Figure 7.

Original: [Figure]
Revised: 4. R ESULT A NALYSIS

Original: By analyzing the features’ L 2 -norm and classiﬁcation accuracy on CIFAR10 for data of each class, wewant to investigate where the concrete improvements come from.
Revised: By analyzing the features’ L 2 -norm and classiﬁcation accuracy on CIFAR10 for data of each class, we investigate where the concrete improvements come from.

Original: Table 5 reports the related details and we ﬁnd 336 examples that are mis-classiﬁed by Softmax but correctly classiﬁed by RN + Softmax.
Revised: Table 6 reports the related details and we ﬁnd 336 examples that are mis-classiﬁed by Softmax but correctly classiﬁed by RN + Softmax.

Original: However, 259 mis-classiﬁed examples are furhter introduced by RN + Softmax, which limits the ﬁnal performance improvement.
Revised: However, 261 mis-classiﬁed examples are further introduced by RN + Softmax, which limits the ﬁnal performance improvement.

Original: We also plot the histograms of average L 2 -norm and accuracy for Softmax and Softmax + RN.
Revised: We also plot the histograms of average L 2 -norm and accuracy for Softmax and Softmax + RN.

Original: The details are illustrated in Figure 6.
Revised: The details are illustrated in Figure 6.

Original: With feature incay, the feature norm is enlarged and the accuracy is on all ten classes.
Revised: With feature incay, the feature norm is enlarged and the accuracy is on all ten classes.

Original:  C ONCLUSIONSAND FUTURE WORK
Revised:  C ONCLUSIONS

Original: In this paper, we propose the feature incay implemented as Reciprocal Norm Loss to increase the feature norm.
Revised: In this paper, we propose the feature incay implemented as Reciprocal Norm Loss to increase the feature norm.

Original: Based on the theoretical analysis of the feature norm, the Reciprocal Norm Loss induces smaller training loss focuses model examples  the feature norm of both the correctly-classiﬁed and mis-classiﬁed feature vectors
Revised: Based on the theoretical analysis of the feature norm, the Reciprocal Norm Loss enlarges the probability margin among different categories and up-weight the contribution from hard examples by managing the feature norm of both the correctly-classiﬁed and mis-classiﬁed feature vectors.

Original: MNIST CIFAR CIFAR100 and LFW verify the effectiveness of our method
Revised: Extensive experiments on MNIST, CIFAR10, CIFAR100 and LFW verify the effectiveness of our method.

Original:  S UPPLEMENTARY6. LEMMA
Revised:  S UPPLEMENTARY

Revised: [Table] 6. LEMMA

Original: Here is the Lemma proposed by Ranjan et al
Revised: Here is the Lemma proposed by Ranjan et al.

Original: and is used in the proof of Property 3 and
Revised: and is used in the proof of Property 3 and

Original: Property 4 .
Revised: Property 4 .

Original: Lemma When the number of classes K is smaller than twice the feature dimension D  we can distribute the classes on a hypersphere of dimension D such that any two class weight vectors are at least 90 ◦ apart. 
Revised: Lemma When the number of classes K is smaller than twice the feature dimension D , we can distribute the classes on a hypersphere of dimension D such that any two class weight vectors are at least 90 ◦ apart. [Figure]

Original: 6. F EATURE NORM WITHIN S OFTMAX LOSS
Revised: 6. F EATURE NORM WITHIN S OFTMAX LOSS

Revised: Here we mainly want to present you that the feature norm of correctly classifed examples fails to be increased within softmax loss, while feature incay can increase the feature norm consistently.

Original: Property 

Revised: Property

Original: [Feature Norm in Softmax Loss]
Revised: [Feature Norm in Softmax Loss]

Original: For any feature vector f
Revised: For any feature vector f

Original: i , if P iy i → 1 and P ij → 0( ∀ j (cid:54) =
Revised: i , if P iy i → 1 and P ij → 0( ∀ j (cid:54) =

Original: y i ) , then ∂ L softmax ∂ f i → 0 .
Revised: y i ) , then ∂ L softmax ∂ f i → 0 .

Original: Proof.
Revised: Proof.

Original: According to deﬁnition of softmax loss in Eq.(3) the gradient of feature vector f i is:
Revised: According to deﬁnition of softmax loss in Eq.(3), the gradient of feature vector f i is:

Original: [Equation]
Revised: [Equation]

Original: (9) where the P ij = e w Tj f i (cid:80)
Revised: where the P ij = e w Tj f i (cid:80)

Original: Kk
Revised: Kk

Original: =1 e w
Revised: =1 e w

Original: Tk
Revised: Tk

Original: f
Revised: f

Original: i .
Revised: i .

Original: When P iy i → 1 and P ij → 0( ∀ j (cid:54) =
Revised: When P iy i → 1 and P ij → 0( ∀ j (cid:54) =

Original: y i ) , ∂ L softmax ∂ f i → 0 .
Revised: y i ) , ∂ L softmax ∂ f i → 0 .

Original: That is after a training sample is conﬁdently classiﬁed correctly, it will have no contribution to its own representation learning.
Revised: That is after a training sample is conﬁdently classiﬁed correctly, it will have no contribution to its own representation learning.

Original: For example, suppose (cid:107) w j (cid:107) =
Revised: For example, suppose (cid:107) w j (cid:107) =

Original: 1 , θ w yi , f i = 0 and θ w j , f i ≥ 90 ◦ ( ∀ j (cid:54) =
Revised: 1 , θ w yi , f i = 0 and θ w j , f i ≥ 90 ◦ ( ∀ j (cid:54) =

Original: y i )
Revised: y i )

Original: according the Lemma , then (cid:107)
Revised: according the Lemma , then (cid:107)

Original: w y
Revised: w y

Original: i (cid:107)(cid:107) f i (cid:107) cos( θ w y i , f i ) = (cid:107) f i (cid:107) , (cid:107) w j (cid:107)(cid:107) 
Revised: i (cid:107)(cid:107) f i (cid:107) cos( θ w yi , f i ) = (cid:107) f i (cid:107) , (cid:107) w j (cid:107)(cid:107) f

Original: f i (cid:107) cos( θ w j , f i ) < 0 , ∀ j (cid:54) =
Revised: i (cid:107) cos( θ w j , f i ) < 0 , ∀ j (cid:54) =

Original: y i .
Revised: y i .

Original: Putting together, we have P iy i ≥
Revised: Putting together, we have P iy i ≥

Original: [Equation]
Revised: e (cid:107) f

Original: +
Revised: i (cid:107) e (cid:107) f

Original: [Equation] 1 , for a modest number of categories say K = 10 ,
Revised: i (cid:107) + K − 1 , for a modest number of categories say K = 10 ,

Original: P iy i > 0 .
Revised: P iy i > 0 .

Original: 999 when (cid:107) f
Revised: 999 when (cid:107) f

Original: i (cid:107) = 10 .
Revised: i (cid:107) = 10 .

Original: (cid:3)
Revised: (cid:3)

Original: 6. F UNCTIONAL FORMAT OF FEATURE INCAY
Revised: 6. F UNCTIONAL FORMAT OF FEATURE INCAY

Original: The choice of the functional format of feature incay is important.
Revised: The choice of the functional format of feature incay is important.

Original: Here we mainly analyze three different choices.
Revised: Here we mainly analyze three different choices.

Original: • Linear.
Revised: [Table]

Revised: Here we mainly analyze the differences of the above three functions by investigating the relationship between the gradients and the original feature norm.

Original: F ( (cid:107) f  (cid:107)2 ) = (cid:107) f (cid:107)∂ F ∂ f= 2 f • Log. F ( (cid:107) f  (cid:107) 2 ) =
Revised: Assuming that we have two features f 1 and fsatisfying (cid:107) f 2 (cid:107) > (cid:107) f 1 (cid:107) .

Original: log ( (cid:107) f (cid:107) 2 ) ∂ F ∂ f = 2 f (cid:107) f (cid:107)• Reciprocal. F ( (cid:107) f (cid:107) 2 ) = − 1 (cid:107)f (cid:107)∂ F ∂ f
Revised: For the linear function case, we have (cid:107)

Original: [Equation]
Revised: [Table]

Original: 2 f (cid:107) f (cid:107) 2 (cid:107) f (cid:107)
Revised: (cid:107) > (cid:107)

Original: Here we mainly analyze the differences of the above three functions by investigating the relationship between the gradients and the original feature norm. Assuming that we have two features f 1 and fsatisfying (cid:107) f 2 (cid:107) > (cid:107) f 1 (cid:107) . For the linear function case, we have (cid:107) ∂ F ∂ f 2 (cid:107) >
Revised: [Table]

Original: (cid:107) ∂ F ∂ f 1 (cid:107) .
Revised: 1 (cid:107) .

Original: Then, the intraclass variance will be increased as the distance between f 1 and f 2 increases with each gradient[Table] [Table] update.
Revised: Then, the intraclass variance will be increased as the distance between f 1 and f 2 increases with each gradient update.

Original: Besides, the gradients of the linear function have the same magnitude with the feature itself, such large gradients update can lead to explosion during the training.
Revised: Besides, the gradients of the linear function have the same magnitude with the feature itself, such large gradients update can lead to explosion during the training.

Original: For the log function case, we have ∂ F ∂ f = 2 f (cid:107) f (cid:107) 2 = 2 u (cid:107) f
Revised: For the log function case, we have ∂ F ∂ f = 2 f (cid:107) f (cid:107) 2 = 2 u (cid:107) f

Original: (cid:107) , where u is the unit vector with feature norm equals 1.
Revised: (cid:107) , where u is the unit vector with feature norm equals 1.

Original: Thus (cid:107) ∂ F∂ f 1 (cid:107) > (cid:107) ∂ F ∂ f 2 (cid:107) . The gradients within reciprocal function is also that (cid:107)
Revised: Thus (cid:107) ∂ F

Original: [Table]
Revised: [Table]

Original: (cid:107) > (cid:107)
Revised: (cid:107) > (cid:107)

Original: [Table]
Revised: [Table]

Original:  2 (cid:107) always holds once (cid:107)
Revised: F [Table] 2 (cid:107) . The gradients within reciprocal function is also that (cid:107) ∂ F ∂ f 1 (cid:107) > (cid:107) ∂ F ∂ f 2 (cid:107) always holds once (cid:107)

Original: f 2 (cid:107) > (cid:107) f 1 (cid:107) .
Revised: f 2 (cid:107) > (cid:107) f 1 (cid:107) .

Original: Although both log function and reciprocal function increase the features with small feature norm faster than the features with larger feature norm, we ﬁnd the performance of reciprocal function is better.
Revised: Although both log function and reciprocal function increase the features with small feature norm faster than the features with larger feature norm, we ﬁnd the performance of reciprocal function is better.

Original: In summary, reciprocal function can increase the overall feature norm and increase the intra-class similarity simultaneously, where the intra-class similarity along the direction of the weight vectors can be decreased with the log function.
Revised: In summary, reciprocal function can increase the overall feature norm and increase the intra-class similarity simultaneously, where the intra-class similarity along the direction of the weight vectors can be decreased with the log function.

Original: We choose the reciprocal function in all of our experiments.
Revised: We choose the reciprocal function in all of our experiments.

Original: Table 6 reports the classiﬁcation accuracies adopting the three different considered feature incay.
Revised: Table 8 reports the classiﬁcation accuracies adopting the three different considered feature incay.

Original: The superiority of Softmax + RN over Softmax + LN and Softmax + LogN is well illustrated, where
Revised: The superiority of Softmax + RN over Softmax + LN and Softmax + LogN is well illustrated, where

Original: Softmax + RN can achieve better intra-class similarity according to the above analysis.
Revised: Softmax + RN can achieve better intra-class similarity according to the above analysis.

Original: 6. CNN A RCHITECTURES S ETTINGS
Revised: 6. CNN A RCHITECTURES S ETTINGS

Original: For LFW [Table] we adopt 20-layer/64-layer SphereNet following the same settings in Liu et al. (2017a).
Revised: For LFW, we adopt 20-layer/64-layer SphereNet following the same settings in Liu et al. (2017a).

Original: We modify the network settings for MNIST/CIFAR10/CIFAR100 based on the previous work(Liu et al.
Revised: We modify the network settings for MNIST/CIFAR10/CIFAR100 based on the previous work(Liu et al.

Original: and list them in Table 8.
Revised: and list them in Table 7.

Original: [Table]

Original: 6. E FFECTS OF W EIGHT D ECAY

Original: Here we also investigate the inﬂuence of the weigth decay on the classiﬁcation accuracy and feature norm, where we conduct experiments considering only Softmax loss for fairness.

Original: The results are reported in Table

Original: [Table]

Original: where we ﬁnd that it fails to improve neither accuracy nor feature norm by simply increasing the weight decay or decreasing the weight decay.

Original: Thus simply changing the weight decay leads to either underﬁtting or overﬁtting.

Original: 6. E FFECTS ON W EIGHTS ’ D ISTRIBUTION

Original: To avoid overﬁtting, we consider weight decay in all experiments.

Original: However, one main concern is the side effect of feature incay may increase the magnitude of the shallow layers.

Original: Here we plot the weights’ distributions of initial state, difference choices of weight decay within Softmax and RN +

Original: Softmax.

Original: We ﬁnd that the inﬂuence of feature incay is limited due to the constraint from weight decay.

Original: In summary, the feature incay enlarges the feature norm without harming the magnitude of weights from previous layers and will not lead to overﬁtting.

Original: Besides, we can also observe that different weight decay has big impact on the ﬁnal weight distribution, such as larger weigth decay(e.g., µ = 0 .

Original: 005 ) results in more weights are constrained to zero, which may lead to underﬁtting according to their ﬁnal classiﬁcation performances.

Original: 6. E XPERIMENTAL A NALYSIS
Revised: 6. E XPERIMENTAL A NALYSIS

Original: Figure 8(a), 8(b) and 8(c) show the accuracy, average feature norm and softmax loss during training by using λ = 1 , 0 . 1 , 0 . 01 respectively on CIFAR10.
Revised: Figure 8(a), 8(b) and 8(c) show the accuracy, average feature norm and softmax loss during training by using λ = 1 , 0 . 1 , 0 . 01 respectively on CIFAR10.

Original: Feature incay achieves better or comparable accuracy compared with softmax loss under a wide range of λ , and results in larger feature norm on both training and test set.
Revised: Feature incay achieves better or comparable accuracy compared with softmax loss under a wide range of λ , and results in larger feature norm on both training and test set.

Original: All methods achieve close to zero softmax loss on training set, while feature incay ensures lower softmax loss on test set.
Revised: All methods achieve close to zero softmax loss on training set, while feature incay ensures lower softmax loss on test set.

Original: Figure 8(d) shows the accuracy vs iteration number on CIFAR100, which is similar to the results on CIFAR10.[Figure] [Figure]
Revised: Figure 8(d) shows the accuracy vs iteration number on CIFAR100, which is similar to the results on CIFAR10.