H EATED -U P S OFTMAX E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples.
The problem is generally solved by learning an embedding, such that the samples of the same category are close (compact) while samples from different categories are far away (spread-out) in the embedding space.
One popular way of generating such embeddings is to use the second-to-last layer of a deep neural network trained as a classiﬁer with the softmax cross-entropy loss.
In this paper, we show that training classiﬁers with different temperatures of the softmax function lead to different distributions of the embedding space.
And ﬁnding a balance between the compactness, “spread-out” and the generalization ability of the feature is critical in metric learning.
Leveraging these insights, we propose a “heating-up” strategy to train a classiﬁer with increasing temperatures.
Extensive experiments show that the proposed method achieves state-of-the-art performances on a variety of metric learning benchmarks.
I NTRODUCTION
Metric learning is a fundamental research topic in machine learning and has been widely explored in a variety of computer vision applications such as clustering (Xing et al., 2003;
Ye et al., 2007), image retrieval (Lee et al., 2008; Zhang et al., 2017), face recognition (Guillaumin et al., 2009; Schroff et al., 2015), and person re-identiﬁcation (Koestinger et al., 2012;
Lisanti et al., 2017).
The objective of metric learning is usually formulated (Chopra et al., 2005; Hoffer & Ailon, 2015;
Schroff et al., 2015; Zhang et al., 2017) as learning a metric space which is “compact”, where samples from the same classes are close to each other, and “spread-out”, where two randomly sampled non-matching features are far away to each other.
The learned embedding should also be able to generalize well to unseen classes and samples of the same domain.
One solution for this problem is to deﬁne a loss function that tries to explicitly enforce the properties of compactness and spread in the metric space.
Two of the most popular loss functions are the contrastive loss (Chopra et al., 2005) and the triplet loss (Hoffer & Ailon, 2015).
However, such losses face challenges in sampling, as usually there are a very large number of constraints (possible pairs or triplets) in one dataset and most of them become quickly non-informative in the training process.
Using features from the second-to-last (a.k.a. bottleneck) layer of a deep network trained as a classiﬁer with the softmax function and the cross-entropy loss works well for many metric learning based applications (Razavian et al., 2014) such as image retrieval (Babenko & Lempitsky, 2015) and face veriﬁcation (Liu et al., 2017).
However, the goals of classiﬁer training and metric learning are different, i.e. ﬁnding the best decision function vs. learning a “compact” and “spread-out” embedding.
This motivates us to investigate the relation between metric learning and classiﬁer training.
In this paper, we study the gradient of the embeddings and show how the temperature parameter in the softmax function (deﬁned by Hinton et al.
(2015) for knowledge transfer) plays a crucial role in determining the distribution of the embeddings.
This motivates a “heating-up” method that learns a classiﬁer with normalized features and weights using ﬁrst an intermediate temperature and increasing it during training.
The proposed “heating-up” strategy produces compact and spread-out embeddings achieving state-of-the-art performance in deep metric learning.
R ELATED W ORKS
Siamese networks with contrastive loss (Chopra et al., 2005) was one of the earliest attempts to solve the metric learning problem.
By sampling either two data samples from the same category (positive pair) or two different categories (negative pair), contrastive loss tries to pull two points from positive pair together and push away the points from negative pair.
Triplet loss (Hoffer & Ailon, 2015) further requires a margin between the positive and negative pairs distances.
One of the main issues of these losses is that the number of possible pairs or triplets is extremely large for a large dataset.
A reasonable solution to address the sampling issue is mining samples that are the most informative for training, also known as “hard mining”.
There is a large body of works addressing this problem (Schroff et al., 2015; Mishchuk et al., 2017; Harwood et al., 2017; Yuan et al., 2017; Wu et al., 2017).
Semihard mining (Schroff et al., 2015) tries to ﬁnd triplets in a training batch, for which the distance of the positive pair and the distance of the negative pair are within a certain margin.
HardNet (Mishchuk et al., 2017) is designed to mine some of the hardest triplets within one training batch.
Designing structured losses to consider all the possible training pairs or triplets within one training batch and perform “soft” hard mining can be an alternative solution for hard mining (Song et al., 2017; 2016; Ustinova & Lempitsky, 2016).
Lifted structured loss (Song et al., 2016) exploits all triplets in a training batch and provides a smooth loss function for hard mining.
A few deep clustering based losses (Law et al., 2017; Song et al., 2017) have also been proposed to solve the problem.
Proxy NCA (Movshovitz-Attias et al., 2017) proposes to learn semantic proxies for training data and use a NCA loss for training.
Applying hard mining with proxies is more efﬁcient than with samples.
In face veriﬁcation, training a classiﬁer and using the output of the bottleneck layer as embedding performs reasonably well (Wang et al., 2017b).
For this task, the normalization of features (Ranjan et al., 2017), weights (Liu et al., 2017) or both (Wang et al., 2017a) have been explored.
In order to achieve promising results, a learnable or ﬁxed scalar is required to be multiplied to the ﬁnal logits in the softmax function.
There are preliminary discussions (Wang et al., 2017a; Ranjan et al., 2017) about the inﬂuence of this scalar on the embedding.
This paper shows that the scalar can be seen as the temperature parameter of the softmax function in Hinton et al.
(2015), where its inﬂuence on the logits for the task of knowldege transfer was studied.
The temperature was also explored as a post-processing step to calibrate a pre-trained model (Guo et al., 2017) and produce probability estimates aligned with the expected accuracy of the model.
Differently from the above works, which focus on the output probability, the goal of this paper is to study the features from the bottleneck layer.
We analyze how the temperature parameter controls the distribution of the embedding by assigning different gradients to different samples and weights.
Inspired by these ﬁndings, we propose a “heating-up” strategy for training the embedding, which uses increasing temperature while training the classiﬁer.
The proposed method makes the embedding trained simply with the softmax cross-entropy loss achieve comparable or better performance than state-of-the-art deep metric learning methods.
T HE H EATED - UP E MBEDDING 3. R EVISITING S OFTMAX E MBEDDING WITH T EMPERATURE
Given a set of n labelled training samples { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } , where x i ∈ R
d is the feature of the i -th sample, d is the number of dimension for the training data, y i ∈ { 1 , . . .
, M } is the category label of sample x i , and M is the number of categories for training samples, we try to learn an embedding function f ( · ) :
R d → R k , which maps a data sample to a vector in R k , such that for all i, j, p with y i =
y j (cid:54) =
y p , l ( f ( x i ) , f ( x j ))
<
l ( f ( x i ) , f ( x p )) , where l ( · , · ) : R k × R k
→ R is a distance function.
We call f ( x ) ∈ R k
the embedding of the data sample x and use f as f ( x ) to simplify the notation.
Considering training a linear classiﬁer W = [ w 1 , . .
. , w M ] ∈ R k × M and b = [ b 1 , . . .
, b M ] T ∈ R M , with w m and b m being respectively the weight vector and bias of the m th category classiﬁer, z =
[ z 1 , . .
z M ] T = W T f + b ∈ R M is called the logits.
The probability that sample x belongs to category m ∈ { 1 , . .
, M } can be predicted by the softmax function as:
[Equation]
T , which is normally set to 1, is the temperature as mentioned in Hinton et al.
We set α = 1 /T as the reciprocal of the temperature to simplify the notations in the paper.
Assuming the ground-truth distribution of the training sample is q ( m | x ) , generally q ( m | x ) is a Dirac delta function, which equals 1 if m = y and 0 otherwise, where y is the ground-truth label of x , the cross entropy loss with respect to x , and its gradient with respect to z m are deﬁned as:
[Equation]
Considering z m = w Tm f + b m , the gradient with respect to the feature f is:
[Equation]
And the gradient with respect to the weight w m is:
[Equation]
The following part of this section (Sec. 3.2) will show how α changes the magnitude of the gradient assignment to different features and weights.
To show this, we (cid:96) 2 -normalize the classiﬁer weights w m and the feature f (as in Wang et al.
(2017a)).
It helps 1) separate the impact of α from the norm of the embedding and the norm of the classiﬁer weights to the softmax function; 2) achieve better performance than the unnormalized model (Wang et al., 2017a)
(a detailed comparison between the normalized model and unnormalized model will be given in Sec. 4).
Sec. 3.3 will study the effect of the normalization .
Exploiting these ﬁndings to derive an effective strategy to learn compact and spread-out embeddings, we ﬁnally detail our proposed “heating-up” idea in Sec. 3.4.
G RADIENT A SSIGNMENT BY α
In this subsection, we will show how training a deep classiﬁcation network with different α values affects the gradients of different training samples and class weights.
From Eq.
(1), p ( m | x , α ) satisﬁes:
[Equation]
where K is the number of logits whose value equals the maximum logits value.
In other words, as α increases, the predicted probability will become more “spiky” at the logits that have the largest value.
On the other hand, if α approaches 0, the predicted probability will approach the uniform distribution.
The training samples can be divided into: (i) “incorrect” samples: samples that are not classiﬁed as the correct category ( { x : ∃ m (cid:54) =
y, z m ≥ z y } ); (ii) “correct” samples: samples being correctly classiﬁed by the classiﬁer ( { x : ∀ m (cid:54) =
y, z y >
z m } ).
We further deﬁne two subtypes of samples in the “correct” category: “boundary” samples are samples close to the decision boundary; “centroid” samples are samples laying close to the center of the region that belongs to the category.
We denote the normalized weights and features as ˆ w m and ˆ f respectively.
The gradient of the loss w.r.t.
the normalized embedding and the normalized weight (i.e. Eq.
(3)-(4) with ˆ f instead of f and ˆ w m instead of w m ) exhibits three important properties for gradient assignment for different α values.
Property 1.
The gradient of the cross entropy loss with respect to the feature and the weights satisfy:
[Equation]
It’s because − 1 ≤ p ( m | x , α ) − q ( m | x ) ≤ 1 .
Therefore, with very small α (e.g. to the left of the green dashed line in Fig. 1, close to 0 ), all the samples will get small gradients.
Similarly, all the weights will get small gradients.
Property 2.
If ˆ f is the normalized feature of a “correct” sample, the gradient of the cross entropy loss with respect to the feature satisﬁes:
[Equation]
α || ∂ l / ∂ ˆ
f
||
[Figure]
α || ∂ l / ∂ ˆ
f
||
[Figure]
α || ∂ l / ∂ ˆ f ||
[Figure]
If ˆ f is the normalized feature of an “incorrect” sample, the gradient, in most cases 1 , satisﬁes: [Equation]
[Equation]
We give the detailed proof of property 2 in the Appendix A.1.
Overall, with large α (e.g. the blue dashed line in Fig. 1) the magnitudes of the gradients for “incorrect” samples will become very large, while the magnitudes of gradients for “correct” samples will become very small.
Property 3.
If ˆ f is the normalized feature of a “correct” sample, the gradient of the cross entropy loss with respect to the weight satisﬁes: [Equation]
[Equation]
If ˆ f is the normalized feature of an “incorrect” sample, the gradient satisﬁes:
[Equation]
A detailed proof will be given in Appendix.
A.2.
Overall, with large α , the network will only consider few weights whose predicted probabilities violate the ground-truth distribution ( lim α →
+ ∞ p ( m | x , α ) (cid:54) =
q ( m | x ) ).
We named those weights as “hard” weights.
3. I NFLUENCE OF THE N ORMALIZATION
We here discuss the inﬂuence of the normalization on the gradient of the loss w.r.t.
the embedding.
The Jacobian matrix of the (cid:96) 2 -normalized embedding ˆ f with respect to the original embedding f is:
[Equation]
11) where I is the identity matrix.
Considering Eq.
(3) and the chain rule, we have:
[Equation]
Considering the norm ||
f
|| 2 in the denominator, the magnitude of the gradient is inversely proportional to the norm of the embedding.
Therefore, even if the normalized embeddings are the same, the gradients w.r.t.
the embeddings would still different for embeddings with different norms.
Speciﬁcally, the embedding with larger norm will have smaller gradient.
This may seem as a problem, and one possible solution is to remove the norm term
||
f
|| 2 in the denominator.
We tried this idea for the experiments in Sec. 5, but it did not give signiﬁcant improvement.
The reason for that is likely that since ∂(cid:96)/∂ f and f are always orthogonal (Wang et al., 2017a), updating the feature along the direction of gradient cannot change much the norm of the feature.
We did observe that in practice.
When applying (cid:96) 2 -normalization to the feature during training, the norms of features before normalization are very similar.
On the contrary, when training without normalization, the norm of the features may have large variations as it is the case for off-the-shelf classiﬁer as detailed in Sec. 4.
For numerical stability and ease of implementation we use Eq.
(12) to calculate the gradient, but the gradient analysis in Sec.
3.2 still holds for unnormalized features before normalization.
We found out that batch normalization (Ioffe & Szegedy, 2015) without the learned scale 2 , ˆ BN ( · ) , can work slightly better than (cid:96) 2 -normalization.
We deﬁne the batch normalized embedding as:
[Equation]
(13) where k is the number of dimensions of f .
Batch normalization tries to make each dimension of the embedding have zero mean and unit variance.
Therefore, after batch normalization, the norm of the embedding is roughly √ k , and the normalized feature ˆ f BN has norm close to 1, which is similar to (cid:96) 2 normalization.
Batch normalization may work better than (cid:96) 2 -normalization because in ﬁne-grained recognition problem, many embeddings from different categories can be very similar.
Batch normalization removes the mean and scales the embedding thus creating more variance.
For classiﬁer weights, (cid:96) 2 -normalization always gives us promising result.
3. T HE “H EATING - UP ” S TRATEGY
In order to get an intra-category compact, inter-category “spread-out” and well generalized feature, we propose a two-step “heating-up” strategy, which 1) uses an intermediate α (temperature) to start training, and then 2) increases the “temperature” (decreasing α value) and decreases the learning rate.
At the beginning of the training, the network should focus more on the “incorrect” samples and “boundary” samples and quickly moves them to the class center (producing compactness, Fig. 1(d) left).
It should also focus more on the “hard” weights, and push them away (producing spread-out).
Using an intermediate α for training (e.g. the red dashed line in Fig. 1) satisﬁes these conditions.
If the training started with a small α (i.e. high temperature), the network would assign similar (and potentially very small, if α is very small) gradients to all samples (left of the green dashed line in
Fig.
1(a)-1(c)) and all weights.
There are two main issues with this, 1) assigning gradient to the centroid samples at the beginning of the training will move them to a very small region close to the class center.
However, in metric learning problem, as the training categories are different from the test categories, this can make the embedding overﬁt the training categories and not generalize well to test categories; 2) “incorrect” samples and “hard” weights affect the accuracy the most.
Failing in giving larger gradients (i.e. higher priority) to those will make the training process inefﬁcient.
On the contrary, choosing a large α (i.e. very low temperature) for the initial training would assign large gradients to “incorrect” samples and very small gradients to all “correct” (“centroid” and “boundary”) samples (right of the red dashed line in Fig.
1(a)-1(c)).
Since the “boundary” samples will not get enough update, they will stay near the decision boundary.
Therefore, the embedding of the samples of the same category will not be compact (as shown in Appendix C, features of the same category are more compact for the model trained with smaller α values).
Also, the network will only focus on the “hard” weights and not pushing away other weights, which makes the weights vector not spread-out.
The non spread-out weight vectors results in the features not being spread-out, since the features are moved towards corresponding weights.
After the ﬁrst step of training with an intermediate α , most of the “incorrect” samples become “boundary” samples and most of the “boundary” samples become “centroid” samples (Fig. 1(d), right), in order to make the feature more compact, we increase the temperature (decrease α value).
It helps
[Figure]
assign sufﬁcient gradient to all “correct” samples and push them towards the class center.
In practice, we also reduce the learning rate to avoid overﬁtting.
Multiple strategies could be deﬁned to increase the temperature during training: (i) gently increasing the temperature; (ii) training with the starting temperature until convergence and using a higher temperature to ﬁne-tune the trained network.
Both methods lead to similar performance.
However, since the former method would introduce an additional parameter to control the speed of increase of the temperature, we used the latter in our experiments in Sec. 5.
The visualization of the feature learned with different models on Car196 dataset (Krause et al., 2013) is provided in Appendix C.
C OMPARING WITH O FF - THE -S HELF C LASSIFIER
It is interesting to compare the embedding from an off-the-shelf classiﬁer, i.e. trained with unnormalized features and weights and α = 1, with the embeddings obtained by a model trained with normalized features and weight and some ﬁxed α value.
We train a LeNet (LeCun et al., 2015) model on the MNIST dataset, and set the number of nodes in the bottleneck layer to 2 for visualization.
50,000 samples are used for training, and 10,000 different test samples are used to draw the ﬁgures.
In Fig.
2, different colors represent different digits, and each diamond corresponds to the weight of the classiﬁer of the corresponding digit (For better visualization, the weight are slightly moved towards origin).
We can see in Fig.2(a), as observed in other works (Wang et al., 2017a;
Ranjan et al., 2017), that for an off-the-shelf classiﬁer (i) the magnitude of the embedding can be extremely large, since “correct” samples with larger norm will produce smaller loss (Wang et al., 2017a); (ii) the embedding is not “compact”.
Even if the feature is (cid:96) 2 -normalized (Fig. 2(b)), the embedding is still not as “compact” as the feature trained with normalization and proper α values (Fig. 2(c), see detailed analysis in Appendix B.1).
Training with (cid:96) 2 -normalization for the feature is different from simply applying normalization to the ﬁnal feature in the test phase.
As discussed and shown in Sec.
3.3 in Eq.(11) and (12), (cid:96) 2normalization will change the gradient of each sample during training.
E XPERIMENTS ON M ETRIC L EARNING
We conduct experiment on the following ﬁne-grained datasets, using the training/test splits of Movshovitz-Attias et al.
In all these datasets, the categories in the training and test splits do not overlap.
For the sake of completeness, we also report ﬁne-grained classiﬁcation results on Car196 and CUB200 in Appendix D.
• Cars (Car196) (Krause et al., 2013) is a ﬁne-grained car category dataset, which contains 16,185 images of 196 car models.
8,054 images of the ﬁrst 98 categories are used for training, while the 8,131 images of the other 98 categories are used for test.
• Caltech-UCSD Birds-200-2011 (CUB200) (Welinder et al., 2010) is a ﬁne-grained dataset which contains 11,788 images of 200 bird species.
5,864 images of the ﬁrst 100 species are used for training, while the 5,924 images of the other 100 species are used for test.
• Stanford Online Product (Product) dataset (Song et al., 2016) contains 120,053 images of 22,634 products categories.
59,551 images of 11,318 categories are used for training, while the other 60,502 images from 11,316 categories are
• In-shop Clothes Retrieval (Fashion) dataset
(Liu et al., 2016) contains 54,642 images of 11,735 categories of ﬁne-grained clothes.
The dataset is split into 3 subsets.
52,712 images of 7,982 categories are used for training.
The other 28,760 images of 3985 categories are kept for test, split into a gallery set (12,612 images) and a query set (14,218 images).
I MPLEMENTATION D ETAILS
We use the TensorFlow Deep Learning framework (Abadi et al., 2016) to implement the proposed method.
For fair comparison, we exactly follow the details in Movshovitz-Attias et al.
We use a GoogLeNet V1 (Szegedy et al., 2015) network pre-trained on ILSVRC 2012-CLS data (Russakovsky et al., 2015) as base network.
The input images are all resized to 256 × 256 .
For training, the resized images are randomly crop to 224 × 224 with random horizontal ﬂipping.
In test phase, we use one single center crop as in Movshovitz-Attias et al.
For Car196, CUB200 and fashion datasets the network is ﬁne-tuned by SGD optimizer with 0.9 momentum.
The learning rate is set to 0.004.
For product dataset, the optimizer is ADAM with learning rate of 0.01.
The embedding size is set to 64 and the batch size is set to 32.
We choose α = 16 for all the datasets, as an “intermediate” temperature, it works well for different embedding sizes (see Sec. 5.3).
For “heating-up”, α will decrease (temperature increases on the other hand) from 16 to 4 and the learning rate will decrease to 1/10 of the original learning rate.
The training process usually converges within 50 training epochs, which is similar to the fastest state-of-the-art method, ProxyNCA (Movshovitz-Attias et al., 2017).
5. E VALUATION
Following previous works on metric learning, we evaluate the clustering quality and the retrieval performance on the images of the test set.
All the features are (cid:96) 2 -normalized before calculating the evaluation metric, as in Song et al.
The normalized features performs slightly better than the unnormalized feature.
For clustering, the K-Means algorithm is run on all the embeddings of the test samples.
The number of cluster is chosen to be the number of categories in the test set.
Each test sample will be assigned a cluster index according to which cluster it belongs to.
Normalized Mutual
Information (NMI) (Schütze et al., 2008) between the clustering index and the ground-truth label is used as the metric for clustering.
For retrieval, the performance is evaluated by Recall@K, which is also a widely used metric for this problem.
Given a query sample from the test set, K samples from the rest of the test set (or the gallery set for the fashion dataset) with the smallest distance are retrieved.
If any retrieved sample is from the same category as the query sample, the recall for this query is 1, otherwise, 0.
The reported Recall@K is the average recall on the whole test set.
We train a classiﬁer on the training dataset with the softmax function and cross-entropy as baseline (SM).
For the baseline classiﬁer, in training, the features and the weights are not normalized and α is set to 1.
4 different versions of classiﬁers trained with the proposed methods are used for evaluation:
• LN : softmax with (cid:96) 2 -normalized embedding, (cid:96) 2 -normalized weights and α = 16 .
• BN : softmax with batch normalized embedding, (cid:96) 2 -normalized weights and α = 16 .
• HLN :
Heated-up model using α = 4 to ﬁne-tune LN.
• HBN :
Heated-up model using α = 4 to ﬁne-tune BN.
We also compare the proposed method with state-of-the-art metric learning methods.
Existing literatures are using different base networks and different evaluation protocols.
For fair comparison, only the methods using GoogleNetV1 as base network and Euclidean distance as the ﬁnal evaluation metric are listed: [1] Triplet learning with semi-hard negative mining (Schroff et al., 2015), [2] Lifted structured loss (Song et al., 2016), [3] Learnable Structured Clustering (Song et al., 2017); [4]
Deep
Clustering Learning without spectral learning (Law et al., 2017); [5] Deep Metric Learning with
Smart Mining (Harwood et al., 2017) and [6] ProxyNCA (Movshovitz-Attias et al., 2017).
For the
Fashion dataset, we compared with [7] FashionNet (Liu et al., 2016) and [8] Hard-Aware Deeply
Cascaded Embedding (Yuan et al., 2017).
5. M ETRIC L EARNING
The performances of all the methods on all datasets are listed in Tables 1 and 2.
The softmax baseline already shows comparable result with many other triplet loss based methods.
The embeddings trained with either (cid:96) 2 -normalization or batch normalization improve the performance of the softmax baseline.
[Table] [Table]
Since in test phase
[Table]
all the features are (cid:96) 2 normalized before calculating the metric, the performance gain is not coming from the (cid:96) 2 -normalization of the ﬁnal features.
Batch normalization works slightly better than (cid:96) 2 -normalization.
The “heated-up” models (HLN and HBN) show better performance in almost all the metrics compared to the embedding trained with a ﬁxed temperature.
We further study how different embedding sizes and α values affect the retrieval performance.
The size of the embedding is chosen in [64 , 128 , 256] , and the α value in [4 , 8 , 16 , 32 , 64] .
Normalization is applied to both the feature and the weight.
The R@1 metric on the test set with different embedding sizes and α values is reported in Table 3.
Performances of the feature learned by softmax function without normalization and feature learned from “heated-up” model are also given as “SM” and “HBN”.
The “heated-up” model outperforms all the other models by a signiﬁcant margin.
Between models trained with ﬁxed α values, the model using α = 16 outperforms others.
[Table]
D ISCUSSION
We have discussed how the temperature parameter in the softmax function affects the distribution of the embedding in the second last layer of a deep classiﬁcation model.
Training with an intermediate temperature will lead to an intra-category compact and inter-category “spread-out” embedding which is beneﬁcial for both clustering and retrieval.
A "heating-up" method is also proposed to further improve the clustering and retrieval performance of the embedding by ﬁne-tuning with a higher temperature.
Our classiﬁer based approach achieves good performance in metric learning problems with a simple and efﬁcient training process.
A PPENDIX A G RADIENT A NALYSIS FOR C LASSIFIER WITH N ORMALIZATION AND T EMPERATURE A. P ROOF OF P ROPERTY
Eq (3) contains M terms in the sum, one term for each of the M categories.
There is 3 types of terms: type 1, term with respect to the ground-truth category; type 2, term with respect to the logits that has the largest value and does not belong to the ground-truth category; type 3, other terms.
For α → + ∞ , considering the term of type 1 for “correct” samples ﬁrst, since lim α → + ∞ p ( y | x , α ) = 1 and q ( y | x ) = 1 , according to the property of exponential function, lim α → + ∞ α ( p ( y | x , α ) − q ( y | x ))
=
and the magnitude for this term will approach 0 .
For other terms, the magnitude will also approach 0 , due to lim α → + ∞ p ( m | x , α ) = 0 , and q ( m | x ) = 0 , m (cid:54) =
y .
Therefore, the magnitude of the gradient will always approach to 0 .
Considering “incorrect” samples, the magnitude of the type 1 term ( α ( p ( y | x , α ) − q ( y | x )) ˆ w y ) will approach + ∞ as α → + ∞ , because p ( y | x , α ) will approach either 0 or 1 /K
( K ≥ 2 ) 3 and q ( y | x ) = 1 .
Similarly, the magnitude of the term of type 2 will also approach + ∞ .
For other terms, due to the property of the exponential function, that if z m (cid:54) = max( z 1 , ..., z M ) , lim α → + ∞ αp ( m | x , α ) = 0 , the magnitude of any term of type 3 will decrease to 0.
Therefore, for “incorrect” samples, as α → + ∞ , unless in some special cases 4 , the magnitude of the gradient with respect to the normalized embedding will approach inﬁnity.
A. P ROOF OF P ROPERTY
From Eq.
(1) and Eq.
(5), similar to the discussion in Appendix A.1, for the normalized feature ˆ f of a “correct” sample, lim α → + ∞ α
( p ( y | x , α ) − 1) = 0 and lim α → + ∞ αp ( m | x , α ) = 0 , ∀ m (cid:54) =
y .
Applying these to Eq.
(4), Eq.
(9) is proved.
For the normalized feature ˆ f of an “incorrect” sample, similar to the discussion in Appendix.
A.1, when lim α → + ∞ ( p ( m | x , α ) − q ( m | x ))
(cid:54) = 0 , which means z m = max( z 1 , ..., z M ) or m =
y , the magnitude of the gradient will approach to + ∞ .
Otherwise, the magnitude of the gradient will approach to 0.
B T HE G RADIENT A NALYSIS FOR O FF - THE -S HELF C LASSIFIER
To understand the feature learned with the off-the-shelf classiﬁer, we need to study the gradient of the loss with respect to the feature and the gradient of the loss with respect to the classiﬁer weights.
B. G RADIENT WITH RESPECT TO THE FEATURE
By setting α = 1 , considering z j = ||
f
|| 2 w
Tj
ˆ f , Eq.
(1) becomes:
[Equation]
14) p ( m | x ) satisﬁes:
[Equation] [Equation]
where K is the number of logits whose value equals the maximum logits value.
In other words, as || f || 2 increases, the predicted probability will be spikier at the logits that have the largest value.
3 K can’t be 1, otherwise x is a “correct”
||
f |||| ∂ l / ∂ f || 0.
(a) Correct centroid sample ||
f |||| ∂ l / ∂ f || 0.
(b) Correct boundary sample ||
f |||| ∂ l / ∂ f || 0.
(c) Incorrect sample
By setting α = 1 , Eq.
(3) becomes,
[Equation]
We also consider the 3 types of terms in Eq.
(16) as deﬁned in Sec. 3.2.
When || f || 2 → + ∞ , for “correct” samples, the predicted probability will become a one-hot vector, and all the terms in Eq.will approach to 0 .
Therefore, the overall gradient will approach to 0 and the magnitude of the gradient is 0.
For “incorrect” samples, when || f || 2 → + ∞ , since p ( y | x ) will approach to either 0 or 1 /K and q ( y | x ) = 1, type 1 term will become either − w y or (1 /K − 1) w y , both are ﬁxed vector.
Similarly, all type 2 terms will become ﬁxed vectors ( 1 /K w m ).
Type 3 terms will approach to 0 .
Therefore,
Eq.
16 will approach to a ﬁxed vector (the vector sum of type 1 term and type 2 terms) and the magnitude of the gradient will approach to a constant.
Unless some special cases, the constant is not 0.
Thus, the properties used in Sec.
4 have been proved.
Similarly to Figure 1, the relation between the magnitude of the gradient w.r.t.
the norm of the feature for “incorrect” and “correct” samples is shown in Fig.
As boundary features trained with large α , boundary features (Fig. 3(b)) with large norm (blue dashed line in Fig.
3) will not get enough update thus leading to a not compact embedding.
B. G RADIENT WITH RESPECT TO THE WEIGHTS
Considering the gradient with respect to the weights, from Eq.
2, we have
[Equation] [Equation]
From Eq.
(14), Eq.
(15) and the property of the exponential function, similar to the discussion in
Appendix.
A.2, we have, for a “correct” sample feature f , lim
||
f || 2 → + ∞
||
f
||
( p ( y | x ) − 1)
=
and lim ||
f || 2 → + ∞
||
f
|| 2 p ( m | x ) = 0 , ∀ m (cid:54) =
y , and therefore
[Equation]
For “incorrect” samples,
[Equation]
Therefore, for a “correct” sample, whose feature has large norm, the network will failed to push the weights of other categories away from the feature, even if the weights are close to the feature.
Also, for an “incorrect” sample, whose feature has large norm, the network will only consider the weights of the categories with largest logits and the weight of the ground-truth category of the sample.
Even if the other weights are close to the feature (but not close enough to be the weight of the maximum logits), the network will not push the weights away.
Therefore, the weights vector may not be spread-out.
Since the network will move the feature towards the corresponding weight in order to increase the logits and decrease the loss, if the weights are not spread-out, the features will not be spread-out.
C V ISUALIZATION OF THE DISTRIBUTION OF THE FEATURE
The normalized histograms of the cosine similarity between the embeddings (learned with different models on the training set of the Car196 dataset) of samples of the same class (red) and samples of different class (blue) are shown in Fig.
4, for both the training (left) and test (right) sets.
We show features trained with a total of 6 models: 1) Softmax, 2) BN (Softmax+BN), α = 4 , 3) BN, α = 8 , 4)
BN, α = 16 , and 5) BN, α = 32 , and 6) HBN, α = 16 → 4 ,
A good embedding should exhibit the following properties:
• the features from the same category should be compact, which would transcribe as the red histogram being concentrated at the location where the cosine similarity is closed to 1; • the features from different categories are spread-out, which would be observed as concentration of the blue histogram at the location where the cosine similarity is close to 0.
In other words, two randomly sampled samples from different categories are close to orthogonal (Zhang et al., 2017).
• the embedding should be able to generalize to unseen classes of the same domain, i.e. the distribution on the test set should also exhibit compactness and spread-out properties.
Fig. 4(a) and 4(b) are the histograms for the model trained with the softmax function without any normalization.
The main drawback of the feature is that the histogram of the feature of different categories is not concentrated (both for the training and test set), which means the samples from different categories are not spread-out (as detailed in Appendix B.2).
Therefore the margin between different categories is not enough, or the features are not discriminative enough.
For BN models learned with different α values (Fig. 4(c)-4(j)), in training set (left hand side), we clearly see the trend that a smaller α value will lead to more compact features.
Unfortunately, this trend doesn’t hold for test set (right hand side).
For the model trained with α = 4 , 8 , although in the training set the histograms show nice compactness and spread-out properties, however the features are not compact at all on the test set.
The reason is that, in metric learning problem, the training categories and the test categories are not overlapped, pushing the training samples of each category too close to each other can lead to overﬁtting.
We are especially interested in comparing the histograms of the features learned without heating-up (Fig. 4(g)-4(h)) and with heating-up (Fig. 4(k)-4(l)).
Applying the proposed “heating-up” strategy, i.e . ﬁne-tuning the network with smaller α and learning rate, makes the positive pairs more compact while keeping the negative pairs spread-out.
It improves the clustering and retrieval performance over model trained without “heating-up” as shown in Sec.
[Figure]
[Table] [Table]
[Table]
D F INE - GRAINED C LASSIFICATION
Metric learning focuses on learning a compact and spread-out embedding.
The goal of ﬁne-grained classiﬁcation focuses on learning a decision function to better classify the samples of different categories.
Although the ﬁne grained classiﬁcation task doesn’t require a compact and spread-out embedding, it’s still interesting to see whether the proposed heating-up strategy helps ﬁne-grained classiﬁcation or not.
We report in Table 4 the results of the ﬁne-grained classiﬁcation performance of different models on the Car196 and CUB200 datasets.
The ﬁne-grained classiﬁcation model is trained with the predeﬁned training/test split on all categories.
Different from the training setting for metric learning in Sec.
5.3, here the training and test categories are the same.
Except this, all other settings are the same.
The classiﬁcation accuracy over all categories is used for evaluation.
The models trained with heating-up strategy outperforms the baseline model by a clear margin.
The (cid:96) 2 normalization model seems to work better than batch normalization model in ﬁne-grained classiﬁcation problem.
[Table]
E C OMPARISON WITH L EARNED α
We compare the heating-up strategy with the strategy of directly learning the value of α proposed by
Wang et al.
(2017a).
For learning α , we apply different α values [1 , 2 , 4 , 8 , 16 , 32] as initialization.
Other experiment settings are kept the same as in Sec. 5.
The initialized α value, ﬁnal α value after convergence, NMI and Recall@1 are shown in Table 5.
According to the result, training with a ﬁxed intermediate α value (BN) outperforms the method with learnable α .
A similar conclusion is also reported by Ranjan et al.
The ﬁnal performances and the ﬁnal α values are different with different initialized α values.
The best performance is achieved when initializing α = 8 .
It is interesting to observe that the ﬁnal learned alpha values are always increased compared to the initial values, which is the opposite of what our heating-up strategy does.
The reason can be interpreted by the properties mentioned in Appendix B.
For “correct” samples, larger α value, which is equivalent to larger feature norm, results in smaller loss.
Since most of the samples will become “correct” samples, the α value will become large.
We also apply the heating-up strategy to both learnable and ﬁxed alpha settings.
For learnable α , the α values are reduced to 1 / 4 of the ﬁnal α values.
The results show that the proposed heating-up strategy provides performance gains to all the settings.
Learning with the ﬁxed α value and heating-up (BN with heating-up, HBN) is still the winner overall.
[Table]
F I MPORTANCE OF N ORMALIZATION
Applying (cid:96) 2 normalization to the classiﬁer weights in network training helps to increase angular margin instead of Euclidean distance between different classes.
It helps to get more discriminative feature representation.
In Table 6, we show the performances of the following models: 1) SM:
Off-the-shelf structure with no normalization to feature or weights, 2)
SM+
α : Off-the-shelf structure trained with different temperature values, 3) BN w/o WN
: Trained with batch normalization to the features but no normalization to the weights, and 4) BN:
Trained with both batch normalization to the features and (cid:96) 2 normalization to the weights.
BN w/o WN gets better performance than SM but worse performance than BN.
For
BN w/o WN models, the best performance is achieved when α = 16 , which is the same as BN.
It shows that the temperature also works for models trained without weights normalization, and the weight normalization helps to improve the performance.
It shows that the temperature also works for models trained without weights normalization, and the weight normalization helps to improve the performance.
[Table]
G F INAL T RAINING A CCURACY WITH D IFFERENT α V ALUES
As we mentioned in Sec.3, training with small α values will assign similar gradients to all “incorrect” samples and “correct” samples.
However, since “incorrect” samples affect the accuracy the most, failing in giving larger gradients (i.e. higher priority) to those samples will affect the ﬁnal accuracy.
To verify this, we show the ﬁnal training accuracy with different α values on Car196 dataset in Table 7.
“1 - Train Acc” indicates the ratio of the number of “incorrect” sample and the number of total samples.
The result shows that, when training with small α value, especially when α = 2 , the training accuracy is signiﬁcantly lower than the that of trained with larger α values.
It veriﬁes that training with large α values will focus more on the “incorrect” samples and provide higher training accuracy.
[Table]