R EDUCING C LASS C OLLAPSE IN M ETRIC L EARNING WITH E ASY P OSITIVE S AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
Metric learning seeks perceptual embeddings where visually similar instances are close and dissimilar instances are apart, but learned representation can be sub-optimal when the distribution of intra-class samples is diverse and distinct sub-clusters are present.
We theoretically prove and empirically show that under reasonable noise assumptions, prevalent embedding losses in metric learning, e.g., triplet loss, tend to project all samples of a class with various modes onto a single point in the embedding space, resulting in class collapse that usually renders the space ill-sorted for classiﬁcation or retrieval.
To address this problem, we propose a simple modiﬁcation to the embedding losses such that each sample selects its nearest same-class counterpart in a batch as the positive element in the tuple.
This allows for the presence of multiple sub-clusters within each class.
The adaptation can be integrated into a wide range of metric learning losses.
Our method demonstrates clear beneﬁts on various ﬁne-grained image retrieval datasets over a variety of existing losses; qualitative retrieval results show that samples with similar visual patterns are indeed closer in the embedding space.
I NTRODUCTION
Metric learning aims to learn an embedding function to lower dimensional space, in which semantic similarity translates to neighborhood relations in the embedding space (Lowe, 1995).
Deep metric learning approaches achieve promising results in a large variety of tasks such as face identiﬁcation (Chopra et al., 2005; Taigman et al., 2014; Sun et al., 2014), zero-shot learning (Frome et al., 2013), image retrieval (Hoffer & Ailon, 2015; Gordo et al., 2016) and ﬁne-grained recognition (Wang et al., 2014).
In this work we investigate the family of losses which optimize for an embedding representation that enforces that all modes of intra-class appearance variation project to a single point in embedding space.
Learning such an embedding is very challenging when classes have a diverse appearance.
This happens especially in real-world scenarios where the class consists of multiple modes with diverse visual appearance.
Pushing all these modes to a single point in the embedding space requires the network to memorize the relations between the different class modes, which could reduce the generalization capabilities of the network and result in sub-par performance.
Recently researchers observed that this phenomena, where all modes of class appearance “collapse” to the same center, occurs in case of the classiﬁcation SoftMax loss
(Qian et al.).
They proposed a multi-center approach, where multiple centers for each class are used with the SoftMax loss to capture the hidden distribution of the data to solve this issue.
Instead of using SoftMax, it was shown that triplet loss may offer some relief from class collapsing (Wang et al., 2014) and this is certainly true in noise-free environments.
However, in this paper, we show that in real-world conditions with modest noise assumptions, triplet and other metric learning loss yet suffer from class collapse.
Rather than reﬁne the loss, we argue the key lies in an improved strategy for sampling and selecting the examples.
Early work (Malisiewicz & Efros, 2008) proposed per-exemplar distance representation as a means to overcome class collapsing; inspired by this we introduce a simple sampling method to select positive pairs of training examples.
Our method can be combined naturally with other popular sampling methods.
In each training iteration, given an anchor and a batch of samples in the same category, our method selects the closest sample to the anchor in the current embedding space as the positive sample.
The metric learning loss is then computed based on the anchor and its positive paired sample.
We demonstrate the class-collapsing phenomena on a real-world dataset, and show that our method is able to create more diverse embedding which result in a better generalization performance.
We evaluate our method on three standard zero-shot benchmarks: CARS196 (Krause et al., 2013),
CUB200-2011 (Wah et al., 2011) and Omniglot
(Lake et al., 2015).
Our method achieves a consistent performance enhancement with respect to various baseline combinations of sampling methods and embedding losses.
R ELATED W ORK
Sampling methods.
Designing a good sampling strategy is a key element in deep metric learning.
Researchers have been proposed sampling methods when sampling both the negative examples as well as the positive pairs.
For negative samples, studies have focused on sampling hard negatives to make training more efﬁcient (Simo-Serra et al., 2015; Schroff et al., 2015; Wang & Gupta, 2015;
Oh Song et al., 2016;
Parkhi et al., 2015).
Recently, it has been shown that increasing the negative examples in training can signiﬁcantly help unsupervised representation learning with contrastive losses (He et al., 2020;
Wu et al., 2018; Chen et al., 2020).
Besides negative examples, methods for sampling hard positive examples have been developed in classiﬁcation and detection tasks (Loshchilov & Hutter, 2015; Shrivastava et al., 2016; Arandjelovic et al., 2016; Cubuk et al., 2019; Singh & Lee, 2017; Wang et al., 2017).
The central idea is to perform better augmentation to improve the generalization in testing (Cubuk et al., 2019).
Apart from learning with SoftMax classiﬁcation, Arandjelovic et al.
(2016) propose to perform metric learning by assigning the near instance from the same class as the positive instance.
As the positive training set is noisy in their setting, this method leads to features invariant to different perspectives.
Different from this approach, we use this method in a clean setting, where the purpose is to get the opposite result of maintaining the inner-class modalities in the embedding space.
Xuan et al.
(2020) also propose to use this positive sampling method with respect to the N-pair loss (Sohn, 2016) in order to relax the constraints of the loss on the intra-class relations. From a theoretic perspective, we prove that in a clean setting this relaxation is redundant for other popular metric losses like the triplet loss (Chechik et al., 2010) and margin loss (Wu et al., 2017). We formulate the noisy-environment setting and prove that in this case the triplet and margin losses also suffer from class-collapsing and using our purpose positive sampling method optimizes for solutions without class-collapsing. We also provide an empirical study that supports the theoretic analysis.
Noisy label problem.
Learning with noisy labels is a practical problem when applied to the real world (Scott et al., 2013; Natarajan et al., 2013; Shen & Sanghavi, 2019;
Reed et al., 2014; Jiang et al., 2017; Khetan et al., 2017; Malach & Shalev-Shwartz, 2017), especially when training with large-scale data (Sun et al., 2017).
One line of work applies a data-driven curriculum learning approach where the data that are most likely labeled correctly are used for learning in the beginning, and then harder data is taken into learning during a later phase (Jiang et al., 2017).
Researchers have also tried on to apply the loss only on easiest top k-elements in the batch, determine by lowest current loss (Shen Sanghavi, 2019).
Inspired by these works, our method focuses on selecting only the top easiest positive relations in the batch.
Beyond memorization.
Deep networks are shown to be extremely easy to memorize and over-ﬁt to the training data (Zhang et al., 2016; Recht et al., 2018; 2019).
For example, it is shown the network can be trained with randomly assigned labels on the ImageNet data, and obtain 100% training accuracy if augmentations are not adopted.
Moreover, even the CIFAR-10 classiﬁer performs well in the validation set, it is shown that it does not really generalize to new collected data which is visually similar to the training and validation set (Recht et al., 2018).
In this paper, we show that when allowing the network the freedom not to have to learn inner-class relation between different class modes, we can achieve much better generalization, and the representation can be applied in a zero-shot setting.
P RELIMINARIES
Let X = { x 1 , .., x n } be a set of samples with labels y i ∈ { 1 , .., m } .
The objective of metric learning is to learn an embedding f ( · , θ ) −→ R k , in which the neighbourhood of each sample in the embedding space contains samples only from the same class.
One of the common approaches for metric learning is using embedding losses in which at each iteration, samples from the same class and samples from different classes are chosen according to same sampling heuristic.
The objective of the loss is to push away projections of samples from different classes, and pull closer projections of samples from a same class.
In this section, we introduce a few popular embedding losses.
Notation: Let x i , x j ∈ X , deﬁne: D fx i ,x j = (cid:107) f ( x i ) − f ( x j ) (cid:107) 2 .
In cases where there is no ambiguity we omit f and simply write D x i ,x j .
We also deﬁne the function δ x i ,x j = (cid:26)
if y i =
y j 0 otherwise .
Lastly, for every a ∈ R , denote ( [Equation]
The Contrastive loss (Hadsell et al.) takes sample embeddings and pushes the samples from the different classes apart and pulls samples from the same class together.
[Equation]
Here α is the margin parameter which deﬁnes the desired minimal distance between samples from different classes.
While the Contrastive loss imposes a constraint on a pair of samples, the Triplet loss (Chechik et al., 2010) functions on a triplet of samples.
Given a triplet x a , x p , x n ∈ X , the triplet loss is deﬁned by
[Equation]
The Margin loss (Wu et al., 2017) aims to exploit the ﬂexibility of Triplet loss while maintaining the computational efﬁciency of the Contrastive loss.
This is done by adding a variable which determines the boundary between positive and negative pairs; given an anchor x a ∈ X the loss is deﬁned by
[Equation]
C LASS - COLLAPSING
The contrastive loss objective is to pull all the samples with the same class to a single point in the embedding space.
We call this the Class-collapsing property.
Formally, an embedding f : X −→ R m has the class-collapsing property, if there exists a label y and a point
p ∈ R m such that { f ( x i ) |
y i
= y }
= { p } .
4. E MBEDDING LOSSES OPTIMAL SOLUTION
It is easy to see that an embedding function f that minimizes:
[Equation]
,x j ∈ X
L fcon ( x i , x j ) 
 has the class-collapsing property with respect to all classes.
However, this is not necessarily true for the Triplet loss and the Margin loss.
For simpliﬁcation for the rest of this subsection we will assume there are only two classes.
Let
A ⊂ X be a subset of elements such that all the elements in A belongs to one class and all the element in A c belong to the other class.
Recall some basic set deﬁnitions.
Deﬁnition 1.
For all sets Y, Z ⊂ R m deﬁne:
The diameter of Y is deﬁned by:
[Equation]
The distance between Y and Z is:
[Equation]
It is easy to see that if f : X −→ R m is an embedding, such that diam ( f ( A )) < 2 · [Equation] + (cid:107)
[Equation]
, then:
[Equation]
Moreover, ﬁxing β x i = α for every x i ∈ X , then:
[Equation]
It can be seen that indeed, the family of embedding which induce the global-minimum with respect to the Triplet loss and the Margin loss, is rich and diverse.
However, as we will prove in the next subsection, this does not remain true in a noisy environment scenario.
4. N OISY ENVIRONMENT ANALYSIS
For simplicity we will also discuss in this section the binary case of two labels, however this could be extended easily to the multi-label case.
The noisy environment scenario can be formulated by adding uncertainty to the label class.
More formally, let Y =
{ Y 1 , .., Y n } be a set of independent binary random variables.
Let A 1 , .., A t ⊂ X , 0 .
5 < p < 1 such that: | A j | = nt and
[Equation]
We can also reformulate δ as a binary random variable such that:
[Equation]
For the Triplet loss deﬁne:
[Equation]
We are searching for an embedding function which minimize
[Equation]
Theorem 1.
Let f : O −→ R
m be an embedding, which minimize EO trip ( f ) , then f has the classcollapsing property with respect to all classes.
Similarly, we can deﬁne:
[Equation]
Theorem 2.
Let f : O −→ R
m be an embedding, which minimize
[Equation]
then f has the class-collapsing property with respect to all classes.
The proof of the last two theorems can be ﬁnd in Appendix A.
In conclusion, although theoretically in clean environments the Triplet loss and Margin loss should allow more ﬂexible embedding solutions, this does not remain true when noise is considered.
On a real-world data, where mislabeling and ambiguity can be usually be found, the optimal solution with respect to both these losses becomes degenerate.
4. E ASY P OSITIVE S AMPLING (EPS)
Using standard embedding losses for metric learning can result in an embedding space in which visually diverse samples from the same class are all concentrated in a single location in the embedding space.
Since the standard evaluation and prediction method for image retrieval tasks are typically based on properties of the K-nearest neighbours in the embedding space, the class-collapsing property is a side-effect which is not necessarily in order to get optimal results.
In the next section, we will show experimental results, which support the assumption that complete class-collapsing can hurt the generalization capability of the network.
To address the class-collapsing issue we propose a simple method for sampling, which results in weakening the objective penalty on the inner-class relations, by applying the loss only on the closest positive sample.
Formally we deﬁne the EPS sampling in the following way; given a mini-batch with N samples, for each sample a , let C a be the set of elements from the same class as a in the mini-batch, we choose the positive sample p a to be
[Equation]
For negative samples n a we can choose according to various options.
In this paper we use the following methods: (a) Choosing randomly from all the elements which are not in C a .
(b) Using distance sampling (Wu et al., 2017).
(c) semi-hard sampling (Schroff et al., 2015), (d) MS hard-mining sampling (Wang et al., 2019).
We then apply the loss on the triplets ( a, p a , n a ) .
Using such sampling changes the loss objective such that instead of pulling all samples in the mini-batch from the same class to be close to the anchor, it only pulls the closest sample to the anchor (with respect to the embedding space) in the mini-batch, see Figure 1.
In Appendix B, we formalize this method in the noisy environment framework.
We prove (Claim 1,2) that every embedding which has the class collapsing property is not a minimal solution with respect to both the margin and the triplet loss with the easy positive sampling.
Furthermore, in Claim 3, we prove that the objective of the losses with EPS on tuples/triplets is to push away every element (including positive elements), that is not in the k-closest elements to the anchor, where k is determined by the noise level p .
Therefore, if we apply the EPS method on a mini-batch which has small numbers of positive elements from each modality, in such case adding the EPS to the losses not only relax the constraints on the embedding, allowing the embedding to have multiple inner-clusters.
It also optimizes the embedding to have this form.
E XPERIMENTS
We test our EPS method on image retrieval and clustering datasets.
We evaluate the image retrieval quality based on the recall@k metric (Jégou et al., 2011) , and the clustering quality by using the normalized mutual information score (NMI) (Manning et al., 2008).
The NMI measures the quality of clustering alignments between the clusters induced by the ground-truth labels and clusters induced by applying clustering algorithm on the embedding space.
The common practice to choose the NMI [Table] clusters is by using K-means algorithm on the embedding space, with K equal to the number of classes.
However, this prevents from the measurement capturing more diverse solutions in which homogeneous clusters appear only when using larger amount of clusters.
Regular NMI prefers solutions with class-collapsing.
Therefore, we increase the number of clusters in the NMI evaluation (denote it by NMI+) we also report the regular NMI score.
5. MNIST E VEN /O DD E XAMPLE
To demonstrate the class-collapsing phenomena, we take the MNIST dataset (Lecun et al., 1998)
[Table]
and split the digits according to odd and even.
From a visual perspective this is an arbitrary separation.
We took the ﬁrst 6 digits for training and left the remaining 4 digits for testing.
We used a simple shallow architecture which result in an embedding function from the image space to R 2 (For implementation details see Appendix C).
We train the network using the triplet loss.
We compare our sampling method to random sampling of positive examples (the regular loss).
As can be seen in Figure 2, the regular training without EPS suffers from class-collapsing.
Training with EPS creates a richer embedding in which there is a clear separation not only between the two-classes, but also between different digits from the same class.
As expected, the class-collapsing embedding preforms worse on the test data with the unseen digits, see Table 1.
5. F INE - GRAINED R ECOGNITION E VALUATION
We compare our approach to previous popular sampling methods and losses.
The evaluation is conducted on standard benchmarks for zero-shot learning and image retrieval following the common splitting and evaluation practice (Wu et al., 2017;
Movshovitz-Attias et al., 2017; Brattoli et al., 2019).
We build our implementation on top of the framework of Roth et al., which allow us to have a fair comparison between all the tested methods with an embedding of ﬁx size (128).
For more implementation details and consistency of the results, see Appendix C.
5.2. D ATASETS
We evaluate our model on the following datasets. [Table]
[Table] • Cars-196 (Krause et al., 2013), which contains 16,185 images of 196 car models.
We follow the split in Wu et al.
(2017), using 98 classes for training and 98 classes for testing.
• CUB200-2011 (Wah et al., 2011), which contains 11,788 images of 200 bird species.
We also follow Wu et al.
(2017), using 100 classes for training and 100 for testing.
• Omniglot
(Lake et al., 2015), which contains 1623 handwritten characters from 50 alphabet.
In our experiments we only use the alphabets labels during the training process, i.e, all the characters from the same alphabet has the same class.
We follow the split in Lake et al.
(2015) using 30 alphabets for training and 20 for testing.
5.2. R ESULTS
We tested our sampling method with 3 different losses: Triplet (Chechik et al., 2010), Margin (Wu et al., 2017) and Multi-Similarity (MS) (Wang et al., 2019).
For the Margin loss experiment, we combine our sampling method with distance sampling (Wu et al., 2017); this could be done because the distance sampling only constrains on the negative samples, where our method only constrains on the positive samples.
We set the margin α = 0 .
2 and initialized β = 1 . 2 as in (Wu et al., 2017).
For the Triplet we combine our method with semi-hard sampling (Schroff et al., 2015) by ﬁxing the positive according to EPS and then using semi-hard sampling for choosing the negative examples.
For the MS loss we replace the positive hard-mining method with EPS and use the same hard-negative method.
We use the same hyper-paremeters as in (Wang et al., 2019)
α = 2 , λ = 1 , β = 50 .
Results are summarized in Tables 2 and 3.
We can see that our method achieves the best performance on all tested datasets.
It is important to note that in the baseline models, when using Semi-hard sampling, the sampling strategy was done also on the positive part as suggest in the original papers.
We see that replacing the semi-hard positive sampling with easy-positive sampling, improve results in all the experiments.
The improvement gain becomes larger as the dataset classes can be partitioned [Figure] more naturally to a small number of sub-clusters which are visually homogeneous.
In Cars196 dataset it is the car viewpoint, where in Omniglot it is the letters in each language.
As can be seen in Table 3, using EPS on the Omniglot dataset result in creating an embedding in which in most cases the nearest neighbor in the embedding consists of element of the same letter, although the network was trained without these labels.
In Figure 4 we can see a qualitatively comparison of CARS16 models results.
EPS seems to create more homogeneous neighbourhood relationships with respect to the the viewpoint of the car.
More results and comparisons can be ﬁnd in Appendix C.
5.2. P OSITIVE BATCH SIZE EFFECT
An important hyperparameter in our sampling method is the number of positive batch samples, from which we select the closest one in the embedding space to the anchor.
If the class is visually diverse and the number of positive samples in batch is low, than with high probability the set of all the positive samples will not contain any visually similar image to the anchor.
In case of the Omniglot experiment, the effect of this hyperparameter is clear; It determines the probability that the set of positive samples will include a sample from the same letter as the anchor letter.
As can be seen in
Figure 3(b), the performance of the model increases as the probability of having another sample with the same letter as the anchor increases.
C ONCLUSION
In this work we demonstrate the importance of positive sampling strategies when using embedding losses for metric learning.
We investigate the class collapsing phenomena with respect to popular embedding losses such as the Triplet loss and the Margin loss.
While in clean environments there is a diverse and rich family of optimal solutions, when noise is present, the optimal solution collapses to a degenerate embedding.
We propose a simple solution to this issue based on ’easy’ positive sampling, and prove that indeed adding this sampling results in non-degenerate embeddings.
We also compare and evaluate our method on standard image retrieval datasets, and demonstrate a consistent performance boost on all of them.
While our method and results have been limited to metric learning frameworks, we believe that our sampling scheme will also be useful in other related settings, including supervised contrastive learning, which we leave to future work
A PPENDIX A: P ROOFS FOR THE T HEOREMS IN SUBSECION 4.
Theorem 1.
Let f : O −→ R
m be an embedding, which minimize EO trip ( f ) , then f has the class-collapsing property with respect to all classes.
Proof.
Deﬁne a new random variables such that for every 1 ≤ r 1 , r 2 ≤ t : h r 1 ,r 2 ( Y, Z ) =
{+(cid:40) 1 Y = r 
 ∧+}
Z = r0 else observe that
[Equation]
Since the variables are independent
[Equation]
Deﬁne: ¯ D ( x 1 , x 2 , x 3 ) :=
( D x 1 ,x 2 − D x 1 ,x 3 + α ) +
Rearranging the terms we get
[Equation]
Therefore [Equation] if
[Equation]
then EO trip ( f ) can be written as
[Equation]
For every x i ∈ X [Equation] deﬁne:
[Equation]
Let f : X −→ R
m be an embedding, ﬁx 1 ≤ r ≤ t and x i ∈ A r , x j , x k ∈ X with
[Equation]
=
[Equation]
[Equation] = h.
By deﬁnition: K ( i, j, k, r 1 , r 2 ) =
  p · ( h − w + α ) + + q ( w − h + α ) + r 1 = r ∧ r 2 (cid:54) =
r q · ( h − w + α ) + + p ( w − h + α )
+ r 2 = r ∧ r 1 (cid:54) = r p · ( h − w + α ) + + p ( w − h + α ) + r 1 = r ∧ r 2 = r q · ( h − w + α ) + + q ( w − h + α ) + r 1 (cid:54) = r ∧ r 2 (cid:54) =
r Since 0 < p < 1 , in order to get minimal K ( i, j, k, r 1 , r 2 ) value, h and w must satisfy | h − w | ≤ α . In this case we have
[Equation]
Therefore, [Equation] We split to three cases: 1.
If x j , x k ∈
A r or x j , x k / ∈ A r then: E ( h r,r 2 ( Y j , Y k )) =
E ( h r 2 ,r
( Y j , Y k )) .
Hence, ( h − w )( p − q )) ( (cid:88) r 2 ∈{
[Equation] [Equation]
If x j ∈ A r and x k / ∈ A r , then E ( h r,r 2 ( Y j , Y k )) >
E ( h r 2 ,r ( Y j , Y k )) , therefore
( h − w )( p −
q
[Equation]
[Equation]
Since p > 0 . 5 and | h − w | ≤ α ,the minimal value is achieved whenever h = 0 and w = α . In the same way if x k ∈ A r and x j / ∈ A r , then E ( h h r 2 ,r ( Y j , Y k )) = E ( h r,r 2 ( Y j , Y k )) and the minimal value is achieved whenever h
= α and w =
In conclusion, if x i ∈ A r , an embedding f ∗ satisﬁes
[Equation]
iff f ∗ ( x j ) =
f ∗ ( x i ) for every x j ∈
A r , and (cid:107) f ∗ ( x j ) − f ∗ ( x i ) (cid:107) =
α for every x j / ∈ A r .
We will now prove the same theorem with respect to the margin loss.
Theorem 2.
Let f : O −→ R
m be an embedding, which minimize
[Equation]
then f has the class-collapsing property with respect to all classes.
Proof.
Observe that if x i , x j ∈ A r , then
[Equation]
x j −
[Equation]
x i +
[Equation]
+ + (1 − p ) · (
[Equation]
x i [Equation] D x i ,x j + α ) + Since 0 < p < 1 , then the maximal value is achieved whenever | D x i ,x j − β x i | ≤ α , in this case:
[Equation]
i
[Equation]
p
[Equation] i [Equation] x
[Equation]
In the same way in case x i ∈ A r and x j / ∈ A r then: [Equation] i ,
[Equation]
Combining both directions we get:
[Equation]
Since: p > 0 . 5 and | D x i ,x j − β x i | ≤ α , the minimal value is achieved whenever D x i ,x j = 0 , D x i ,x k = 2 α and β x i = α , for every x i , x j ∈ A r , x k [Equation]
∈ [Equation] r .
B: E ASY P OSITIVE S AMPLING IN NOISY ENVIRONMENT
In this subsection we analyse the EPS method from the theoretical prospective, using the framework deﬁned in Section 4.
We use the same notions as in sections 3 and 4.
Deﬁne: Φ( y i , y j ) =
[Equation]
=
[Equation]
∧
[Equation]
x i ,
[Equation]
j = min { D x i ,x k | y k
= y i } 0 else .
Then, the easy positive sampling loss can be deﬁned by:
[Equation]
for the triplet loss and
[Equation]
for the margin loss.
In the noisy environment stochastic case, using section 4 notions, Φ become a random variable:
[Equation]
else Therefore, the triplet loss with EPS in the noisy environment case, become:
[Equation]
and for the margin loss with EPS we have:
[Equation]
β
[Equation] i − D fx i ,x j + α ) + As in section 4.2 we assume that Y
=
{ Y 1 , .., Y n } is a set of independent binary random variables.
Let A 1 , .., A t ⊂ X , 0 .
5 < p < 1 such that: | A j | = nt and P [Equation] Y i
[Equation]
For simplicity we assume that every 1 ≤ i ≤ nt satisﬁes x n · it +1 , .., x n · it + t ∈
A i We prove ﬁrst that the minimal embedding with respect to both losses does not satisfy the class collapsing property.
Let f 1 be an embedding function such that:
[Equation]
α else and f 2 an embedding such that:
[Equation]
α else f 1 represent the case of class collapsing, where f 2 represent the case where there are two modalities for the ﬁrst class.
In order to show that the minimal embedding does not satisfy the class collapsing property it sufﬁce to prove that
[Equation]
and
[Equation]
Remark:
For both losses the deﬁnition requires a strict order between
Claim 1.
There exists M such that if n ≥ M , then:
[Equation]
Proof.
Fix x 1 , WOLOG we may assume in both embeddings that D f j x 1 ,x i <
D f j x 1 ,x k for every j ∈ { 1 , 2 } and 1 ≤ i < k ≤ n .
It sufﬁce to prove that
[Equation]
Let q = (1 − p ) , observe that
[Equation]
where m = |{ t | t ≤ j, Y t ∈ A 1 }| .
Thus if j ≥ n 2 t , we have
[Equation]
[Equation]
Therefore, n (cid:88)
[Equation]
For j ≤ n 2 t and k ≤ n 2 t of k > nt , we have L f 1 EPStrip ( x 1 , x j , x k ) =
L f 2 EPStrip ( x 1 , x j , x k ) .
Hence, the only case left is j ≤ n 2 t and n 2 t < k ≤ nt .
In this case: L f 2 EPStrip ( x 1 , x j , x k ) = 0 , where
[Equation]
and we get:
[Equation]
Choosing M such that
[Equation]
will satisfy that for every n > M :
[Equation]
Claim 2.
There exists M such that if n ≥ M then:
[Equation]
Proof.
For every 1 ≤ j ≤ n 2 t or nt < j ≤ n we have:
[Equation]
For n 2 t < j ≤ n 2 :
[Equation]
[Equation]
Since j > n 2 t the second therm tend to zero.
Therefore, taking M such that
[Equation]
will satisfy that for each n ≥ M
[Equation]
In the previous two claims we prove that the class collapsing solution is not minimal with respect to both the EPSmargin and the EPStriplet .
In the following claims we prove that not only it is not the minimal solution, looking locally on the direct effect of the EPS losses on a sample which is not one of the closest elements to to the anchor.
We prove that the optimal solution in this case is an embedding in which the distance between the sample to the anchor is equal to the margin hyperparameter.
Claim 3.
Let f be an embedding.
For every i , let i 1 , .., i n be such that D fx i ,x i 1 < D fx i ,x i 2 < ... <
D fx i ,x n ,
Then there exists M such that for every j > M the minimal embedding for L fEPSmargin ( x i , x j ) is achived whenever D fx i ,x j = β x i
+ α .
Proof.
Fix x 1 , as in the previous claims we will assume:
[Equation]
As was prove in in Claim 1 P ( (cid:86) 1 ≤ t<j Y i (cid:54)
=
Y t ) ≤ 4 p j , thus
[Equation]
Since the minimal solution for
[Equation]
i + α ) + + E [Equation] [Equation] ¯ δ Y i ,Y j ) ·
[Equation] x i [Equation] D fx i ,x j + α
[Equation]
+ satisﬁes | β x i − D fx i ,x j | ≤ α , we have:
[Equation]
Since E (1 − ¯ δ Y 1 ,Y j ) ≥ 2 pq , there exists M such every j > M satisﬁes
[Equation]
Therefore the minimal value is achieved whenever D fx 1 ,x j = α + β x 1 .
The proof in the EPStriplet loss case is similar.
Claim 4.
Let f be an embedding.
For every i , let i 1 , , .., i n be such that D fx i ,x i 2 < ... <
D fx i ,x n .
Then there exists M such that for every j > M the minimal embedding for:
[Equation]
is achieved whenever D x i ,x t + j = D x i ,x t + α .
Proof.
Deﬁne K
( Y i , Y i , Y k ) :
=
E (cid:0) ¯Φ( Y i , Y j ) · ¯ δ Y i ,Y j · (1 − ¯ δ Y
i ,Y k ) (cid:1) .
Fixing x 1 , assuming D fx 1 ,x 1 <
D fx 1 ,x 2 < ... <
D fx 1 ,x n , We have:
[Equation]
K ( Y 1 , Y t , Y t + j ) · (cid:16) D fx 1 ,x t − D fx 1 ,x t + j + α (cid:17) + + K ( Y 1 , Y t + j , Y t ) · (cid:16) D f x 1 ,x t + j − D f x 1 ,t + α (cid:17) +
As in the previous claim, the minimal value is achieved whenever | D fx 1 ,x t + j − D fx 1 ,x t | ≤ α in this case:
L fEPStrip ( x 1 , x t , x t + j ) + L fEPStrip ( x 1 , x t + j , x t ) =
α · ( K ( Y 1 , Y t , Y t + j ) + K ( Y 1 , Y t + j , Y t )) Y t )) + ( D f x 1 ,x t − D f x 1 ,x t + j ) · ( K ( Y 1 , Y t , Y t + j ) − K
( Y 1 , Y t + j , Y t ))
[Table]
On the one hand: K ( Y 1 , Y t , Y t + j ) =
( (cid:81) i ∈{ 1 , 2 ,..
[Table]
t
[Table]
t + j } p t i q 1 − t i ) +( (cid:81)
[Table] [Table]
[Table]
+ j } p 1 − t i q t i ) ≥ q t +1 where
t k = (cid:40) 1 Y k / ∈ A 0 else for k ∈ { 2 , .., t − 1 , t + j } and t k = (cid:40) 1 Y k ∈
A 0 else for k ∈ { 1 , t } .
On the other hand
K ( Y 1 Y t + j , Y t ) ≥ Prob ( (cid:86) 1 ≤ k<t + j Y 1 (cid:54) =
Y k ) ≤ 4 p t + j − 1 .
Taking j large enough such that q t +1 ≤ 4 p t + j − 1 , we have: ( E (cid:0) ¯ Φ( Y 1 , Y t ) · ¯ δ Y 1 ,Y t · (1 − ¯ δ Y 1 ,Y t + j ) (cid:1) − E (cid:0) ¯ Φ( Y 1 , Y t + j ) · ¯ δ Y 1 ,Y t + j · (1 − ¯ δ Y 1 ,Y t ) (cid:1) ) >therefore in such case the minimum is archived whenever D fx 1 ,x t + j = D fx 1 ,x t + α .
C: M ORE EXPERIMENTS AND IMPLEMENTATION DETAILS E MBEDDING BEHAVIOR ON TRAINING SETS
The class-collapsing phenomena also occur in the training process of the image retrieval datasets.
Figure 5 visualise the t-SNE embedding (van der Maaten & Hinton, 2008) of Cars196 training classes.
As can be seen, when training without EPS each class ﬁts well to a bivariate normal-distribution with small variance and different means.
Training with EPS result in more diverse distributions and in some of the classes ﬁts batter to a mixture of multiple different distributions.
This can also be measured qualitatively on the Omniglod detest; although training without the EPS results in batter overﬁtting to training samples, the results on the letters ﬁne-grained task are signiﬁcantly inferior comparing to training with the EPS (Table 4).
It is also important to note the low NMI score when using EPS with the number of clusters equal to the number of languages, and the increment of this score when increasing the [Table] [Table] number of clusters to the number of letters.
This indicates that training with EPS results in more homogeneous small clusters, which are more blended in the embedding space comparing to training without
[Table]
MNIST ARCHITECTURE DETAILS
For the MNIST even/odd experiment we use a model consisting of two consecutive convolutions layer with (3,3) kernels and 32,64 (respectively) ﬁlter sizes.
The two layers are followed by Relu activation and batch normalization layer, then there is a (2,2) max-pooling follows by 2 dense layers with 128 and 2 neurons respectively.
R ECOGNITION DATASETS ARCHITECTURE DETAILS
We use an embedding of size 128, and an input size of 224X224 for the ﬁrst two datasets, and 80X80 for the Omniglot dataset.
For all the experiments we used the original bounding boxes without cropping around the object box.
As a backbone for the embedding, we use ResNet50 (He et al., 2016) with pretrained weights on imagenet.
The backbone is followed by a global average pooling and a linear layer which reduces the dimension to the embedding size.
Optimization is performed using Adam with a learning rate of 10 − 5 , and the other parameters set to default values from Kingma & Ba (2014).
S TABILITY ANALYSIS
Following Musgrave et al.
(2020); Fehervari et al.
(2019), it was important to us to have a fair comparison between all tested models.
Therefore, for all the experiments we use the same framework (Roth et al.), with the same architecture and embedding size (128).
We also did not change the default hyper-parameters in all tested methods.
We run each experiment 8 times with different random seeds, the reported results are the mean of all the experiments.
The std of the Recall@1 results of all experiments can be seen in Table 5.
In all cases the differences between the results with and without the EPS are signiﬁcance.
M ULTI - SIMILARITY COMPARISON
From our experiments, the Multi-similarity loss is highly affected by the batch size.
Using Resnet50 backbone, we restrict the number of batch size to 160 for all tested model, which cause to the inferior results of the multi-similarity loss comparing to other methods.
For the sake of completeness we provide the results also on inception backbone with embedding size of 512 as in Wang et al.
(2019), and batch size of 260.
As can be seen in Table 6, also in these cases the results improve when using EPS instead of semi-hard sampling on the positive samples. [Figure]
T RIMMED L OSS COMPARISON
The situation where a class consists of multiple modes can also be seen as a noisy data scenario with respect to the embedding loss, where positive tuples consisting of examples from different modes are considered as ‘bad‘ labelling.
One approach to address noisy labels is by back-propagating the loss only on the k-elements in the batch with the lowest current loss (Shen & Sanghavi, 2019).
Although this approach resembles Malisiewicz &
Efros (2008), the difference is that in Malisiewicz & Efros (2008) they apply the trimming only on the positive tuples.
We test the effect of using Trimmed Loss on random sampled triplets with different level of trimming percentage.
As can be seen in Figure 6, there is only a minor improvement when applying the loss on top of the distance-margin loss on the Omniglot-letters dataset.
This emphasizes the importance of constraining the trimming to the positive sampling only.