U NDERSTANDING M ETRIC L EARNING ON U NIT H YPERSPHERE AND G ENERATING B ETTER E XAMPLES FOR A DVERSARIAL T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent works have shown that the adversarial examples can improve the performance of representation learning tasks.
In this paper, we boost the performance of deep metric learning (DML) models with adversarial examples generated by attacking two new objective functions: intra-class alignment and hyperspherical uniformity .
These two new objectives are motivated by our theoretical and empirical analysis of the tuple-based metric losses on the hyperspherical embedding space.
Our analytical results reveal that a) the metric losses on positive sample pairs are related to intra-class alignment; b) the metric losses on negative sample pairs serve as uniformity regularization on hypersphere.
Based on our new understanding on the DML models, we propose A dversarial D eep M etric
L earning model with adversarial samples generated by A lignment or U niformity objective (ADML+A or U).
With the same network structure and training settings, our ADML+A and ADML+U consistently outperform the state-of-the-art vanilla DML models and the baseline model, adversarial DML model with attacking triplet objective function, on four metric learning benchmark datasets.
I NTRODUCTION
Deep metric learning (DML) has been applied to various computer vision tasks ranging from face recognition (Schroff et al., 2015;
Liu et al., 2017) to zero-shot learning (Romera-Paredes & Torr, 2015; Bucher et al., 2016) and image retrieval (Song et al., 2016; Wu et al., 2017).
It has been proved to be one of the most effective methods for learning the distance-preserving features of images.
The intuition of DML is to pull the embedding of positive images pairs together and push the negative pairs apart, where the embedding function could be a deep neural network.
Most of the metric losses in DML are tuple-based (Schroff et al., 2015; Song et al., 2016; Wu et al., 2017; Wang et al., 2019) or classification-based (Movshovitz-Attias et al., 2017; Kim et al., 2020; Boudiaf et al., 2020), these different losses have been shown to achieve similar performance in the recent reviews of DML (Roth et al., 2020; Musgrave et al., 2020).
One common ground of existing DML models is that the embedding space is a unit hypersphere.
It is widely known that achieving uniformity on hypersphere can increase the generalization of models and preserve as much information as possible (Bachman et al., 2019; Liu et al., 2018; 2021; Hjelm et al., 2018), and the objective function that lead to uniformity is called uniformity regularization.
Meanwhile, the downstream tasks in DML favor the models with small intra-class alignment (Wu et al., 2017; Wang et al., 2019).
In this work, we investigate these two properties, intra-class alignment and hyperspherical uniformity (Wang & Isola, 2020) for tuple-based metric losses.
We derive the theoretical analysis for the triplet loss to prove that the triplet loss on the positive sample pairs minimizes the intra-class alignment by mapping all samples from one class to the same vector, while the triplet loss on the negative sample pairs achieves hyperspherical uniformity .
We further conduct empirical studies to show that the same statement is also valid for other tuple-based metric losses.
We utilize our new understanding on DML to design novel robust DML methods to enhance the performance via improved adversarial training.
Adversarial training aims at improving the robustness of models towards to certain types of attacks by training with perturbed samples.
However, as shown in the recent work on contrastive representation learning (Jiang et al., 2020), adversarial training can also enhance the natural performance on the downstream classification task.
Due to the similarity between contrastive learning and deep metric learning, we believe it’s also possible to improve the nature performance of metric learning models with adversarial samples.
Following our new insights on positive and negative metric losses, we generate perturbations by attacking the alignment or uniformity objective, and create adversarial DML models augmented with both normal samples and perturbed samples.
Our experimental results show that the new adversarial DML models can significantly boost the natural performance.
The major contributions of our paper can be summarized as follows:
• We analyze the intra-class alignment and hyperspherical uniformity for tuple-based metric losses, and establish the connections between these two properties and the positive/negative metric losses.
• Based on our new analysis and understanding, we propose two new adversarial DML models, ADML+A and ADML+U, via attacking the alignment or uniformity objective.
ADML+A and ADML+U improve the natural performance on benchmarks significantly.
R ELATED WORKS
Deep metric learning.
There are mainly two kind of metric losses in DML, tuple-based and classification based losses.
Tuple-based losses include contrastive loss (Hadsell et al., 2006), triplet loss (Schroff et al., 2015), margin loss (Wu et al., 2017), and multi-similarity loss (Wang et al., 2019), where the objective function is based on the distance between positive pairs and negative pairs.
In classification-based losses, the learning objective is not depend on the positive or negative pairs but a fixed (Boudiaf et al., 2020) or learnable proxy (Kim et al., 2020).
In the recent reviews of metric learning methods (Roth et al., 2020; Musgrave et al., 2020), it’s concluded the improvement on the
DML performance is mainly due to different training strategies and unfair comparison.
The original contrastive loss and triplet loss still achieved comparable result with other metric losses under the same network and training strategies.
In experiments we apply the training framework of (Roth et al., 2020) to ensure fair comparison.
Learning with hyperspherical uniformity.
Hyperspherical learning regards learning tasks where the embedding space is a unit hypersphere.
The uniformity of the hypersphere represents the diversity of vectors on the sphere.
It encourages vectors to be spaced with angles as large as possible so that these vectors can be evenly distributed on the hypersphere (Liu et al., 2018).
Achieving hyperspherical uniformity can help with preventing overfitting and improving generalization of the neural works (Liu et al., 2021).
The objective functions which can lead to the uniformity on hypersphere are called uniformity regularization.
Hyperspherical embedding is widely applied in representation learning tasks such as contrastive representation learning (Oord et al., 2018; Hjelm et al., 2018) and
DML (Wu et al., 2017; Liu et al., 2017).
Wang & Isola (2020) showed that the objective function in contrastive representation learning optimizes for intra-class alignment and uniformity together.
Adversarial examples improves natural performance.
In classification tasks, it is well known that the clean accuracy of adversarially trained model is typically worse than the normal model.
However, Xie et al.
(2020) showed that adversarial samples can be used to improve the clean accuracy of image classification models.
According to (Jiang et al., 2020), training with adversarial samples can help improve the natural performance of contrastive learning models on the downstream classification tasks.
The authors presented adversarial attacks based on the objective of contrastive learning and achieved improvement on both natural and robustness performance.
It’s believed that adversarial examples contain extra features, thus the generalization of models augmented with adversarial example is increased (Ilyas et al., 2019;
Salman et al., 2020; Xie et al., 2020), which contributes to better natural performance.
Duan et al.
(2018) utilized the triplet loss as the attacking objective to generate adversarial examples to improve DML models.
To our best knowledge, this is the only one work using adversarial example to boost the natural performance of DML.
In our work, we show that alignment and uniformity loss can generate stronger adversarial examples comparing to triplet loss, and thus lead to better generalization of DML models.
A LIGNMENT AND HYPERSPHERICAL UNIFORMITY IN TUPLE - BASED METRIC LOSSES
In this section, we will study tuple-based metric losses on the unit hypersphere embedding space.
We assume having n classes X 1 , · · · , X n in training set and denote the encoder by f : R d → S k − where S k − 1 is the surface of a k -dimensional unit ball.
Let p data ( · ) be the data distribution over R d , p pos ( · , · ) be the distribution of positive pairs over R d × R d , and p tri ( · , · , · ) be the distribution of triplet pairs over R d × R d × R d , where the first two entries have the same label and the third entry is a sample from different classes.
Please note that all detailed proofs are included in supplementary material
Appendix E. We also conduct experiments to validate our theoretical analysis, the details is in Appendix C.
The major intuition of DML is to pull the representations of similar samples together and push dissimilar samples apart.
Thus, we reformulate the metric losses as the combination of two parts:
• Positive metric loss : minimizes the distance between embedded positive sample pairs.
• Negative metric loss : maximizes the distance between embedded negative sample pairs.
Although in DML models the positive metric losses have different representations, they share one common optimal solution pattern, where samples from the same class are mapped to the same feature vector.
Thus, we define the alignment loss with minimizing the intra-class distance.
Definition 1.
(Intra-class alignment)
The expectation of intra-class distance is given by:
[Equation]
the minimum of this loss is achieved when the samples with the same label are encoded to the same embedding.
Proposition 1.
If the support set of the data distribution is connected and the support set of each class distribution is closed, the minimum of L alignment is reached when all samples are projected to the same vector.
In Sec.
C.1, we conduct the empirical studies to verify our analysis.
Results in Table 8 show that samples are roughly projected to the same vector if only the positive metric losses are used.
The negative metric losses aim at positioning the embedding of dissimilar samples as far as possible.
However, because the embedding space of DML is a unit hypersphere, where the maximum distance between two points is 2 , it’s not possible to separate all negative embeddings with a large margin.
Actually on S k − 1 , the number of points with pairwise distance larger or equal than √ 2 is at most 2 k and the embedding dimension k is always much smaller than the number of feature vectors, thus it’s impossible to make all distances between negative pairs exceed √ 2 .
Therefore investigating the properties of negative metric losses on the unit hypersphere is an interesting and important topic.
We believe the negative metric losses are closely related to the uniformity on the hypersphere and our experimental results support this argument.
Definition 2.
(Hyperspherical uniformity)
The embedded samples should be evenly distributed on the spherical surface.
In practice, the hyperspherical uniformity can be achieved by optimizing the uniformity regularization.
There exist many different representations of the regularization, and we utilize the hyperspherical energy (HE)
(Liu et al., 2018):
[Equation]
and Gaussian hyperspherical energy (G-HE)
(Wang & Isola, 2020):
[Equation]
in the experiments for comparison.
The values of HE and G-HE can also be used as measurements on the uniformity of the embedded samples.
We expect the value to be small in order to achieve good hyperspherical uniformity.
We also want to mention that simply maximizing the distance between samples will not lead to hyperspherical uniformity, and the detailed discussion is in Sec.
B.2. Because finding the optimal solution of the HE or G-HE problem is NP-hard (Liu et al., 2018), we are not able to calculate the exact position of vectors which are evenly distributed on the sphere.
We provide a primary insight about how should finite vectors be uniformly distributed on unit hypersphere, and our conclusion is consist with the empirical results.
Since there exist many different tuple-based metric losses, analyzing all of them theoretically is impossible in this work.
In Sec. 3.1, we will provide the theoretical analysis of the triplet loss. The analysis of linear loss can be found in Sec.
B.2. In Appendix C we will show the empirical results on four popularly used tuple-based metric losses to verify our statement.
3. T RIPLET METRIC LOSSES
In this subsection, we provide our theoretical analysis on the triplet metric losses with the following assumptions.
Assumption.
Distributions p data , p pos , p tri should satisfy:
• Random positive sampling: ∀ x, y, p pos ( x, y ) =
p data ( x ) p data ( y | X x ) , where p data ( ·| X x ) is the conditional pdf of p data on the set of samples X x similar to x i.e. p data ( ·| X x ) =
p data ( · ) p data ( X x ) .
• Random negative sampling: p tri ( x, y, x − ) =
p pos ( x, y ) p − data ( x − ) , where p − data ( x − ) =
p data ( x − ) (cid:82) x − p data ( x − ) dx − • Class-balanced learning: p data ( X i ) = 1 n , then (cid:82) x − p data ( x − ) dx − = n − 1 n
where x − is a negative sample w.r.t.
x and n is the number of classes.
Definition 3.
(Triplet loss)
[Equation]
Triplet loss can be rewritten into the form of naive linear loss with a different distribution of triplets.
We consider a new distribution:
[Equation]
when
[Equation]
|
[Equation]
x
[Equation]
− )
||
22 + τ < 0 ,
Cp tri , else , where C = 1 / E ( x,y,x − ) ∼ p tri [1 {|| f ( x ) − f ( y )
||
22 −|| f ( x ) − f ( x − )
||
22 + τ ≥ 0 } ] , then
L triplet ( f,
[Equation]
) =
L linear ( f ; X, p [Equation] ) = [Equation]
x,
[Equation]
,x
[Equation]
∼ p [Equation] [ |
[Equation]
Apparently the positive part of the triplet loss is minimizing the intra-class distance under a new distribution p ′ tri , which has similar effect as the alignment loss with p tri shown in the experiments (Table 8).
Now we focus on the negative part of the triplet loss.
Theorem 1.
Denote the probability density function (pdf) of d 2 ( x, y ) :
=
|| f
( x ) − f ( y )
|| 22 w.r.t.
y ∼ p data ( ·| X x ) by q ( d 2 ( x, y )) .
Then the pdf of u = || f
( x ) − f ( y )
||
22 − || f
( x ) − f ( x ′ )
||
22 + τ with fixed x , x ′ is q ( u − τ + d 2 ( x, x ′ )) , let S ( x, x ′ )
= (cid:82) ∞ {+
+}
q ( u − τ + d 2 ( x, x ′ )) du ∈ [0 , 1] , then −
[Equation]
The negative triplet loss consists of two parts, where the first part dominants the second part because n is always large in practice.
The first part is actually a weighted unbiased regularization with weight S ( x, x ′ ) .
We think S ( x, x ′ ) may help the unbiased regularization to achieve hyperspherical uniformity.
Because the closed form of q ( d 2 ( x, y )) is intractable, it’s impossible to analyze
S ( x, x ′ ) theoretically without any assumptions.
We assume q ( d 2 ( x, y )) is exponentially distributed and show the gradient flow of negative triplet loss is asymptotically equal to the gradient of Gaussian hyperspherical energy.
Therefore in this case the negative triplet loss can lead to hyperspherical uniformity.
Proposition 2.
Assume q
( d 2 ( x, y )
[Equation]
=
[Equation]
[Equation]
, S ( x, x
[Equation]
=
1 A e − A (
[Equation]
2 ( x,x −
[Equation]
τ ) and for the network parameter θ , we have
[Equation]
the negative triplet loss has asymptotically the same gradient as Gaussian hyperspherical energy.
Despite the theoretical analysis, we also empirically show that the negative triplet loss achieves hyperspherical uniformity without any assumption on S ( x, x ′ ) .
Besides, the negative part of other metric losses are also shown to achieve hyperspherical uniformity in our empirical study.
The details is shown in Appendix C.
In summary, the tuple-based metric losses on the unit hypersphere are closely related to intra-class alignment and hyperspherical uniformity.
The positive metric losses target at minimizing the intraclass alignment and the negative metric losses try to keep all samples distributed uniformly on the hypersphere.
Connection to adversarial examples and adversarial training.
The goal of adversarial examples is to fool the neural network by reducing the model performance.
Attacking alignment loss, which positions the embedding of similar samples apart, or attacking the uniformity loss, which pulls dissimilar samples together, can definitely destroy the representation learned by DML models.
Thus alignment and uniformity loss are suitable objectives for generating adversarial examples.
D ESIGNING N EW A DVERSARIAL DML MODELS B ASED ON B ETTER U NDERSTANDING OF DML L OSS
In this section, we introduce our new adversarial DML models: adversarial DML with alignment or uniformity objective (ADML+A or ADML+U).
Before we introduce the details of our models, we share our motivation for designing ADML+A/U models by answering the following questions:
How can adversarial training helps improve DML models?
One of the most reasonable explanations is that training with adversarial examples brings additional features to neural networks.
For example, compared with clean images, adversarial examples make network representations more consistent with salient data features and human perception (Tsipras et al., 2018).
Another possible reason is that adversarial examples can be regarded as a data augmentation method, which prevents overfitting of the neural networks.
Augmentation techniques which are similar to adversarial training, e.g. using masking out (DeVries & Taylor, 2017) or adding Gaussian noise (Lopes et al., 2019) to regions in images, can help to achieve better performance on image recognizing tasks.
Why do we need new objectives for adversarial DML models?
Based on our analysis in Sec. 3, the DML embedding of each image is depend on all other positive and negative samples from perspective of alignment and uniformity objective.
Therefore if we want to generate the adversarial sample x ′ for one image x , we need to push the adversarial sample away from the similar samples of x (maximize the alignment loss w.r.t.
x ′ ), or pull the adversarial sample close to the dissimilar samples of x (maximize the uniformity loss w.r.t.
x ′ ).
Currently, the existing adversarial DML models (Duan et al., 2018; Panum et al., 2021) generate adversarial samples by attacking the triplet loss (Definition 3).
In this case, only one positive and one negative samples are used to generating adversarial sample x ′ , which is obviously less powerful than the alignment/uniformity objective (which utilizes more positive or negative samples).
Our experimental results in Table 1 also show the adversarial examples generated by alignment/uniformity objectives is more powerful than the triplet objective.
Thus it is critical to design new objectives for attacking DML models, which can take advantage of the representation information from more data samples.
Adversarial training.
We first recall the standard tuple-based DML training setting.
Denote the metric loss function by L ( · ; θ ) , where θ is the model parameters, our learning objective is:
[Equation]
In the regular adversarial training framework (Madry et al., 2017), we train networks with perturbed samples from distribution p ( adv )
[Equation]
As our goal is to improve the DML performance on clean images by leveraging the regularization power of adversarial examples, we treat adversarial images as additional data augmentations and train networks with a mixture of adversarial examples and clean images.
Our learning objective is
[Equation]
where λ is the strength of the adversarial training.
[Table]
Generate adversarial samples.
We use l ∞ PGD-FSGM (Madry et al., 2017) method for generating adversarial samples.
We consider DML models require class-balanced batches for training [Table] and propose to generate perturbations by maximizing the intra-batch alignment or uniformity.
Given a batch S and a sample x , the l ∞ FSGM adversarial sample of x generated by maximizing the alignment objective is given by,
[Equation]
(6) where Π B ∞ ( x,ϵ ) is the projecting function on the l ∞ ball centering at x with radius ϵ [Table] and α is the attack strength.
Analogously, the adversarial sample generated by maximizing the uniformity objective (Eq. 2 and Eq. 3) is [Equation]
[Equation]
In PGD-FSGM method, we will update the adversarial samples iterative by x ( l +1) =
ADV ( x ( l ) ) for
L steps.
The output perturbed samples x ( L ) will be used in our adversarial training objective Eq. 5.
In ADML-A we use alignment loss (Eq. 1) to generate adversarial samples and in ADML-U we use
Gaussian uniformity loss G-HE (Eq. 3).
We include the algorithm of ADML-A and ADML-U in the appendix (Alg. 1 and Alg. 2).
In experiments, we apply multi-similarity losses as the metric loss L with attacking alignment and uniformity objective, both models achieve significantly better results on benchmarks (Table 2 and Table 3).
E XPERIMENTS
In our experiments, we first conduct the empirical studies to verify the theoretical analysis results in
Sec. 3 (the results are discussed in Appendix C).
After that, we show that alignment and uniformity objectives can help to generate better adversarial examples than the triplet loss, then we compare the natural and robust performance of our adversarial DML model with the state-of-the-art methods.
The natural performance is the performance of DML models evaluated with clean samples, while the robust performance is evaluated with adversarial samples.
5. E XPERIMENTAL SETUP
Datasets.
We test our model on four DML benchmarks, CUB200-2011 (Wah et al., 2011),
CARS196 (Krause et al., 2013), Online-product (Song et al., 2016), and In-shop (Liu et al., 2016).
We follow the previous work (Song et al., 2016) and (Liu et al., 2016) for the train-test split.
The statistics of these datasets is introduced in Sec.
D.
Training Frameworks.
In all experiments, we use the DML framework from (Roth et al., 2020) for training.
This framework enables us to train and evaluate DML models under the same settings and ensure fair comparison of the model performance.
The backbone network is ResNet50 (He et al., 2016) with ImageNet pretrained (Krizhevsky et al., 2012) and frozen Batch-Normalization layers, the embedding dimension of samples is 128.
The initial learning rate is 0.00001 with no scheduling and the batch size is 112.
Experiments are performed on a 24GB Nvidia Tesla P40.
Baseline models.
We compare ADML+A/U with the state-of-the-art DML models.
First we select three of the best DML models according to (Roth et al., 2020): margin loss with distance sampling (Wu et al., 2017), multisimilarity loss (Wang et al., 2019) and triplet loss with distance sampling.
Besides, we take another two SOTA DML models which were published recently but not included [Table] in Roth et al.
(2020)’s work: Proxy-Anchor loss (Kim et al., 2020) and Cross-Entropy loss (Boudiaf et al., 2020).
Next we apply the Info-NCE loss (Wang & Isola, 2020), which is a contrastive learning objective, as one of the baselines.
Finally we compare our models with the only existing adversarial
DML model (Duan et al., 2018), ADML+T (triplet objective), in the experiments.
Evaluation Metrics.
We measure the performance of DML and ADML models with Recall@k (R@k) (Jegou et al., 2010), Normalized Mutual Information (NMI) (Christopher et al., 2008) and
Mean Average Precision measured on recall (mAP@k) (Musgrave et al., 2020).
The details of these metrics are introduced in Sec.
D.3.
5. C OMPARE THE QUALITY OF ADVERSARIAL EXAMPLES WITH DIFFERENT OBJECTIVES
Settings.
The threatened model is a pretrained DML model with triplet loss and distance sampling (Wu et al., 2017), which is one of the most competitive DML models according to Roth et al.
We consider three different attack objectives, triplet loss (Eq. 4), alignment loss (Eq. 1), and
Gaussian hyperspherical uniformity (Eq. 3), for generating adversarial samples.
We use l ∞ PGDFSGM attacks with strength ϵ = 0 .
L = 7 steps and step size α = 0 .
007 , we keep this settings for all three attack objectives in our experiments.
Results.
In Table 1, the adversarial samples generated by alignment or uniformity objectives are significantly stronger than the samples generated by triplet loss.
This indicates that adversarial samples from alignment or uniformity contain more features that are not captured by the vanilla DML models.
Thus we believe the Adversarial DML model with alignment or uniformity objectives could be more generalized than the vanilla DML models or ADML with triplet loss.
Our experimental results in Table 2 and Table 3, which show that ADML+A and ADML+U outperform the baseline models on metric learning tasks, also support our analysis.
We also notice that using both A and U in ADML have similar performance as ADML+T, which suggests we should use the attack objective (A or U) separately.
In Appendix F we also shows the T-SNE plot of the embedding generated by the vanilla DML and ADML+A, which shows ADML+A can better separate different classes.
5. A DVERSARIAL DML MODELS IMPROVE NATURAL PERFORMANCE
Settings.
We train all DML models for 100 epochs.
For our adversarial DML models we apply
ADML+A and ADML+U.
The adversarial training strength λ in ADML+A/U is 0.1 for CUB2002011 and 0.15 for CARS196.
For generating adversarial examples we use l ∞ PGD-FSGM attacks with strength ϵ = 0 . 0314 , L = 7 steps and step size α = 0 .
For Online-products and In-shop we use λ = 0 .
005 , ϵ = 0 .
01 , L = 5 , and α = 0 .
The ImageNet model is the model only with
ImageNet pretrain.
Results.
In Table 2, our ADML+A and ADML+U models outperform the SOTA metric learning models.
ADML+A/U improve the R@1 over 1% on CUB200-2011 and CARS196, and also have considerable improvement on Online-product and In-shop dataset.
Besides, the performance of
ADML+A/U under the NMI and mAP@C metrics is also comparable or better than the SOTA. [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure]
Since all the models are training under the same framework and settings, we can conclude that adversarial training helps to enhance the natuaral performance of DML models.
5. R OBUSTNESS PERFORMANCE OF A DVERSARIAL DML MODELS
We evaluate the robustness performance of our ADML+A and ADML+U models against attacking the alignment objective, which is the strongest DML attack according to our experiment in Sec.
5.2, on CUB200-2011 and CARS196 datasets.
For baseline models we use margin, multi-similarity, and
ADML+Triplet models.
The settings of alignment attack are the same as the settings in Sec.
5.2, the settings of ADML+A/U and baseline models are the same as the settings in Sec.
As seen in
Table 4, ADML+T has slightly better performance than the vanilla DML models, while ADML+A and ADML+U outperform the baseline models with a large margin under the alignment attacks.
Notice ADML+A takes the advantage of adversarial training with alignment loss, it’s reasonable that the performance ADML+A is slightly better than ADML+U under alignment attacks. [Table] Online- [Table]
[Table]
5. A BLATION STUDY
Strength of adversarial training.
In this part, we evaluate the effect of adversarial training strength λ on the performance of ADML+A and ADML+U. The backbone is DML with multi-similarity loss. We expect the performance will first increase with λ then decrease, because when λ is close to 0, the ADML models can hardly learn adversarial features and the improvement is small, when λ is large, the adversarial features will dominate the DML models and the performance on clean features will be poor.
The model settings are the same as in Sec.
The experimental results illustrated in Fig.
1 are consistent with our analysis, when increasing the adversarial training strength, the natural performance of ADML+A and ADML+U is first improved then decreased.
Table 5 shows the robustness of ADML+A and ADML+U model with adversarial training strength λ and alignment attacks.
Both models become increasingly robust against alignment attacks.
ADML on different metric loss.
In this experiment, we apply our ADML approach with triplet, alignment, uniformity, and alignment+uniformity objectives on different metric losses, including triplet, margin, multi-similarity, and info-NCE losses.
The metric is Recall@1.
The model settings are the same as in Sec. From Table 6, we can observe that all ADML methods boost the performance of all DML losses. ADML+A achieves the most significant improvement across all attacks and metric losses. [Table]
A mixture of alignment and uniformity attacks.
In this experiment, we study the effect of different weight of alignment and uniformity objectives in ADML+A+U.
The backbone is DML with multisimilarity loss.
The model settings are the same as in Sec.
Specifically, we assign weight (1 − β ) to the alignment objective and β to the uniformity objective.
The results are shown in Table 7.
When β increases, the performance of the ADML model first decreases and then increases, which indicates that attacking alignment and uniformity loss separately can lead to better results.
[Table]
C ONCLUSION
In this work, we investigated two important properties, intra-class alignment and hyperspherical uniformity, of tuple-based metric losses on unit sphere.
According to our theoretical analysis and experimental results, the positive metric losses contribute to the intra-class alignment and the negative metric losses achieve hyperspherical uniformity.
Based on our new understanding, we design two novel adversarial DML models, ADML+A and ADML+U, where the perturbations are generated by maximizing the alignment loss or the uniformity loss.
Our ADML+A and ADML+U improve both of natural and robust DML performance by enhancing model generalization.
Potential future work directions include analyzing other tuple-based metric losses theoretically, designing new metric losses based on the alignment and uniformity, and exploring the trade-off between nature and robustness performance in metric learning.
E THICS S TATEMENT
In this paper we design new adversarial deep metric learning (DML) models for improving the vanilla DML.
Our method can boost the robustness of neural networks again adversarial attacks, which contributes to the trustworthy AI and machine learning.
R EPRODUCIBILITY
All datasets, baseline models, general training settings are provided in Sec. 5.1.
For specific tasks we also include the detailed settings in the corresponding sections.
For example, the detailed model structure and hyperparameter settings for the natural performance of ADML models are written in
Sec.
We will release the code if the paper is accepted
A PPENDIX A C ODE
We use the open source DML training framework https://github.com/Confusezius/
Deep-Metric-Learning-Baselines .
We include the code of our Alg. 1 and Alg. 2 in the supplementary material.
All datasets used in this paper are available online.
We will release the full code package if our paper is accepted.
A PPENDIX B T HEORETICAL EXPLANATION OF POSITIVE AND NEGATIVE METRIC LOSSES B. N UMBER OF NEARLY ORTHOGONAL VECTORS IN HIGH DIMENSIONAL SPHERE .
Lemma 1. (Kabatjanskii-Levenstein bound (Kabatjanskii & Levenstein, 1978)) For a hypersphere
S k − 1 ∈ R k , there exist at least k M vectors with pairwise distance in the range of √ 2 ±
O ( (cid:113) M log k k ) .
Lemma 1 shows the number of nearly orthogonal vectors we can take from a high dimensional sphere.
If k = 128 and M = 3 , the amount of nearly orthogonal vectors could be more than one million, which exceeds the volume of benchmark datasets X in DML a lot.
Following the former discussion, we know the pairwise distance cannot be larger than √ 2 for all negative pairs.
Therefore we think the uniformity regularization with DML benchmarks will lead to an embedding space where the feature vectors are nearly orthogonal.
The plots of pairwise distance distribution in Fig.with hyperspherical regularization also support our analysis.
B. N AIVE LINEAR METRIC LOSSES
With the same Assumption in Sec.
3.1, we can prove that the positive part of the linear loss minimizes the intra-class distance and the negative part maximizes an unbiased term, which doesn’t achieve hyperspherical uniformity.
The theoretical analysis results are summarized in the following theorem.
Definition 4.
(Naive linear loss)
[Equation]
Definition 5.
(Unbiased regularization)
[Equation]
the minimum of this loss is reached when the centroid coincide with the origin.
Theorem 2.
Naive linear loss consists of alignment loss and unbiased loss with a constant multiplier.
Positive
[Equation]
Negative part
[Equation]
− E
[Equation]
y
[Equation]
x
[Equation]
unbiased
[Equation]
Combining them we have L linear =
nn − 1 (2 L unbiased + L alignment − 2) .
Because the number of classes n is always large and the magnitude of L unbiased − 1 and L alignment are similar, in negative linear loss the dominant objective is the unbiased regularization.
Thus simply maximizing the distance between negative pairs will lead to the unbiased regularization instead of hyperspherical uniformity.
In Figure 2(e) and Figure 2(c) of the appendix, we can observe the difference between the models with negative linear loss and uniformity regularization.
We also show the interesting connection between the naive linear loss and linear discriminant analysis (LDA) in
Sec.
E.5.
Remark.
Naive linear loss doesn’t work at all in practice (see the experimental results of Linear model in Table 2 and Table 3).
Based on our analysis, we believe it’s because linear loss doesn’t optimize the hyperspherical uniformity.
In the next section, we will introduce our theoretical analysis of triplet loss, which is a simple variant of linear loss.
We find triplet loss optimize the hyperspherical uniformity, which could be the reason that triplet loss works well empirically.
A PPENDIX C E MPIRICAL STUDY OF POSITIVE AND NEGATIVE METRIC LOSSES
In this subsection [Table] we study the effect of tuple-based metric losses on positive or negative pairs empirically, and focus on the four tuple-based losses: naive linear, triplet, margin, and multi-similarity loss.
In experiments we train all DML models with either positive metric losses or negative metric losses.
C. DML MODELS WITH POSITIVE METRIC LOSSES
Settings .
We train 4 DML models with linear, triplet, margin, and multi-similarity losses on the positive sample pairs.
The gradient flow of negative pairs is stopped.
We train all models forepochs.
We compare the average distance between positive/negative/all pairs of embedded samples.
Results.
In Table 8, the DML models trained with only positive metric losses have average pairwise distance close to 0.
Therefore, minimizing the intra-class alignment will lead to a model which maps all samples to the same feature vector [Table] [Table] .316e
[Table]
2.242e
[Table]
264e
[Table]
[Table]
277e
[Table]
C. C OMPARE NEGATIVE METRIC LOSSES WITH UNIFORMITY REGULARIZATION
Settings.
We compare 8 different models in this experiment, including a) the original model with only ImageNet pretrain; b) four models trained with linear, triplet, margin, and multi-similarity losses on negative pairs; c) three models trained with HE(s=0), HE(s=1)
(Eq. 2) and G-HE(s=1)
(Eq. 3) regularization, those regularization functions are introduced in Appendix C. We train all models with 50 epochs.
For the models with negative metric losses, the gradient flow of positive pairs is stopped.
Evaluation Metrics.
We use the regularization score on HE(s=0)
(Eq. 2) and G-HE(s=1)
(Eq. 3) to measure the uniformity of embedded samples on hypersphere.
Smaller score indicates better uniformity.
We also check the average of pairwise distance of different models, we expect it to be close to √ 2 for good hyperspherical uniformity.
Results.
In experiments we compare the regularization strength of negative metric losses with HE and G-HE.
Fig.
2 and Fig.
3 illustrates the pairwise distance of the test samples on CUB200and CARS196 dataset.
The DML models with negative metric losses have similar pairwise distance distributions as the uniformity regularization methods, and the only exception is the negative naive linear loss, which will not lead to the hyperspherical uniformity based on our analysis in Sec.
B.2. We also include the distance distribution before training (Figure 2(a)) as a reference.
Besides, we also compare those models under the hyperspherical uniformity metrics.
In Table 9, the negative metric losses achieve comparable results with the uniformity regularization methods. The G-HE(s=1) outperforms the rest models. [Table]
[Figure]
models trained with negative metric losses or uniformity regularization.
The details of models and training settings are in Sec. C.2.
[Figure]
A PPENDIX D ADML ALGORITHMS AND EXPERIMENTAL SETTINGS D. A LGORITHM OF ADML+A AND ADML+U Algorithm
Input: training set X ; number of epochs N ; original classifier f ; weight of adversarial training λ ;
PGD attack step L ; PGD attack strength ϵ
Initialize class balanced sampler S ; for i ∈ epochs do for S ∈ mini-batch { S 1 , ..., S n } do
S (0) =
S ;
Generate adversarial samples of S with PGD-FSGM attack for t ∈ 0 :
L − 1 do
S ( t +1) =
Π B ∞ ( S (0) ,ϵ ) ( S ( t ) + α ∇ S ( t )
L alignment ( f, S ( t ) )) ; end
S adv = S ( L ) ;
Calculate objective function
L total =
L ( f, S ) + λ L ( f, S adv ) ;
L can be an arbitrary metric loss, in our paper we use margin and multisimilarity loss
Update network parameters of f with L total ; end end
Input: training set X ; number of epochs N ; original classifier f ; weight of adversarial training λ ;
PGD attack step L ; PGD attack strength ϵ
Initialize class balanced sampler S ; for i ∈ epochs do for S ∈ mini-batch { S 1 , ..., S n } do
S (0) =
S ;
Generate adversarial samples of S with PGD-FSGM attack for t ∈ 0 :
L − 1 do
S ( t +1) =
Π B ∞ ( S (0) ,ϵ ) ( S ( t ) + α ∇ S ( t ) L uniformity ( f, S ( t ) )) ; end
S adv = S ( L ) ;
Calculate objective function
L total =
L ( f, S ) + λ L ( f, S adv ) ;
L can be an arbitrary metric loss, in our paper we use margin and multisimilarity loss
Update network parameters of f with L total ; end end
D. D ATASET D ETAILS
• CUB200-2011 contains 200 species of birds and 11,788 images (Wah et al., 2011).
We use the first 100 species as training set and the rest as test set.
• CARS196 has 196 models of cars and 16,185 images.
(Krause et al., 2013).
We use the firstmodels as training set and the rest as test set.
• Online-product includes 22,634 classes of products and 120,053 images (Song et al., 2016).
We use the first 11,318 classes as training set and the rest as test set.
• In-shop contains 7982 classes of clothing and 54,624 images (Liu et al., 2016).
We use the first 3,997 classes as training set and the rest as test set.
The test set is further partitioned into a query set with 14,218 images of 3,985 classes and a gallery set with 12,612 images of 3, classes.
D. E VALUATION M ETRICS
We use the following metrics to evaluate the DML models with retrieval and clustering downstream tasks.
Recall@k.
For the retrieval task we apply the Recall@k (R@k) metric (Jegou et al., 2010).
For a test set M := { ( x 1 , y 1 ) , · · · , ( x n , y n ) } , the indices of the first k nearest neighborhood of a sample x i is given by S k ( x i ) :
= arg max | S | =
k (cid:80) j ∈ S,j ̸
= i ||
f ( x i ) − f ( x j )
|| 2 , and
R@k := 1 n n (cid:88) i = 1 { ∃ j ∈ S k ( x i ) ,y j = y i } .
NMI.
We use Normalized Mutual Information (NMI) (Christopher et al., 2008) to measure the quality of the clustering task.
We use K-means to generate the clusters of the embedded samples, then we calculate the label assignment Γ = { γ 1 , ..., γ n } from clustering.
Denote the ground truth labels by Ω
=
{ y 1 , ..., y n } , the NMI is computed as
[Equation]
where I ( · , · ) is the mutual information function and H ( · ) is the entropy function.
mAP@C.
According to (Musgrave et al., 2020), we also include mean average precision measured on recall (mAP@k) as an additional metric.
We first compute the recalled samples, which are determined by the k nearest neighbour ranking.
Then compute the mAP-score follows the standard mAP procedure.
mAP@C is the mean over the class-wise average precision@ k c , where k c is the number of samples in class c , which means we only recall k c nearest neighbour.
Following the notation of Recall@k, the value of mAP@C is given by mAP @
C :
= 1 n (cid:88) c ∈ C (cid:88)
y
q =
c |{ x i ∈ S k c ( x q ) | y i
=
y q }| k c
. A PPENDIX E M ISSING P ROOFS E. P ROOF OF P ROPOSITION
Denote the support set of the distribution of each class by S 1 , . .
. , S n , according to Definition 1, the minimum of alignment loss is reached when the encoder f ∗ maps all samples in one class to the same feature vector i.e. ∀ i, f ∗ ( S i )
= { v i } .
For arbitrary i, j , because ∪ nk =1 S k is connected and each S k is closed, we can select a set sequence S k 0 , . . .
, S k m such that S k
=
S i , S k m = S j and
S k l ∩ S k l +1 ̸ = ∅ , l ∈ [ m − 1] .
By S k l ∩ S k l +1 ̸ = ∅ we have v k l = v k l +1 for all l ∈ [ m − 1] , thus ∀ i, j, f ∗ ( S i ) =
f ∗ ( S j ) .
So all samples are projected to the same feature vector.
E. P ROOF OF T HEOREM
Proof.
Recall that the naive linear loss is given by
[Equation]
Consider the positive part
[Equation]
Consider the negative part
[Equation]
and
[Equation]
where we denote the class of sample x by X x
[Equation]
Combining Eq.
8 and Eq.
9 together, we have
[Equation]
E. P ROOF OF T HEOREM
Proof.
The triplet loss is
[Equation]
Consider the negative part
[Equation]
where X x is the set of samples have the same label as x .
The last equation is based on
[Equation]
x ) [1 {|| f ( x ) − f ( y ) ||
22 −|| f ( x ) − f ( x ′ )
||
22 + τ ≥ 0 } ]
E. P ROOF OF P ROPOSITION
Proof.
By q ( d 2 ( x, y ))
=
1 A e − Ad 2 ( x,y ) , the pdf of u = d 2 ( x, y ) − d 2 ( x, x ′ )
+ τ is 1 A e − A ( u + d 2 ( x,x ′ )
− τ ) , then
[Equation]
Consider the gradient of the negative triplet loss E ( x,y,x − ) ∼ p ′ tri [ || f ( x ) − f ( x − ) || 22 ] , during training we first sampling from p ′ tri then calculate the gradient.
In this case the actual gradient flow is given by
[Equation]
Analogous with the discussion in Sec.
E.3, we have
[Equation]
E. C ONNECTION BETWEEN NAIVE LINEAR LOSS AND LDA
The intuition of multiple linear discriminant analysis (LDA) is to maximize the inter-class variance while minimizing the intra-class variance.
In this section we will show linear metric loss have a similar effect
[Equation]
Definition 6.
(Total variation)
For a random vector x , the total variation is
TV
[Equation]
Definition 7.
(Centroid) Define the centroid of samples in an arbitrary set Y by
[Equation]
Proposition
[Equation]
(Intra-class total variation)
The within-class variation
[Equation]
is proportional to alignment loss
[Equation]
Proof.
[Equation]
Proposition 4.
(Inter-class total variation)
The inter-class total variation
[Equation]
where c is the centroid of all samples, is proportional to triplet loss
[Equation]
Proof.
TV
[Equation]
Firstly, c
[Equation]
Hence c T c =
L unbiased ( f ) .
Next,
[Equation]
Thus E x ∼ p data [ c TX x c X x ] = 1 − 12 L aligned ( f ) , and TV inter ( X ) = 1 − 12 L aligned ( f ) −
L unbiased ( f ) = − n − 1 2 n L linear ( f )
Proposition 5.
(Total variation of the dataset)
The total variation of dataset X
TV
[Equation]
[Equation]
:= E
[Equation]
∼
[Equation]
[Equation]
x − c )
[Equation]
( x − c )] , where c is the centroid of all samples, is proportional to the unbiased loss
[Equation]
Proof.
[Equation]
We can also check if TV total ( X )
=
TV within ( X ) + TV between ( X ) holds to validate the proofs above.
A PPENDIX F T-SNE EVALUATION OF ADML+A AND VANILLA DML.
In experiments, we visualize the embedding of the first 10 classes of CUB200-2011 generated by a vanilla DML model (with multi-similarity loss) and an ADML+A model (with multi-similarity loss).
The Recall@1 of the DML model and the ADML+A model are 62.71 and 66.13, respectively.
Fig.
4 (a) plots the T-SNE result for vanilla DML and Fig.
4 (b) plots the T-SNE result for ADML+A.
Comparing two figures, we can see that ADML+A has better separation on the (red, green, orange) samples.
[Figure]