MetricFormer: A Unified Perspective of Correlation Exploring in Similarity Learning
Abstract
Similarity learning can be significantly advanced by informative relationships among different samples and features.
The current methods try to excavate the multiple correlations in different aspects, but cannot integrate them into a unified framework.
In this paper, we provide to consider the multiple correlations from a unified perspective and propose a new method called MetricFormer, which can effectively capture and model the multiple correlations with an elaborate metric transformer.
In MetricFormer, the feature decoupling block is adopted to learn an ensemble of distinct and diverse features with different discriminative characteristics.
After that, we apply the batch-wise correlation block into the batch dimension of each mini-batch to implicitly explore sample relationships.
Finally, the feature-wise correlation block is performed to discover the intrinsic structural pattern of the ensemble of features and obtain the aggregated feature embedding for similarity measuring.
With three kinds of transformer blocks, we can learn more representative features through the proposed MetricFormer.
Moreover, our proposed method can be flexibly integrated with any metric learning framework.
Extensive experiments on three widely-used datasets demonstrate the superiority of our proposed method over state-of-the-art methods.
Introduction
Exploring how to effectively learn semantic similarities among examples is an important problem in the field of machine learning and computer vision.
Metric learning aims to transform the original data into an embedding space and learn a distance metric in this space to promote intra-class compactness and inter-class separability.
In recent years, benefiting from the emerging advance of deep learning technique [20, 17], deep metric learning (DML) can capture more representative embeddings and has been widely applied to various downstream tasks including image retrieval [29, 28], face recognition [19, 36], and person re-identification [57].
The core idea of DML is to excavate the intrinsic structural relations of data, and utilize them to learn discriminative embeddings.
As shown in Figure 1, such relations can be divided into two main categories: sample relationship and feature relationship.
The former, i.e. sample relationship, includes pairwise correlation and batch-wise correlation .
Most of existing supervised DML methods utilize pairwise correlations ( i.e. , positive and negative pairs) to construct and utilize valid tuples of pairs through well-designed pair-based losses, e.g. triplet loss [36] and multi-similarity loss [46].
Due to training with random sampling, such methods suffer from heavily low convergence and model degradation.
To discover the implicit global structure of the dataset, recent studies started to explore [Figure] the batch-wise correlation shown in Figure 1 (b) in the embedding space [18, 12, 1, 49, 37].
For example, IBC [12] adopt the message passing strategy to capture the overall structure of data in each mini-batch.
Considering that using only a single vector for similarity measurement cannot fully capture the semantics of inputs and exacerbates the unexplainability caused by the black-box nature of deep neural networks, recent studies [22, 29, 56] learn an ensemble of embeddings to jointly represent the input instance.
Such ensemble-based method generally outperforms those that use a single feature vector since the ensemble encourages the learned embeddings to capture more comprehensive characteristics.
However, simply concatenating the ensemble of sub-features ignores the structural relations among them, i.e. feature-wise correlation shown in Figure 1 (c), which is crucial to ensure each sub-feature modeling different and discriminative characteristics.
For better performance and generalization ability, the feature-level correlation should also be considered during training.
In this paper, we provide a unified perspective to learn the multiple correlations and propose a metric transformer framework, named MetricFormer, which enables the information propagation along features of different samples within each mini-batch.
Specifically, we introduce a correlation-aware transformer into the internal structure of deep metric networks, which contains a feature decoupling block, a feature-wise correlation block, and a batch-wise correlation block.
We adopt the feature decoupling block with several trainable query vectors to learn a series of sub-features with different discriminative characteristics.
Meanwhile, the feature-wise correlation block is performed to excavate the intrinsic relations among sub-features, which can ensure the learned ensemble of features is representative and sufficient for similarity measurement.
Furthermore, to implicitly capture and model the sample relationships, we apply the batch-wise correlation block into the batch dimension of each mini-batch with the learned ensemble of features.
Moreover, a local embedding module is proposed to focus on more informative neighborhood relationships without redundancy.
Guided by the sparse graph, we can encode the local neighborhood relationships of each sample into the transformer feature.
Our work delivers the following contributions:
• We propose to simultaneously explore the multiple correlations among data from a unified learning perspective.
• We provide a simple yet effective method termed as MetricFormer to implicitly capture three levels of correlations in an end-to-end model.
• We achieve significant performance gains against the state-of-the-art methods on three widely-used benchmark datasets.
Preliminaries 2.1 Problem Formulation of DML
Given the training dataset D = { ( x i , y i ) } Ni =1 where y i is the corresponding label of the sample x i , DML methods aims to learn a feature mapping f : X (cid:55)→ Z ∈ R D with deep neural networks, which maps input samples to a D -dimensional embedding space Z .
With the learned projection f : z i = f
( x i ) , we can effectively measure the similarity between any two samples x i and x j using
Euclidean distance
[Equation]
or cosine similarity
[Equation]
To improve the ability of similarity measurement, a majority of DML studies develop various metric loss functions L M , which can impose a discriminative constraint on the feature embeddings.
2.2 Revisiting Transformer Encoder
Transformer architecture has achieved great success in the field of natural language processing and computer vision and brings novel paradigms for several fundamental tasks ( e.g. translation and segmentation).
The overall transformer model usually consists of a stack of multiple transformer encoder blocks.
The transformer encoder contains a multi-head self-attention layer (MHSA) and a feed-forward network (FFN), which are followed by a LayerNorm (LN), respectively.
Given a sequence of input features X ∈ R M × D with the length of the sequence M and the dimension of the input features D , we can derive the output of the transformer encoder as follows,
[Equation]
where l ≥ 1 is the index of layers in the transformer encoder, and Z 0 = X .
The self-attention mechanism is widely-used to capture and model the relationships from the channel and spatial dimensions.
With Q , K , V ∈ R M × D as the query, key, and value learned from the same input, we can calculate the output Z for the self-attention module:
[Equation]
And then, the representations from different heads are concatenated.
The Proposed Method
In this section, we first introduce the overall motivation and framework of the proposed Metric
Transformer, and then describe the three transformer blocks, respectively.
3.1 Metric Transformer
Since the relationships among samples and features are both various and complex, how to effectively discover the intrinsic structure of data and excavate more representative information from the training data is an essential problem in DML.
Most of existing methods can only utilize the pair-based similarity, which neglects the batch-wise correlation and feature-wise correlation.
Recently, some studies are proposed to mine such rich relations.
However, they can only learn the batch-wise or feature-wise correlations individually, and the multiple correlations cannot be automatically learned in a unified end-to-end DML model.
[Figure]
After obtaining the feature maps generated by a backbone network, we can derive the ensemble of sub-features to represent the input samples from different aspects in the feature decoupling block.
And then, the batch-wise correlation block and feature-wise correlation block with the same architecture employ the cross-attention mechanism from different dimensions to explore the relationships among samples and features.
To tackle the inconsistency issue, we utilize the consistency preserving strategy during training.
We introduce a metric transformer (MetricFormer) to enable DML model itself with the ability to learn the sample relationships in each mini-batch and the structural relationships among the ensemble of sub-features.
As shown in Figure 2, our MetricFormer consists of one feature decoupling block, one feature-wise correlation block, and T batch-wise correlation block.
Specifically, we adopt a backbone network to extract the feature maps from input samples.
After that, a feature decoupling block is performed to split the feature maps into several sub-features with different discriminative characteristics.
And then, the batch-wise correlation block is employed to model the relationships among different samples by utilizing the cross-attention mechanism in the transformer.
Finally, we use the feature-wise correlation block to discover the structural pattern in the feature ensembles and obtain the aggregated features guided by the learned feature-wise relations.
Consistency Preserving Strategy Considering the inconsistency of input data structure between training and testing as the mini-batch of data is not available in testing, we apply a consistency preserving strategy, which forces the out features learned with the batch-wise block to be consistent with the ones learned without the batch-wise block.
By doing this, we can remove the batch-wise correlation block during the test, while still benefiting from the sample relationship learning.
3.2 Feature Decoupling
Compared with learning the single embedding vector, employing an ensemble of embeddings with sufficient diversity can provide different characteristics of the input samples and improve the performance of DML.
To achieve this, we exploit the self-attention mechanism with several trainable queries to automatically split the feature map into an ensemble of several distinct and diverse subfeatures.
The learned ensemble of features can comprehensively characterize the input sample from different aspects.
Specifically, we apply a backbone network to extract original feature maps J ∈ R H × W × C , where
H, W, C denotes the height, width, and the number of channels, respectively.
After that, we use two different linear layers g 1 ( · ) and g 2 ( · ) to process the feature map tensor J and derive the corresponding key and value matrices in the feature decoupling block as follows:
[Equation]
where C K denotes the numbers of feature maps in the key matrix, and d is the embedding dimension of the learned sub-features.
Here, we set d =
[ D/k ] according to the number of sub-features k and the final embedding dimension D .
To automatically learn the representative ensemble of features with the self-attention mechanism, we introduce a set of trainable query vectors Q 1 =
[ q 1 ; q 2 ; · · · ; q k ] ∈ R k × C K , where k is the number of sub-features.
These query vectors are randomly initialized and adaptively updated along with the other parameters in our model.
To do so, we can calculate the attention score matrix as
[Equation]
which measures the importance of spatial positions to guide the learned sub-features to identify different spatial positions.
To this end, we can obtain the ensemble of features
[Equation]
Different rows of Y represent the learned sub-features with different characteristics.
Moreover, we employ a diversity loss to encourage the diversity between every two sub-features as follows:
[Equation]
where s i,j is the cosine similarity between two sub-features Z
i and Z
j of the same sample.
And the metric loss is applied to each sub-feature of training samples separately to ensure that the learned ensemble of features can effectively represent the inputs from different aspects.
3.3 Multiple Correlations Exploration
[Figure]
The feature decoupling block enables the spatial information propagation and learns the ensemble of sub-features with discriminative characteristics from different aspects.
With the learned ensembles of features, we can further explore the information propagation among different samples within each training mini-batch and different sub-features within each ensemble.
Both of them can significantly improve the performance of DML.
To effectively capture the multiple correlations, we design two transformer encoder blocks with the same architecture, i.e. batch-wise correlation block and feature-wise correlation block.
As shown in Figure 3, if we treat each sample in the mini-batch as a node of a sequence, the batch-wise correlation block can implicitly mine the relationships among samples in the mini-batch.
While if we regard each sub-feature of the sample as a node of a sequence, the structural relations among the sub-features through the feature-wise block can be explored.
Considering a set of features
F ∈ R B × k ×
d in a mini-batch, we can calculate the corresponding query, key, and value tensors
Q , K , V ∈ R B × k × d .
After that, we apply the attention mechanism from different dimensions.
Batch-wise Correlation Block To explore the sample relationships in each mini-batch, we generalize the attention mechanism into the batch dimension as follows:
[Equation]
where Q : i : , K : i : , V : i : ∈ R B × d .
The attention score matrix A B can measure the sample relationship in this mini-batch.
To further enhance the local manifold embedding, we introduce a local embedding graph G defined as G i,
[Equation]
= (cid:26) s i,j if x i ∈ N ( x i , n ) and s i,j ≥ ϵ, otherwise , (11) where N ( x i , n ) denotes the set of top- n samples of x i according to the rank list { s i,j } Bi =1 , and ϵ is the threshold.
And then, the attention score matrix can be updated as follows:
[Equation]
where ⊙ denotes the element-wise product operation.
To this end, we can derive the output embeddings:
[Equation]
Feature-wise Correlation Block To excavate the structural relations among different sub-features, we apply the attention mechanism from the feature dimension.
For each sample x i ( i = 1 , 2 , · · · , B ), we treat its ensemble of sub-features as a sequence, i.e. , we have B sequences with the length of k .
Specifically, the output embeddings are learned according to the query, key and value matrices
Q i :: , K i :: , V i :: ∈
R k × d in the position of the i -th sample:
[Equation]
Related Work
Deep Metric Learning Recently, deep metric learning has been well studied and achieved great advances [6, 36, 27, 47, 3, 4, 54, 53].
The majority of works focus on designing discriminative metric loss functions [14, 28, 46, 39, 21, 31, 38] and hard mining strategies [11, 10, 15, 48, 52, 50, 9] for training more effectively.
Recently, some methods consider to learn an ensemble of embeddings to capture more characteristics of inputs from different aspects [22, 52, 35, 26, 56].
Nevertheless, the rich relationships among samples and embeddings are usually ignored in the existing methods.
Transformer
In recent years, based on multi-head self-attention mechanism [42], transformers, represented by BERT [7] and ViT [8], develop rapidly and have been widely used in the field of natural language processing and computer vision.
Visual transformer [8, 25] has gradually become a new backbone for computer vision tasks, including classification [8, 25], detection [2], segmentation [55] and representation learning [5, 16].
Experiments
In this section, we evaluate the effectiveness of our method on three widely-used datasets on image clustering and retrieval tasks.
5.1 Experimental Settings
[Table]
SOP
[Table]
Datasets To evaluate the performance of the proposed
MetricFormer, we conduct the experiments on three widelyused benchmarks CUB-200- (CUB) [43], Cars196 [24] and
Stanford Online Products (SOP) [28].
Following the standard protocol [38, 28], we split the training and testing sets without intersection.
As shown in Table 1,
CUB-200-2011 contains 200 bird species of 11 , 788 images.
The first 100 categories of 5,864 images are used as the training set and the remaining 5,924 images from the other 100 categories are used for testing.
Cars196 includes 16,185 images from 196 car categories.
The first 98 classes (8,054 images) are used for training and the rest 98 classes (8,131 images) for testing.
Standard Online Products contains 120,035 images from 22,634 products.
We use 59,551 images of the first 11,318 products for training and 60,502 images of the remaining 11,316 categories for testing.
Evaluate Metrics Following the existing works [38, 28, 44], we conduct experiments on both image retrieval and clustering tasks.
For the retrieval task, we calculate Recall@K to evaluate the retrieval performance.
Recall@K measures the percentage of well-separated samples acknowledged if we can find at least one corrected retrieved sample in the n nearest neighbors.
For the clustering task, we adopt the Normalized Mutual Information (NMI) based on the K-means algorithm as the evaluation metric.
Baselines We apply the proposed MetricFormer to the margin loss [48] and the ProxyAnchor loss [21] for comparison.
We also compare our method with other state-of-the-art methods including pair-based methods, N-Pair [38], HTL [13], Angular [44], MS [46], Circle [39], proxy-based methods SoftTriple [31], ProxyGML [58], and ensemble methods HDC [52], A-BIER [29], ABE [22],
DiVA [26], DRML [56].
Implementation Details We implement all the following experiments using PyTorch [30] on an
NVIDIA RTX 1080ti GPU, and the ADAM optimizer [23] with a learning rate of 0.0001 is adopted to train the model.
We use ResNet-50 [17] as backbone model pre-trained on ImageNet dataset [34] for fair comparisons.
For data augmentation, we use random cropping and horizontal mirroring.
We set the number of sub-features k to 3, and the embedding size of each sub-feature is fixed to 170.
For each iteration, we fix the batch size to 112.
For the margin loss [48], we set the margin factors α and β to 1.2 and 0.2, respectively.
For the ProxyAnchor loss [21], we set the temperature α = 16 , positive margin γ pos = 1 . 8 , and negative margin γ neg = 2 . 2 .
5.2 Experimental Results
Quantitative Results We evaluate the performance of our method on three benchmarks CUB-2002011 [43], Cars196 [24] and Stanford Online Products (SOP)
The experimental results are shown in Table 2.
Note that MetricFormer-PA and MetricFormer-M represent our method combined with ProxyAnchor and Margin loss, respectively.
The best results are marked in bold red.
We highlight our superior results over the associated methods without MetricFormer in bold black.
We can easily observe that our MetricFormer can improve the performance of the original DML methods by a significant margin.
Compared with the original methods, our method can mine more informative relations from different aspects, which is useful for similarity learning.
Moreover, our method outperforms the state-of-the-art methods on all three datasets.
These empirical results support our proposal that our MetricFormer can discover the relationships among samples and features, which are ignored by most of the existing DML methods.
Within the multiple relations, we can capture the characteristics of samples more comprehensively.
Qualitative Results To further demonstrate the performance of our method, we give the visualization results of our MetricFormer on the CUB-200-2011 dataset.
As shown in Figure 5, the decoupled sub-features focus on the body, neck, and head of birds respectively, which demonstrates that the learned ensemble of features can capture and model different characteristics.
The trainable queries in the feature decoupling block can be regarded as pattern detectors and large weights are generated once a particular pattern is detected.
Moreover, we visualize the sub-features for all the samples on the CUB-200-2011 dataset using t-SNE [41] in Figure 5.
We use different colors to represent the different sub-features in a 2-dimensional space.
We observe that the learned sub-features are very diverse in the feature space.
These visualization results support our proposal that our MetricFormer can learn the ensemble of distinct and diverse features with different discriminative characteristics.
[Table]
5.3 Ablation Study
[Table]
Ablation Study of Different Components We conduct experiments with the margin loss to analyze the effectiveness of different blocks in our MetricFormer.
Margin denotes the baseline method of using the margin loss. +FDB , +BCB , +FCB are short for ‘feature decoupling block’, ‘batch-wise correlation block’ [Table] and ‘feature-wise correlation block’, respectively.
Table 3 shows the experimental results on the CUB-2002011 dataset.
It is clear that our MetricFormer achieves better performance than all the compared counterparts and all of the proposed blocks contribute to the overall improvement, which demonstrates that exploring and capturing informative [Figure] relations is crucial for similarity learning.
The relationships between samples and sub-features are both useful for similarity measurement.
Influence of Different Number of Sub-features We employ k individual features to describe each input sample from different aspects.
We conduct experiments with the margin loss ( i.e. MetricFormerM) to analyze the influence of the number of individual features.
Table 4 shows the experimental results on the CUB-200-2011 dataset.
The value of k ranges from 2, 3 to 4, rendering the size of each sub-feature to 256, 170, and 128, respectively.
We can observe that our method achieves the best performance when k = 3 , and using a larger k slightly harms the performance since each sub-feature captures less information with a larger k .
Influence of Different Number of Batch-wise Correlation Blocks T
We conduct experiments with the margin loss on the CUB-200-2011 dataset to analyze the effect of the number of batch-wise correlation blocks.
According to the experimental results shown in Table 4, we can see that exploiting more batch-wise blocks can improve the model performance.
Considering the amount of model parameters, we apply three batch-wise block in our MetricFormer, i.e. T = 3 .
[Table]
Conclusion
In this paper, we propose a metric transformer (MetricFormer) framework to explore multiple correlations in a unified perspective.
In our MetricFormer, we take the pairwise relationship, the sample relationship among each mini-batch, and the structural relationship among the ensemble of features into consideration through the feature decoupling block, the batch-wise correlation block, and the feature-wise correlation block respectively.
To this end, we can the ensemble of distinct and diverse features with different and discriminative characteristics.
Moreover, our proposed method can be flexibly integrated with any metric learning method.
We have conducted extensive experiments on three commonly used datasets which have demonstrated that our method can effectively boost the performance of the existing DML methods.
Acknowledgment
Our work was supported in part by the National Natural Science Foundation of China under Grant 62132016, Grant and Grant 62071361, in part by Key Research and Development Program of Shaanxi under Grant 2021ZDLGY01-03, and in part by the Fundamental Research Funds for the Central Universities ZDRC2102.
Erkun Yang was supported in part by the National Natural
Science Foundation of China under Grant 62202365, Guangdong Basic and Applied Basic Research
Foundation (2021A1515110026), and Natural Science Basic Research Program of Shaanxi (program
No.2022JQ-608). Broader Impact Limitation Due to the usage of transformer, the parameters of the model are increased, which leads to more computation cost and training times.
Negative Impact
The proposed method predicts content based on learned statistics of the training dataset and as such will reflect biases in those data, including ones with negative societal impacts.
This issue warrants further research and consideration
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in the appendix.
This section will often be part of the supplemental material.